Skip to main content
Top
Published in: Israel Journal of Health Policy Research 1/2019

Open Access 01-12-2019 | Original research article

Measuring the academic value of academic medical centers: describing a methodology for developing an evaluation model at one Academic Medical Center

Authors: Rafael Hod, Oded Maimon, Eyal Zimlichman

Published in: Israel Journal of Health Policy Research | Issue 1/2019

Login to get access

Abstract

Background

Academic Medical Centers (AMCs) must simultaneously serve different purposes:
Delivery of high quality healthcare services to patients, as the main mission, supported by other core missions such as academic activities, i.e., researching, teaching and tutoring, while maintaining solvency.
This study aims to develop a methodology for constructing models evaluating the academic value provided by AMCs and implementing it at the largest AMC in Israel.

Methods

Thirty five practiced educators and researchers, academic experts, faculty members and executives, all employed by a metropolitan 1500-bed AMC, were involved in developing academic quality indicators. First, an initial list of AMCs’ academic quality indicators was drafted, using a literature review and consulting scholars. Afterwards, additional data and preferences were collected by conducting semi-structured interviews, complemented by a three-round Delphi Panel. Finally, the methodology for constructing a model evaluating the academic value provided by the AMC was developed.

Results

The composite academic quality indicators methodology consists of nine indicators (relative weight in parentheses): ‘Scientific Publications Value’ (18.7%), ‘Completed Studies’ (13.5%), ‘Authors Value’ (13.0%), ‘Residents Quality’ (11.3%), ‘Competitive Grants Budget’ (10.2%), ‘Academic Training’ (8.7%), ‘Academic Positions’ (8.3%), ‘Number of Studies’ (8.3%) and ‘Academic Supervision’ (8.0%).
These indicators were grouped into three core categories: ‘Education’, ‘Research’ and ‘Publications’, having almost the same importance on a scale from zero to one (0–1), i.e., 0.363, 0.320, and 0.317, respectively. The results demonstrated a high level of internal consistency (Cronbach-alpha range: 0.79–0.86).

Conclusions

We have found a gap in the ability to measure academic value provided by AMCs. The main contribution of this research is the development of methodology for constructing evaluation models for AMCs academic performance. Further studies are needed to further test the validity and reliability of the proposed methodology at other sites.
Appendix
Available only for authorised users
Literature
1.
go back to reference Academic health centers: Leading change in the 21st century. Institute of Medicine (US) Committee on the Roles of Academic Health Centers in the 21st Century. Kohn L. T, editor. Washington (DC): National. https://doi.org/10.17226/10734. Accessed 26 Oct 2018. Academic health centers: Leading change in the 21st century. Institute of Medicine (US) Committee on the Roles of Academic Health Centers in the 21st Century. Kohn L. T, editor. Washington (DC): National. https://​doi.​org/​10.​17226/​10734. Accessed 26 Oct 2018.
3.
go back to reference Rosko MD. Performance of major teaching hospitals during the 1990s: adapting to turbulent times. J Health Care Finance. 2004;30(3):34–48.PubMed Rosko MD. Performance of major teaching hospitals during the 1990s: adapting to turbulent times. J Health Care Finance. 2004;30(3):34–48.PubMed
4.
go back to reference Patel VM, Ashrafian H, Ahmed K, Arora S, Jiwan S, Nicholson JK, et al. How has healthcare research performance been assessed? A systematic review. J R Soc Med. 2011;104(6):251–61.CrossRef Patel VM, Ashrafian H, Ahmed K, Arora S, Jiwan S, Nicholson JK, et al. How has healthcare research performance been assessed? A systematic review. J R Soc Med. 2011;104(6):251–61.CrossRef
5.
go back to reference Morey RC, Retzlaff-Roberts DL, Fine DJ, Loree SW. Assessing the operating efficiencies of teaching hospitals by an enhancement of the AHA/AAMC method. Acad Med. 2000;75(1):28–40.CrossRef Morey RC, Retzlaff-Roberts DL, Fine DJ, Loree SW. Assessing the operating efficiencies of teaching hospitals by an enhancement of the AHA/AAMC method. Acad Med. 2000;75(1):28–40.CrossRef
6.
go back to reference Pizzini MJ. The relation between cost-system design, managers’ evaluations of the relevance and usefulness of cost data, and financial performance: an empirical study of US hospitals. Acc Organ Soc. 2006;31(2):179–210.CrossRef Pizzini MJ. The relation between cost-system design, managers’ evaluations of the relevance and usefulness of cost data, and financial performance: an empirical study of US hospitals. Acc Organ Soc. 2006;31(2):179–210.CrossRef
7.
go back to reference Isaac T., Zaslavsky A.M., Cleary P.D., Landon B.E. The relationship between patients’ perception of care and measures of hospital quality and safety. Health Serv Res. 2010:1;45(4):1024–1040.CrossRef Isaac T., Zaslavsky A.M., Cleary P.D., Landon B.E. The relationship between patients’ perception of care and measures of hospital quality and safety. Health Serv Res. 2010:1;45(4):1024–1040.CrossRef
8.
go back to reference Porter M.E. What is value in health care? N Engl J Med. 2010:23; 363(26):2477–2481. (App 2).CrossRef Porter M.E. What is value in health care? N Engl J Med. 2010:23; 363(26):2477–2481. (App 2).CrossRef
9.
go back to reference Schreyögg J., von Reitzenstein C. Strategic groups and performance differences among academic medical centers. Health Care Manag Rev. 2008:1;33(3):225–233.CrossRef Schreyögg J., von Reitzenstein C. Strategic groups and performance differences among academic medical centers. Health Care Manag Rev. 2008:1;33(3):225–233.CrossRef
10.
go back to reference Hilton C, Fisher W, Lopez A, Sanders C. A relative-value-based system for calculating faculty productivity in teaching, research, administration, and patient care. Academic Medicine: journal of the Association of American Medical Colleges. 1997;72(9):787–93.CrossRef Hilton C, Fisher W, Lopez A, Sanders C. A relative-value-based system for calculating faculty productivity in teaching, research, administration, and patient care. Academic Medicine: journal of the Association of American Medical Colleges. 1997;72(9):787–93.CrossRef
11.
go back to reference Catuogno S. Balanced performance measurement in research hospitals: the participative case study of a hematology department. BMC Health Serv Res. 2017;17(1):522.CrossRef Catuogno S. Balanced performance measurement in research hospitals: the participative case study of a hematology department. BMC Health Serv Res. 2017;17(1):522.CrossRef
12.
go back to reference Brocato J.J., Mavis B. The research productivity of faculty in family medicine departments at US medical schools: A national study. Acad Med. 2005:1;80(3):244–252.CrossRef Brocato J.J., Mavis B. The research productivity of faculty in family medicine departments at US medical schools: A national study. Acad Med. 2005:1;80(3):244–252.CrossRef
13.
go back to reference Holmes E.W., Burks T.F., Dzau V., Hindery M.A., Jones R.F., Kaye C.I., et al. Measuring contributions to the research mission of medical schools. Acad Med. 2000:1; 75(3):304–313.CrossRef Holmes E.W., Burks T.F., Dzau V., Hindery M.A., Jones R.F., Kaye C.I., et al. Measuring contributions to the research mission of medical schools. Acad Med. 2000:1; 75(3):304–313.CrossRef
14.
go back to reference Wootton R. A simple, generalizable method for measuring individual research productivity and its use in the long-term analysis of departmental performance, including between-country comparisons. Health Research Policy and Systems. 2013;11(1):2.CrossRef Wootton R. A simple, generalizable method for measuring individual research productivity and its use in the long-term analysis of departmental performance, including between-country comparisons. Health Research Policy and Systems. 2013;11(1):2.CrossRef
15.
go back to reference Flanders SA, Centor B., Weber V., McGinn T., De Salvo K., Auerbach A. Challenges and opportunities in academic hospital medicine: report from the academic hospital medicine summit. J Gen Intern Med. 2009:1;24(5):636–641.CrossRef Flanders SA, Centor B., Weber V., McGinn T., De Salvo K., Auerbach A. Challenges and opportunities in academic hospital medicine: report from the academic hospital medicine summit. J Gen Intern Med. 2009:1;24(5):636–641.CrossRef
16.
go back to reference Ovseiko PV, Oancea A, Buchan AM. Assessing research impact in academic clinical medicine: a study using research excellence framework pilot impact indicators. BMC Health Serv Res. 2012;12(1):478.CrossRef Ovseiko PV, Oancea A, Buchan AM. Assessing research impact in academic clinical medicine: a study using research excellence framework pilot impact indicators. BMC Health Serv Res. 2012;12(1):478.CrossRef
17.
go back to reference Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–15.PubMed Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–15.PubMed
18.
go back to reference Cope A., Bezemer J., Mavroveli S., Kneebone R. What attitudes and values are incorporated into self as part of professional identity construction when becoming a surgeon? Acad Med. 2017;1;92(4):544–549.CrossRef Cope A., Bezemer J., Mavroveli S., Kneebone R. What attitudes and values are incorporated into self as part of professional identity construction when becoming a surgeon? Acad Med. 2017;1;92(4):544–549.CrossRef
19.
go back to reference Jacobs R., Smith P.C., Goddard M.K. Measuring performance: an examination of composite performance indicators: a report for the Department of Health. Centre of Health Economics, University of York; 2004:27–92. Jacobs R., Smith P.C., Goddard M.K. Measuring performance: an examination of composite performance indicators: a report for the Department of Health. Centre of Health Economics, University of York; 2004:27–92.
20.
go back to reference Landeta J. Current validity of the Delphi method in social sciences. Technol Forecast Soc Chang. 2006:1;73(5):467–482.CrossRef Landeta J. Current validity of the Delphi method in social sciences. Technol Forecast Soc Chang. 2006:1;73(5):467–482.CrossRef
21.
go back to reference Maxwell JA. Designing a qualitative study. The SAGE Handbook of Applied Social Research Methods. 2008;2:214–53. Maxwell JA. Designing a qualitative study. The SAGE Handbook of Applied Social Research Methods. 2008;2:214–53.
22.
go back to reference Mason M. Sample size and saturation in PhD studies using qualitative interviews. In Forum: Qualitative Social Research. 2010;24:1–19. Mason M. Sample size and saturation in PhD studies using qualitative interviews. In Forum: Qualitative Social Research. 2010;24:1–19.
23.
go back to reference Diamond I.R., Grant R.C., Feldman B.M., Pencharz P.B., Ling S.C., Moore A.M., Wales PW. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014:1; 67(4):401–409.CrossRef Diamond I.R., Grant R.C., Feldman B.M., Pencharz P.B., Ling S.C., Moore A.M., Wales PW. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014:1; 67(4):401–409.CrossRef
24.
go back to reference Globerson S. Issues in developing a performance criteria system for an organization. Int J Prod Res. 1985:1;23(4):639–646.CrossRef Globerson S. Issues in developing a performance criteria system for an organization. Int J Prod Res. 1985:1;23(4):639–646.CrossRef
25.
go back to reference Chassin MR, Jerod ML, Stephen PS, Robert M. Accountability measures: using measurement to promote quality improvement. N Engl J Med. 2010;363:683–8.CrossRef Chassin MR, Jerod ML, Stephen PS, Robert M. Accountability measures: using measurement to promote quality improvement. N Engl J Med. 2010;363:683–8.CrossRef
26.
go back to reference Adler Y, Kinori M, Zimlichman E, Rosinger A, Shalev G, Talmi R, et al. “The Talpiot medical leadership program”- Advancing the brightest young physicians and researchers to fill future leadership roles. Harefuah. 2015;154(2):107–9.PubMed Adler Y, Kinori M, Zimlichman E, Rosinger A, Shalev G, Talmi R, et al. “The Talpiot medical leadership program”- Advancing the brightest young physicians and researchers to fill future leadership roles. Harefuah. 2015;154(2):107–9.PubMed
27.
go back to reference Boulkedid R., Abdoul H., Loustau M., Sibony O., Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: A systematic review. PLoS One. 2011;9;6:e20476.CrossRef Boulkedid R., Abdoul H., Loustau M., Sibony O., Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: A systematic review. PLoS One. 2011;9;6:e20476.CrossRef
28.
go back to reference Souba WW, Wilmore DW. Judging surgical research: how should we evaluate performance and measure value? Ann Surg. 2000;232(1):32–41.CrossRef Souba WW, Wilmore DW. Judging surgical research: how should we evaluate performance and measure value? Ann Surg. 2000;232(1):32–41.CrossRef
29.
go back to reference Copeland HL, Hewson MG. Developing and testing an instrument to measure the effectiveness of clinical teaching in an academic medical center. Acad Med. 2000;1(75):161–6.CrossRef Copeland HL, Hewson MG. Developing and testing an instrument to measure the effectiveness of clinical teaching in an academic medical center. Acad Med. 2000;1(75):161–6.CrossRef
31.
go back to reference Hanney SR, Grant J, Wooding S, Buxton MJ. Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK’s’ arthritis research campaign’. Health Research Policy and Systems. 2004;2(1):4.CrossRef Hanney SR, Grant J, Wooding S, Buxton MJ. Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK’s’ arthritis research campaign’. Health Research Policy and Systems. 2004;2(1):4.CrossRef
32.
go back to reference Stronach I. On promoting rigor in educational research: The example of the RAE. Journal of Education Policy. 2007:1;22(3):343–52.CrossRef Stronach I. On promoting rigor in educational research: The example of the RAE. Journal of Education Policy. 2007:1;22(3):343–52.CrossRef
34.
go back to reference Noruzi A. Impact Factor, h-index, i10-index and i20-index of Webology. Webology. 2016;1;13(1):1. Noruzi A. Impact Factor, h-index, i10-index and i20-index of Webology. Webology. 2016;1;13(1):1.
35.
go back to reference Thonon F., Boulkedid R., Delory T., Rousseau S., Saghatchian M., van Harten W., et al. Measuring the outcome of biomedical research: a systematic literature review. PLoS One. 2015; 2;10(4):e0122239.CrossRef Thonon F., Boulkedid R., Delory T., Rousseau S., Saghatchian M., van Harten W., et al. Measuring the outcome of biomedical research: a systematic literature review. PLoS One. 2015; 2;10(4):e0122239.CrossRef
36.
go back to reference Ziglio E. The Delphi method and its contribution to decision making. In: Adler M, Ziglio E, editors. Gazing into the oracle: the Delphi method and its application to social policy and public health, vol. 5. London: Jessica Kingsley publishers; 1996. p. 3–33. Ziglio E. The Delphi method and its contribution to decision making. In: Adler M, Ziglio E, editors. Gazing into the oracle: the Delphi method and its application to social policy and public health, vol. 5. London: Jessica Kingsley publishers; 1996. p. 3–33.
37.
go back to reference Cooke M, Irby DM, O'Brien BC. Educating physicians: a call for reform of medical school and residency. John Wiley & Sons; 2010:24–26. Cooke M, Irby DM, O'Brien BC. Educating physicians: a call for reform of medical school and residency. John Wiley & Sons; 2010:24–26.
Metadata
Title
Measuring the academic value of academic medical centers: describing a methodology for developing an evaluation model at one Academic Medical Center
Authors
Rafael Hod
Oded Maimon
Eyal Zimlichman
Publication date
01-12-2019
Publisher
BioMed Central
Published in
Israel Journal of Health Policy Research / Issue 1/2019
Electronic ISSN: 2045-4015
DOI
https://doi.org/10.1186/s13584-019-0334-4

Other articles of this Issue 1/2019

Israel Journal of Health Policy Research 1/2019 Go to the issue