Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
Canadian Journal of Anesthesia/Journal canadien d'anesthésie
Published in: Canadian Journal of Anesthesia/Journal canadien d'anesthésie 2/2012

01-02-2012 | Review Article/Brief Review

Review article: Assessment in anesthesiology education

Authors: John R. Boulet, PhD, David Murray, MD

Published in: Canadian Journal of Anesthesia/Journal canadien d'anesthésie | Issue 2/2012

Login to get access

Abstract

Purpose

The purpose of this article is to provide a review of the competencies and associated assessment techniques relevant to the practice of anesthesiology. Although many of the competencies are difficult to define and measure specifically, advances in assessment techniques have provided more opportunities to gather meaningful performance data.

Principal findings

Establishing the competence of anesthesiologists demands a host of measures, including standardized tests and less-structured peer evaluations. Simulation-based assessment will play an increasingly important role both in certification and in maintenance of certification for anesthesiologists.

Conclusions

While there are many psychometric challenges associated with the assessments pertinent to the education of anesthesiologists, technological advances combined with an increased awareness of sound measurement principles will yield more meaningful competency measures that can be used to improve the practice of anesthesiology.
Footnotes
1
American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for Educational and Psychological Testing. 1999. Washington, DC, American Educational Research Association.
 
Literature
1.
Hofer TP, Hayward RA. Are bad outcomes from questionable clinical decisions preventable medical errors? A case of cascade iatrogenesis. Ann Intern Med 2002; 137(5 Part 1): 327-33.PubMed
2.
3.
4.
5.
Bould MD, Naik VN, Hamstra SJ. Review article: New directions in medical education related to anesthesiology and perioperative medicine. Can J Anesth 2012; 59: this issue. DOI:10.​1007/​s12630-011-9633-0.
6.
Boulet JR, Murray DJ. Simulation-based assessment in anesthesiology: requirements for practical implementation. Anesthesiology 2010; 112: 1041-52.PubMedCrossRef
7.
8.
Norcini JJ, McKinley DW. Assessment methods in medical education. Teaching and Teacher Education 2007; 23: 239-50.CrossRef
9.
10.
Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA 2009; 302: 1316-26.PubMedCrossRef
11.
Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990; 65(9 Suppl): S63-7.PubMedCrossRef
12.
Edler AA, Fanning RG, Chen MI, et al. Patient simulation: a literary synthesis of assessment tools in anesthesiology. J Educ Eval Health Prof 2009; 6: 3.PubMedCrossRef
13.
Boulet JR, Smee SM, Dillon GF, Gimpel JR. The use of standardized patient assessments for certification and licensure decisions. Simul Healthc 2009; 4: 35-42.PubMedCrossRef
14.
Dillon GF, Boulet JR, Hawkins RE, Swanson DB. Simulations in the United States Medical Licensing Examination (USMLE). Qual Saf Health Care 2004; 13 Suppl 1: i41-5.PubMedCrossRef
15.
Nestel D, Kneebone R, Black S. Simulated patients and the development of procedural and operative skills. Med Teach 2006; 28: 390-1.PubMed
16.
Amin Z, Boulet JR, Cook DA, et al. Technology-enabled assessment of health professions education: consensus statement and recommendations from the Ottawa 2010 conference. Med Teach 2011; 33: 364-9.PubMedCrossRef
17.
Bould MD, Crabtree NA, Naik VN. Assessment of procedural skills in anaesthesia. Br J Anaesth 2009; 103: 472-83.PubMedCrossRef
18.
19.
Norcini J, Anderson B, Bollela V, et al. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach 2011; 33: 206-14.PubMedCrossRef
20.
Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med 2006; 119: 166.e7.CrossRef
21.
Devitt JH, Kurrek MM, Cohen MM, Cleave-Hogg D. The validity of performance assessments using simulation. Anesthesiology 2001; 95: 36-42.PubMedCrossRef
22.
Boulet JR. Generalizability theory: basics. In: Everitt BS, Howell D (Eds). Encyclopedia of Statistics in Behavioral Science. Chichester: John Wiley & Sons, Ltd; 2005. 704-11.
23.
Boulet JR, Swanson DB. Psychometric challenges of using simulations for high-stakes assessment. In: Dunn WF (Ed). Simulators in Critical Care Education and Beyond. Des Plains, IL: Society of Critical Care Medicine; 2004: 119-30.
24.
Clauser BE, Margolis MJ, Swanson DB. Issues of validity and reliability for assessments in medical education. In: Holmboe ES, Hawkins RE (Eds). Practical Guide to the Evaluation of Clinical Competence, 1st ed. Philadelphia, PA: Mosby/Elsevier; 2008: 10-23
25.
Raymond MR. Job analysis and the specification of content for licensure and certification examinations. Applied Measurement in Education 2001; 14: 369-415.CrossRef
26.
Kearney RA, Sullivan P, Skakun E. Performance on ABA-ASA in-training examination predicts success for RCPSC certification. American Board of Anesthesiology-American Society of Anesthesiologists. Royal College of Physicians and Surgeons of Canada. Can J Anesth 2000; 47: 914-8.PubMedCrossRef
27.
Gale TC, Roberts MJ, Sice PJ, et al. Predictive validity of a selection centre testing non-technical skills for recruitment to training in anaesthesia. Br J Anaesth 2010; 105: 603-9.PubMedCrossRef
28.
Murray DJ, Boulet JR, Avidan M, et al. Performance of residents and anesthesiologists in a simulation-based skill assessment. Anesthesiology 2007; 107: 705-13.PubMedCrossRef
29.
McClintock JC, Gravlee GP. Predicting success on the certification examinations of the American Board of Anesthesiology. Anesthesiology 2010; 112: 212-9.PubMedCrossRef
30.
31.
Ottestad E, Boulet JR, Lighthall GK. Evaluating the management of septic shock using patient simulation. Crit Care Med 2007; 35: 769-75.PubMedCrossRef
32.
Wilkinson JR, Crossley JG, Wragg A, Mills P, Cowan G, Wade W. Implementing workplace-based assessment across the medical specialties in the United Kingdom. Med Educ 2008; 42: 364-73.PubMedCrossRef
33.
Miller A, Archer J. Impact of workplace based assessment on doctors’ education and performance: a systematic review. BMJ 2010; 341: c5064.PubMedCrossRef
34.
Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007; 29: 855-71.PubMedCrossRef
35.
Berkenstadt H, Erez D, Munz Y, Simon D, Ziv A. Training and assessment of trauma management: the role of simulation-based medical education. Anesthesiol Clin 2007; 25: 65-74.PubMedCrossRef
36.
Hunt EA, Walker AR, Shaffner DH, Miller MR, Pronovost PJ. Simulation of in-hospital pediatric medical emergencies and cardiopulmonary arrests: highlighting the importance of the first 5 minutes. Pediatrics 2008; 121: e34-43.PubMedCrossRef
37.
Cooper JB, Taqueti VR. A brief history of the development of mannequin simulators for clinical education and training. Qual Saf Health Care 2004; 13 Suppl 1: i11-8.PubMedCrossRef
38.
Flin R, Patey R, Glavin R, Maran N. Anaesthetists’ non-technical skills. Br J Anaesth 2010; 105: 38-44.PubMedCrossRef
39.
Schuwirth LW, Southgate L, Page GG, et al. When enough is enough: a conceptual basis for fair and defensible practice performance assessment. Med Educ 2002; 36: 925-30.PubMedCrossRef
40.
Downing SM, Haladyna TM. Validity threats: overcoming interference with proposed interpretations of assessment data. Med Educ 2004; 38: 327-33.PubMedCrossRef
41.
Wass V, Van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet 2001; 357: 945-9.PubMedCrossRef
42.
43.
Dillon GF, Clyman SG, Clauser BE, Margolis MJ. The introduction of computer-based case simulations into the United States Medical Licensing Examination. Acad Med 2002; 77(10 Suppl): S94-6.PubMedCrossRef
44.
Moore DE Jr, Pennington FC. Practice-based learning and improvement. J Contin Educ Health Prof 2003; 23 Suppl 1: S73-80.PubMedCrossRef
45.
Lynch DC, Swing SR, Horowitz SD, Holt K, Messer JV. Assessing practice-based learning and improvement. Teach Learn Med 2004; 16: 85-92.PubMedCrossRef
46.
Carraccio C, Englander R. Evaluating competence using a portfolio: a literature review and web-based application to the ACGME competencies. Teach Learn Med 2004; 16: 381-7.PubMedCrossRef
47.
48.
Graham J, Hocking G, Giles E. Anaesthesia non-technical skills: can anaesthetists be trained to reliably use this behavioural marker system in 1 day? Br J Anaesth 2010; 104: 440-5.PubMedCrossRef
49.
Canas M, Moreno R, Rhodes A, Grounds RM. Patient safety in anesthesia. Minerva Anestesiol 2010; 76: 753-7.PubMed
50.
Evans RG, Edwards A, Evans S, Elwyn B, Elwyn G. Assessing the practising physician using patient surveys: a systematic review of instruments and feedback methods. Fam Pract 2007; 24: 117-27.PubMedCrossRef
51.
Lurie SJ, Mooney CJ, Lyness JM. Measurement of the general competencies of the Accreditation Council for Graduate Medical Education: a systematic review. Acad Med 2009; 84: 301-9.PubMedCrossRef
52.
Duffy FD, Gordon GH, Whelan G, et al. Assessing competence in communication and interpersonal skills: the Kalamazoo II report. Acad Med 2004; 79: 495-507.PubMedCrossRef
53.
Symons AB, Swanson A, McGuigan D, Orrange S, Akl EA. A tool for self-assessment of communication skills and professionalism in residents. BMC Med Educ 2009; 9: 1.PubMedCrossRef
54.
Ong LM, de Haes JC, Hoos AM, Lammes FB. Doctor-patient communication: a review of the literature. Soc Sci Med 1995; 40: 903-18.PubMedCrossRef
55.
Nicolai J, Demmel R. The impact of gender stereotypes on the evaluation of general practitioners’ communication skills: an experimental study using transcripts of physician-patient encounters. Patient Educ Couns 2007; 69: 200-5.PubMedCrossRef
56.
Schirmer JM, Mauksch L, Lang F, et al. Assessing communication competence: a review of current tools. Fam Med 2005; 37: 184-92.PubMed
57.
Mercer SJ, Moneypenny MJ, Guha A. Communication and simulation for anaesthetists. Anaesthesia 2009; 64: 1259-60.PubMedCrossRef
58.
59.
Whelan GP, McKinley DW, Boulet JR, Macrae J, Kamholz S. Validation of the doctor-patient communication component of the Educational Commission for Foreign Medical Graduates Clinical Skills Assessment. Med Educ 2001; 35: 757-61.PubMedCrossRef
60.
Smith AF, Greaves JD. Beyond competence: defining and promoting excellence in anaesthesia. Anaesthesia 2010; 65: 184-91.PubMedCrossRef
61.
Lynch DC, Surdyk PM, Eiser AR. Assessing professionalism: a review of the literature. Med Teach 2004; 26: 366-73.PubMedCrossRef
62.
Wilkinson TJ, Wade WB, Knock LD. A blueprint to assess professionalism: results of a systematic review. Acad Med 2009; 84: 551-8.PubMedCrossRef
63.
Veloski JJ, Fields SK, Boex JR, Blank LL. Measuring professionalism: a review of studies with instruments reported in the literature between 1982 and 2002. Acad Med 2005; 80: 366-70.PubMedCrossRef
64.
Hodges BD, Ginsburg S, Cruess R, et al. Assessment of professionalism: recommendations from the Ottawa 2010 Conference. Med Teach 2011; 33: 354-63.PubMedCrossRef
65.
Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX: a method for assessing clinical skills. Ann Intern Med 2003; 138: 476-81.PubMed
66.
van Zanten M, Boulet JR, Norcini JJ, McKinley D. Using a standardised patient assessment to measure professional attributes. Med Educ 2005; 39: 20-9.PubMedCrossRef
67.
68.
Massagli TL, Carline JD. Reliability of a 360-degree evaluation to assess resident competence. Am J Phys Med Rehabil 2007; 86: 845-52.PubMedCrossRef
69.
Heffron MG, Simspon D, Kochar MS. Competency-based physician education, recertification, and licensure. WMJ 2007; 106: 215-8.PubMed
70.
Hingle ST, Robinson S, Colliver JA, Rosher RB, McCann-Stone N. Systems-based practice assessed with a performance-based examination simulated and scored by standardized participants in the health care system: feasibility and psychometric properties. Teach Learn Med 2011; 23: 148-54.PubMedCrossRef
71.
Gaba DM. Crisis resource management and teamwork training in anaesthesia. Br J Anaesth 2010; 105: 3-6.PubMedCrossRef
72.
Wright MC, Phillips-Bute BG, Petrusa ER, Griffin KL, Hobbs GW, Taekman JM. Assessing teamwork in medical education and practice: relating behavioural teamwork ratings and clinical performance. Med Teach 2009; 31: 30-8.PubMedCrossRef
73.
Volk MS, Ward J, Irias N, Navedo A, Pollart J, Weinstock PH. Using Medical Simulation to Teach Crisis Resource Management and Decision-Making Skills to Otolaryngology Housestaff. Otolaryngol Head Neck Surg 2011; 145: 35-42.PubMedCrossRef
74.
Morgan PJ, Pittini R, Regehr G, Marrs C, Haley MF. Evaluating teamwork in a simulated obstetric environment. Anesthesiology 2007; 106: 907-15.PubMedCrossRef
75.
Case SM, Swanson DB. Constructing Written Test Questions for the Basic and Clinical Sciences, Third Edition (revised). Philadelphia, PA: National Board of Medical Examiners; 2010 .
76.
Edler AA. The use of simulation education in competency assessment: more questions than answers. Anesthesiology 2008; 108: 167.PubMedCrossRef
77.
Weller JM, Jolly B, Robinson B. Generalisability of behavioural skills in simulated anaesthetic emergencies. Anaesth Intensive Care 2008; 36: 185-9.PubMed
78.
Hodges B, Regehr G, McNaughton N, Tiberius R, Hanson M. OSCE checklists do not capture increasing levels of expertise. Acad Med 1999; 74: 1129-34.PubMedCrossRef
79.
Hodges B, McNaughton N, Regehr G, Tiberius R, Hanson M. The challenge of creating new OSCE measures to capture the characteristics of expertise. Med Educ 2002; 36: 742-8.PubMedCrossRef
80.
Scavone BM, Sproviero MT, McCarthy RJ, et al. Development of an objective scoring system for measurement of resident performance on the human patient simulator. Anesthesiology 2006; 105: 260-6.PubMedCrossRef
81.
De Champlain AF. Ensuring that the competent are truly competent: an overview of common methods and procedures used to set standards on high-stakes examinations. J Vet Med Educ 2004; 31: 61-5.PubMedCrossRef
82.
Downing SM, Tekian A, Yudkowsky R. Procedures for establishing defensible absolute passing scores on performance examinations in health professions education. Teach Learn Med 2006; 18: 50-7.PubMedCrossRef
83.
Boulet JR, Murray D, Kras J, Woodhouse J. Setting performance standards for mannequin-based acute-care scenarios: an examinee-centered approach. Simul Healthc 2008; 3: 72-81.PubMedCrossRef
Metadata
Title
Review article: Assessment in anesthesiology education
Authors
John R. Boulet, PhD
David Murray, MD
Publication date
01-02-2012
Publisher
Springer-Verlag
Published in
Canadian Journal of Anesthesia/Journal canadien d'anesthésie / Issue 2/2012
Print ISSN: 0832-610X
Electronic ISSN: 1496-8975
DOI
https://doi.org/10.1007/s12630-011-9637-9

Other articles of this Issue 2/2012

Canadian Journal of Anesthesia/Journal canadien d'anesthésie 2/2012 Go to the issue

Correspondence

A flexible gel pad as an effective medium for scanning irregular surface anatomy

Review Article/Brief Review

Review article: Closing the research gap at the interface of learning and clinical practice

Correspondence

New paradigms for managing preoperative anemia

Review Article/Brief Review

Review article: New directions in medical education related to anesthesiology and perioperative medicine

Editorials

A user’s guide to reading the scholarship of anesthesia education

Review Article/Brief Review

Review article: The influence of psychology and human factors on education in anesthesiology