Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

Keynote webinar | Spotlight on medication adherence

LIVE: Thursday 27th June 2024, 18:00-19:30 (CEST)
Join our expert panel to discover why you need to understand the drivers of non-adherence in your patients, and how you can optimize medication adherence in your clinics to drastically improve patient outcomes.
ADVANCED SEARCH
Log in
Top
Canadian Journal of Anesthesia/Journal canadien d'anesthésie
Published in: Canadian Journal of Anesthesia/Journal canadien d'anesthésie 2/2019

01-02-2019 | Reports of Original Investigations

The reliability of a portfolio of workplace-based assessments in anesthesia training

Authors: Damian J. Castanelli, MBBS, Joyce M. W. Moonen-van Loon, PhD, Brian Jolly, PhD, Jennifer M. Weller, MD

Published in: Canadian Journal of Anesthesia/Journal canadien d'anesthésie | Issue 2/2019

Login to get access

Abstract

Purpose

Competency-based anesthesia training programs require robust assessment of trainee performance and commonly combine different types of workplace-based assessment (WBA) covering multiple facets of practice. This study measured the reliability of WBAs in a large existing database and explored how they could be combined to optimize reliability for assessment decisions.

Methods

We used generalizability theory to measure the composite reliability of four different types of WBAs used by the Australian and New Zealand College of Anaesthetists: mini-Clinical Evaluation Exercise (mini-CEX), direct observation of procedural skills (DOPS), case-based discussion (CbD), and multi-source feedback (MSF). We then modified the number and weighting of WBA combinations to optimize reliability with fewer assessments.

Results

We analyzed 67,405 assessments from 1,837 trainees and 4,145 assessors. We assumed acceptable reliability for interim (intermediate stakes) and final (high stakes) decisions of 0.7 and 0.8, respectively. Depending on the combination of WBA types, 12 assessments allowed the 0.7 threshold to be reached where one assessment of any type has the same weighting, while 20 were required for reliability to reach 0.8. If the weighting of the assessments is optimized, acceptable reliability for interim and final decisions is possible with nine (e.g., two DOPS, three CbD, two mini-CEX, two MSF) and 15 (e.g., two DOPS, eight CbD, three mini-CEX, two MSF) assessments respectively.

Conclusions

Reliability is an important factor to consider when designing assessments, and measuring composite reliability can allow the selection of a WBA portfolio with adequate reliability to provide evidence for defensible decisions on trainee progression.
Literature
1.
2.
Van Gessel E, Mellin-Olsen J, Ostergaard HT. Niemi-Murola L; Education and Training Standing Committee, European Board of Anaesthesiology, Reanimation and Intensive Care. Postgraduate training in anaesthesiology, pain and intensive care: the new European competence-based guidelines. Eur J Anaesthesiol 2012; 29: 165-8.CrossRefPubMed
3.
Stodel EJ, Wyand A, Crooks S, Moffett S, Chiu M, Hudson CC. Designing and implementing a competency-based training program for anesthesiology residents at the University of Ottawa. Anesthesiol Res Pract 2015; 2015: 713038.PubMedPubMedCentral
4.
van der Vleuten CP, Schuwirth LW, Driessen EW, et al. A model for programmatic assessment fit for purpose. Med Teach 2012; 34: 205-14.CrossRefPubMed
5.
Bok HG, Teunissen PW, Favier RP, et al. Programmatic assessment of competency-based workplace learning: when theory meets practice. BMC Med Educ 2013; 13: 123.CrossRefPubMedPubMedCentral
6.
Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007; 29: 855-71.CrossRef
7.
Weller JM, Castanelli DJ, Chen Y, Jolly B. Making robust assessments of specialist trainees’ workplace performance. Br J Anaesth 2017; 118: 207-14.CrossRefPubMed
8.
Weller JM, Jolly B, Misur MP, et al. Mini-clinical evaluation exercise in anaesthesia training. Br J Anaesth 2009; 102: 633-41.CrossRefPubMed
9.
Weller JM, Misur M, Nicolson S, et al. Can I leave the theatre? A key to more reliable workplace-based assessment. Br J Anaesth 2014; 112: 1083-91.CrossRefPubMed
10.
Watson MJ, Wong DM, Kluger R, et al. Psychometric evaluation of a direct observation of procedural skills assessment tool for ultrasound-guided regional anaesthesia. Anaesthesia 2014; 69: 604-12.CrossRefPubMed
11.
Moonen-van Loon JM, Overeem K, Donkers HH, van der Vleuten CP, Driessen EW. Composite reliability of a workplace-based assessment toolbox for postgraduate medical education. Adv Health Sci Educ Theory Pract 2013; 18: 1087-102.CrossRefPubMed
12.
Wragg A, Wade W, Fuller G, Cowan G, Mills P. Assessing the performance of specialist registrars. Clin Med (Lond) 2003; 3: 131-4.CrossRef
13.
14.
Crossley J, Russell J, Jolly B, et al. ‘I’m pickin’ up good regressions’: the governance of generalisability analyses. Med Educ 2007; 41: 926-34.CrossRefPubMed
15.
Moonen-van Loon JM, Overeem K, Govaerts MJ, Verhoeven BH, van der Vleuten CP, Driessen EW. The reliability of multisource feedback in competency-based assessment programs: the effects of multiple occasions and assessor groups. Acad Med 2015; 90: 1093-9.CrossRefPubMed
16.
Crossley J, Jolly B. Making sense of work-based assessment: ask the right questions, in the right way, about the right things, of the right people. Med Educ 2012; 46: 28-37.CrossRefPubMed
17.
Pelgrim EA, Kramer AW, Mokkink HG, van den Elsen L, Grol RP, van der Vleuten CP. In-training assessment using direct observation of single-patient encounters: a literature review. Adv Health Sci Educ Theory Pract 2011; 16: 131-42.CrossRefPubMed
18.
Lockyer J, Violato C, Fidler H. A multi source feedback program for anesthesiologists. Can J Anesth 2006; 53: 33-9.CrossRefPubMed
19.
Brennan RL. Generalizability Theory and classical test theory. J Appl Meas Educ 2010; 24: 1-21.CrossRef
20.
van der Vleuten CP, Schuwirth LW, Scheele F, Driessen EW, Hodges B. The assessment of professional competence: building blocks for theory development. Best Pract Res Clin Obstet Gynaecol 2010; 24: 703-19.CrossRefPubMed
21.
Castanelli DJ, Jowsey T, Chen Y, Weller JM. Perceptions of purpose, value, and process of the mini-Clinical Evaluation Exercise in anesthesia training. Can J Anesth 2016; 63: 1345-56.CrossRefPubMed
22.
Driessen EW, van Tartwijk J, Govaerts M, Teunissen P, van der Vleuten CP. The use of programmatic assessment in the clinical workplace: a Maastricht case report. Med Teach 2012; 34: 226-31.CrossRefPubMed
23.
Van Tartwijk J, Driessen EW. Portfolios for assessment and learning: AMEE Guide no. 45. Med Teach 2009; 31: 790-801.CrossRefPubMed
24.
Ten Cate O, Chen HC, Hoff RG, Peters H, Bok H, van der Schaaf M. Curriculum development for the workplace using Entrustable Professional Activities (EPAs): AMEE Guide No. 99. Med Teach 2015; 37: 983-1002.CrossRefPubMed
25.
Wisman-Zwarter N, van der Schaaf M, Ten Cate O, Jonker G, van Klei WA, Hoff RG. Transforming the learning outcomes of anaesthesiology training into entrustable professional activities: a Delphi study. Eur J Anaesthesiol 2016; 33: 559-67.CrossRefPubMed
26.
Jonker G, Hoff RG, Ten Cate OT. A case for competency-based anaesthesiology training with entrustable professional activities: an agenda for development and research. Eur J Anaesthesiol 2015; 32: 71-6.CrossRefPubMed
27.
Eva KW, Macala C. Multiple mini-interview test characteristics: ‘tis better to ask candidates to recall than to imagine. Med Educ 2014; 48: 604-13.CrossRefPubMed
Metadata
Title
The reliability of a portfolio of workplace-based assessments in anesthesia training
Authors
Damian J. Castanelli, MBBS
Joyce M. W. Moonen-van Loon, PhD
Brian Jolly, PhD
Jennifer M. Weller, MD
Publication date
01-02-2019
Publisher
Springer International Publishing
Published in
Canadian Journal of Anesthesia/Journal canadien d'anesthésie / Issue 2/2019
Print ISSN: 0832-610X
Electronic ISSN: 1496-8975
DOI
https://doi.org/10.1007/s12630-018-1251-7

Other articles of this Issue 2/2019

Canadian Journal of Anesthesia/Journal canadien d'anesthésie 2/2019 Go to the issue

Correspondence

When and how should we cluster and cross over: methodological and ethical issues (letter 1)

Correspondence

In reply: Because it’s 2018: the need for early career development for female anesthesiologists

Correspondence

Évaluation de l’impact du projet Lifebox au Burkina Faso sur la pratique de l’oxymétrie pulsée et la check-list de l’organisation mondiale de la santé pour la sécurité du patient au bloc opératoire

Correspondence

Because it’s 2018: the need for early career development for female anesthesiologists

Images in Anesthesia

Massive subcutaneous emphysema during bag-mask ventilation after failed intubation

Correspondence

A friend in need: evaluating the impact of Lifebox in Burkina Faso