Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
Surgical Endoscopy
Published in: Surgical Endoscopy 1/2020

Open Access 01-01-2020

Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training

Authors: Keith Siau, James Crossley, Paul Dunckley, Gavin Johnson, Mark Feeney, Neil D. Hawkes, Ian L. P. Beales, The Joint Advisory Group on Gastrointestinal Endoscopy (JAG)

Published in: Surgical Endoscopy | Issue 1/2020

Login to get access

Abstract

Background

Validated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of the formative direct observation of procedural skills (DOPS) assessment tool in diagnostic gastroscopy and study competency development using DOPS.

Methods

This was a prospective multicentre (N = 275) analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory factor analysis and reliability estimated using generalisability theory. Item and global DOPS scores were stratified by lifetime procedure count to define learning curves, using a threshold determined from receiver operator characteristics (ROC) analysis. Multivariable binary logistic regression analysis was performed to identify independent predictors of DOPS competence.

Results

In total, 10086 DOPS were submitted for 987 trainees. Exploratory factor analysis identified three distinct item groupings, representing ‘pre-procedure’, ‘technical’, and ‘post-procedure non-technical’ skills. From generalisability analyses, sources of variance in overall DOPS scores included trainee ability (31%), assessor stringency (8%), assessor subjectivity (18%), and trainee case-to-case variation (43%). The combination of three assessments from three assessors was sufficient to achieve the reliability threshold of 0.70. On ROC analysis, a mean score of 3.9 provided optimal sensitivity and specificity for determining competency. This threshold was attained in the order of ‘pre-procedure’ (100–124 procedures), ‘technical’ (150–174 procedures), ‘post-procedure non-technical’ skills (200–224 procedures), and global competency (225–249 procedures). Higher lifetime procedure count, DOPS count, surgical trainees and assessors, higher trainee seniority, and lower case difficulty were significant multivariable predictors of DOPS competence.

Conclusion

This study establishes milestones for competency acquisition during gastroscopy training and provides validity and reliability evidence to support gastroscopy DOPS as a competency assessment tool.
Appendix
Available only for authorised users
Literature
1.
Ooi J, Wilson P, Walker G et al (2017) Dedicated Barrett’s surveillance sessions managed by trained endoscopists improve dysplasia detection rate. Endoscopy 49(6):524–528CrossRef
2.
Teh JL, Tan JR, Lau LJF et al (2015) Longer examination time improves detection of gastric cancer during diagnostic upper gastrointestinal endoscopy. Clin Gastroenterol Hepatol 13(3):480–487.e2CrossRef
3.
Siau K, Li J, Fisher NC et al (2017) Intubation failure during gastroscopy: incidence, predictors and follow-up findings. J Gastrointest Liver Dis 26(4):339–344
4.
Ward ST, Hancox A, Mohammed MA et al (2017) The learning curve to achieve satisfactory completion rates in upper GI endoscopy: an analysis of a national training database. Gut 66(6):1022–1030CrossRef
5.
6.
7.
Mehta T, Dowler K, McKaig BC et al (2010) Development and roll out of the JETS e-portfolio: a web based electronic portfolio for endoscopists. Frontline Gastroenterology 2(1):35CrossRef
8.
Siau K, Dunckley P, Valori R et al (2018) Changes in scoring of Direct Observation of Procedural Skills (DOPS) forms and the impact on competence assessment. Endoscopy 50(8):770–778CrossRef
9.
10.
11.
12.
Cass OW, Freeman ML, Peine CJ et al (1993) Objective evaluation of endoscopy skills during training. Ann Intern Med 118(1):40–44CrossRef
13.
Messick S (1995) Validity of psychological assessment: validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. Am Psychol 50(9):741CrossRef
14.
Cook DA, Hatala R (2016) Validation of educational assessments: a primer for simulation and beyond. Adv Simul 1(1):31CrossRef
15.
Gaskin CJ, Happell B (2014) On exploratory factor analysis: a review of recent evidence, an assessment of current practice, and recommendations for future use. Int J Nurs Stud 51(3):511–521CrossRef
16.
Manly BF, Alberto JAN (2016) Multivariate statistical methods: a primer. Chapman and Hall/CRC, Boca Raton
17.
Crossley J, Johnson G, Booth J et al (2011) Good questions, good answers: construct alignment improves the performance of workplace-based assessment scales. Med Educ 45(6):560–569CrossRef
18.
Webb NM, Shavelson RJ, Haertel EH (2006) Reliability coefficients and generalizability theory. In: Rao CR, Sinharay S (eds) Handbook of statistics. Elsevier, Amsterdam, pp 81–124
19.
Walsh CM (2016) In-training gastrointestinal endoscopy competency assessment tools: types of tools, validation and impact. Best Pract Res Clin Gastroenterol 30(3):357–374CrossRef
20.
Barton JR, Corbett S, van der Vleuten CP (2012) The validity and reliability of a Direct Observation of Procedural Skills assessment tool: assessing colonoscopic skills of senior endoscopists. Gastrointest Endosc 75(3):591–597CrossRef
21.
Leung FW (2018) Assessment of trainees’ performance in colonoscopy. Gastrointest Endosc 87(1):270–271CrossRef
22.
Jairath V, Kahan BC, Gray A et al (2015) Restrictive versus liberal blood transfusion for acute upper gastrointestinal bleeding (TRIGGER): a pragmatic, open-label, cluster randomised feasibility trial. Lancet 386(9989):137–144CrossRef
23.
Ward ST, Mohammed MA, Walt R et al (2014) An analysis of the learning curve to achieve competency at colonoscopy using the JETS database. Gut 63(11):1746CrossRef
24.
Ekkelenkamp VE, Koch AD, de Man RA et al (2016) Training and competence assessment in GI endoscopy: a systematic review. Gut 65(4):607–615CrossRef
25.
Vassiliou MC, Kaneva PA, Poulose BK et al (2010) Global Assessment of Gastrointestinal Endoscopic Skills (GAGES): a valid measurement tool for technical skills in flexible endoscopy. Surg Endosc 24(8):1834–1841CrossRef
26.
Sedlack RE, Coyle WJ, Obstein KL et al (2014) ASGE’s assessment of competency in endoscopy evaluation tools for colonoscopy and EGD. Gastrointest Endosc 79(1):1–7CrossRef
27.
28.
29.
Jones DB, Hunter JG, Townsend CM et al (2017) SAGES rebuttal. Gastrointest Endosc 86(4):751–754CrossRef
Metadata
Title
Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training
Authors
Keith Siau
James Crossley
Paul Dunckley
Gavin Johnson
Mark Feeney
Neil D. Hawkes
Ian L. P. Beales
The Joint Advisory Group on Gastrointestinal Endoscopy (JAG)
Publication date
01-01-2020
Publisher
Springer US
Published in
Surgical Endoscopy / Issue 1/2020
Print ISSN: 0930-2794
Electronic ISSN: 1432-2218
DOI
https://doi.org/10.1007/s00464-019-06737-7

Other articles of this Issue 1/2020

Surgical Endoscopy 1/2020 Go to the issue

OriginalPaper

Microscopic positive tumor margin does not increase the rate of recurrence in endoscopic resected gastric mesenchymal tumors compared to negative tumor margin

OriginalPaper

Intraoperative angiography with indocyanine green to assess anastomosis perfusion in patients undergoing laparoscopic colorectal resection: results of a multicenter randomized controlled trial

OriginalPaper

Feasibility and safety of laparoscopy-assisted distal gastrectomy performed by trainees supervised by an experienced qualified surgeon

OriginalPaper

CLEAN-NET: a modified laparoendoscopic wedge resection of the stomach to minimize the sacrifice of innocent gastric wall

OriginalPaper

Proposal of high-risk adenoma detection rate as an impactful, complementary quality indicator of colonoscopy

SAGES White Paper

Surgical management of gastroesophageal reflux disease in the obese patient