Top

Published in:

01-12-2018

Development and validation of the TOCO–TURBT tool: a summative assessment tool that measures surgical competency in transurethral resection of bladder tumour

Authors: Anna H. de Vries, Arno. M. M. Muijtjens, Hilde G. J. van Genugten, Ad. J. M. Hendrikx, Evert L. Koldewijn, Barbara M. A. Schout, Cees P. M. van der Vleuten, Cordula Wagner, Irene M. Tjiam, Jeroen J. G. van Merriënboer

Published in: Surgical Endoscopy | Issue 12/2018

Abstract

Background

The current shift towards competency-based residency training has increased the need for objective assessment of skills. In this study, we developed and validated an assessment tool that measures technical and non-technical competency in transurethral resection of bladder tumour (TURBT).

Methods

The ‘Test Objective Competency’ (TOCO)–TURBT tool was designed by means of cognitive task analysis (CTA), which included expert consensus. The tool consists of 51 items, divided into 3 phases: preparatory (n = 15), procedural (n = 21), and completion (n = 15). For validation of the TOCO–TURBT tool, 2 TURBT procedures were performed and videotaped by 25 urologists and 51 residents in a simulated setting. The participants’ degree of competence was assessed by a panel of eight independent expert urologists using the TOCO–TURBT tool. Each procedure was assessed by two raters. Feasibility, acceptability and content validity were evaluated by means of a quantitative cross-sectional survey. Regression analyses were performed to assess the strength of the relation between experience and test scores (construct validity). Reliability was analysed by generalizability theory.

Results

The majority of assessors and urologists indicated the TOCO–TURBT tool to be a valid assessment of competency and would support the implementation of the TOCO–TURBT assessment as a certification method for residents. Construct validity was clearly established for all outcome measures of the procedural phase (all r > 0.5, p < 0.01). Generalizability-theory analysis showed high reliability (coefficient Phi ≥ 0.8) when using the format of two assessors and two cases.

Conclusions

This study provides first evidence that the TOCO–TURBT tool is a feasible, valid and reliable assessment tool for measuring competency in TURBT. The tool has the potential to be used for future certification of competencies for residents and urologists. The methodology of CTA might be valuable in the development of assessment tools in other areas of clinical practice.

Available only for authorised users

Aggarwal R, Grantcharov T, Moorthy K, Milland T, Darzi A (2008) Toward feasible, valid, and reliable video-based assessments of technical surgical skills in the operating room. Ann Surg 247:372–379. https://doi.org/10.1097/SLA.0b013e318160b371 CrossRefPubMed

Zevin B, Bonrath EM, Aggarwal R, Dedy NJ, Ahmed N, Grantcharov TP, ATLAS group (2013) Development, feasibility, validity, and reliability of a scale for objective assessment of operative performance in laparoscopic gastric bypass surgery. J Am Coll Surg 216:955–965.e8. https://doi.org/10.1016/j.jamcollsurg.2013.01.003 (quiz 1029-31, 1033)CrossRefPubMed

Hampton T (2015) Efforts seek to develop systematic ways to objectively assess surgeons’ skills. JAMA 313:782–784. https://doi.org/10.1001/jama.2015.233 CrossRefPubMed

de Montbrun S, Satterthwaite L, Grantcharov TP (2016) Setting pass scores for assessment of technical performance by surgical trainees. Br J Surg 103:300–306. https://doi.org/10.1002/bjs.10047 CrossRefPubMed

Baker GR, Norton PG, Flintoft V, Blais R, Brown A, Cox J, Etchells E, Ghali WA, Hebert P, Majumdar SR, O’Beirne M, Palacios-Derflingher L, Reid RJ, Sheps S, Tamblyn R (2004) The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada. CMAJ 170:1678–1686CrossRefPubMedPubMedCentral

Vincent C, Moorthy K, Sarker SK, Chang A, Darzi AW (2004) Systems approaches to surgical quality and safety: from concept to measurement. Ann Surg 239:475–482CrossRefPubMedPubMedCentral

Miskovic D, Ni M, Wyles SM, Kennedy RH, Francis NK, Parvaiz A, Cunningham C, Rockall TA, Gudgeon AM, Coleman MG, Hanna GB, National Training Programme in Laparoscopic Colorectal Surgery in England (2013) Is competency assessment at the specialist level achievable? A study for the national training programme in laparoscopic colorectal surgery in England. Ann Surg 257:476–482. https://doi.org/10.1097/SLA.0b013e318275b72a CrossRefPubMed

Fried GM, Feldman LS (2008) Objective assessment of technical performance. World J Surg 32:156–160. https://doi.org/10.1007/s00268-007-9143-y CrossRefPubMed

Gallagher AG, Ritter EM, Satava RM (2003) Fundamental principles of validation, and reliability: rigorous science for the assessment of surgical education and training. Surg Endosc 17:1525–1529. https://doi.org/10.1007/s00464-003-0035-4 CrossRefPubMed

10.

Larson JL, Williams RG, Ketchum J, Boehler ML, Dunnington GL (2005) Feasibility, reliability and validity of an operative performance rating system for evaluating surgery residents. Surgery 138:640–647 (discussion 647–9)CrossRefPubMed

11.

Hull L, Arora S, Aggarwal R, Darzi A, Vincent C, Sevdalis N (2012) The impact of nontechnical skills on technical performance in surgery: a systematic review. J Am Coll Surg 214:214–230. https://doi.org/10.1016/j.jamcollsurg.2011.10.016 CrossRefPubMed

12.

Shepherd W, Arora KS, Abboudi H, Shamim Khan M, Dasgupta P, Ahmed K (2014) A review of the available urology skills training curricula and their validation. J Surg Educ 71:289–296. https://doi.org/10.1016/j.jsurg.2013.09.005 CrossRefPubMed

13.

Clark R, Feldon D, van Merrienboer J, Yates K (2008) Cognitive task analysis. Anonymous Handbook of research on educational communications and technology. Taylor and Francis Group, Boca Raton, pp 577–593

14.

Yates K, Feldon D (2011) Advancing the practice of cognitive task analysis: a call for taxonomic research. Theor Issues Ergon Sci 11:1464–1536

15.

Sullivan ME, Ortega A, Wasserberg N, Kaufman H, Nyquist J, Clark R (2008) Assessing the teaching of procedural skills: can cognitive task analysis add to our traditional teaching methods? Am J Surg 195:20–23CrossRefPubMed

16.

Babjuk M, Burger M, Zigeuner R, Shariat SF, van Rhijn BW, Comperat E, Sylvester RJ, Kaasinen E, Bohle A, Palou Redorta J, Roupret M, European Association of Urology (2013) EAU guidelines on non-muscle-invasive urothelial carcinoma of the bladder: update 2013. Eur Urol 64:639–653. https://doi.org/10.1016/j.eururo.2013.06.003 CrossRefPubMed

17.

Brausi M, Collette L, Kurth K, van der Meijden AP, Oosterlinck W, Witjes JA, Newling D, Bouffioux C, Sylvester RJ, EORTC Genito-Urinary Tract Cancer Collaborative Group (2002) Variability in the recurrence rate at first follow-up cystoscopy after TUR in stage Ta T1 transitional cell carcinoma of the bladder: a combined analysis of seven EORTC studies. Eur Urol 41:523–531CrossRefPubMed

18.

Allard CB, Meyer CP, Gandaglia G, Chang SL, Chun FK, Gelpi-Hammerschmidt F, Hanske J, Kibel AS, Preston MA, Trinh QD (2015) The effect of resident involvement on perioperative outcomes in transurethral urologic surgeries. J Surg Educ 72:1018–1025CrossRefPubMed

19.

Jancke G, Rosell J, Jahnson S (2014) Impact of surgical experience on recurrence and progression after transurethral resection of bladder tumour in non-muscle-invasive bladder cancer. Scand J Urol 48:276–283. https://doi.org/10.3109/21681805.2013.864327 CrossRefPubMed

20.

Barton JR, Corbett S, van der Vleuten CP, English Bowel Cancer Screening Programme, UK Joint Advisory Group for Gastrointestinal Endoscopy (2012) The validity and reliability of a Direct Observation of Procedural Skills assessment tool: assessing colonoscopic skills of senior endoscopists. Gastrointest Endosc 75:591–597. https://doi.org/10.1016/j.gie.2011.09.053 CrossRefPubMed

21.

de Vries AH, van Genugten HG, Hendrikx AJ, Koldewijn EL, Schout BM, Tjiam IM, van Merrienboer JJ, Muijtjens AM, Wagner C (2016) The Simbla TURBT simulator in urological residency training: from needs analysis to validation. J Endourol. https://doi.org/10.1089/end.2015.0723 CrossRefPubMed

22.

Luker KR, Sullivan ME, Peyre SE, Sherman R, Grunwald T (2008) The use of a cognitive task analysis-based multimedia program to teach surgical decision making in flexor tendon repair. Am J Surg 195:11–15CrossRefPubMed

23.

Crossley J, Davies H, Humphris G, Jolly B (2002) Generalisability: a key to unlock professional assessment. Med Educ 36:972–978CrossRefPubMed

24.

Fraenkel JR, Wallen NE (2009) How to design and evaluate research in education. McGraw Hill, Boston

25.

Cohen J (1988) Statistical power analysis for the behavioural sciences. Lawrence Erlbaum, London

26.

Walsh CM, Ling SC, Khanna N, Cooper MA, Grover SC, May G, Walters TD, Rabeneck L, Reznick R, Carnahan H (2014) Gastrointestinal endoscopy competency assessment tool: development of a procedure-specific assessment tool for colonoscopy. Gastrointest Endosc 79(e5):798–807. https://doi.org/10.1016/j.gie.2013.10.035 CrossRefPubMed

27.

Tjiam IM, Persoon MC, Hendrikx AJ, Muijtjens AM, Witjes JA, Scherpbier AJ (2012) Program for laparoscopic urologic skills: a newly developed and validated educational program. Urology 79:815–820. https://doi.org/10.1016/j.urology.2012.01.014 CrossRefPubMed

28.

Ten Cate O (2013) Nuts and bolts of entrustable professional activities. J Grad Med Educ 5:157–158. https://doi.org/10.4300/JGME-D-12-00380.1 CrossRefPubMedPubMedCentral

29.

Miller GE (1990) The assessment of clinical skills/competence/performance. Acad Med 65:S63-7CrossRefPubMed

30.

Ram P, van der Vleuten C, Rethans JJ, Grol R, Aretz K (1999) Assessment of practicing family physicians: comparison of observation in a multiple-station examination using standardized patients with observation of consultations in daily practice. Acad Med 74:62–69CrossRefPubMed

31.

Tjiam IM, Schout BM, Hendrikx AJ, Muijtjens AM, Scherpbier AJ, Witjes JA, Van Der Vleuten CP (2013) Program for laparoscopic urological skills assessment: setting certification standards for residents. Minim Invasive Ther Allied Technol 22:26–32. https://doi.org/10.3109/13645706.2012.686918 CrossRefPubMed

Title: Development and validation of the TOCO–TURBT tool: a summative assessment tool that measures surgical competency in transurethral resection of bladder tumour
Authors: Anna H. de Vries
Arno. M. M. Muijtjens
Hilde G. J. van Genugten
Ad. J. M. Hendrikx
Evert L. Koldewijn
Barbara M. A. Schout
Cees P. M. van der Vleuten
Cordula Wagner
Irene M. Tjiam
Jeroen J. G. van Merriënboer
Publication date: 01-12-2018
Publisher: Springer US
Published in: Surgical Endoscopy / Issue 12/2018
Print ISSN: 0930-2794
Electronic ISSN: 1432-2218
DOI: https://doi.org/10.1007/s00464-018-6251-8

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Development and validation of the TOCO–TURBT tool: a summative assessment tool that measures surgical competency in transurethral resection of bladder tumour

Abstract

Background

Methods

Results

Conclusions

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 12/2018

Does obesity impact postoperative outcomes following robotic-assisted surgery for rectal cancer?

Analysis of laparoscopic laser liver resection in standardized porcine model

Recurrence patterns after laparoscopic resection of colorectal liver metastases

Prospective study of the effect of topical application of Mitomycin C in refractory pediatric caustic esophageal strictures

Laparoscopic versus open emergent ventral hernia repair: utilization and outcomes analysis using the ACSNSQIP database

Laparoscopic transcystic common bile duct exploration: surgical indications and procedure strategies