Top

Published in:

01-12-2017

Development and validation of a tool for non-technical skills evaluation in robotic surgery—the ICARS system

Authors: Nicholas Raison, Thomas Wood, Oliver Brunckhorst, Takashige Abe, Talisa Ross, Ben Challacombe, Mohammed Shamim Khan, Giacomo Novara, Nicolo Buffi, Henk Van Der Poel, Craig McIlhenny, Prokar Dasgupta, Kamran Ahmed

Published in: Surgical Endoscopy | Issue 12/2017

Abstract

Background

Non-technical skills (NTS) are being increasingly recognised as vital for safe surgical practice. Numerous NTS rating systems have been developed to support effective training and assessment. Yet despite the additional challenges posed by robotic surgery, no NTS rating systems have been developed for this unique surgical environment. This study reports the development and validation of the first NTS behavioural rating system for robotic surgery.

Methods

A comprehensive index of all relevant NTS behaviours in robotic surgery was developed through observation of robotic theatre and interviews with robotic surgeons. Using a Delphi methodology, a panel of 16 expert surgeons was consulted to identify behaviours important to NTS assessment. These behaviours were organised into an appropriate assessment template. Experts were consulted on the feasibility, applicability and educational impact of ICARS. An observational trial was used to validate ICARS. 73 novice, intermediate and expert robotic surgeons completed a urethrovesical anastomosis within a simulated operating room. NTS were tested using four scripted scenarios of increasing difficulty. Performances were video recorded. Robotic and NTS experts assessed the videos post hoc using ICARS and the standard behavioural rating system, NOn-Technical Skills for Surgeons (NOTSS).

Results

28 key non-technical behaviours were identified by the expert panel. The finalised behavioural rating system was organised into four principle domains and seven categories. Expert opinion strongly supported its implementation. ICARS was found to be equivalent to NOTSS on Bland–Altman analysis and accurately differentiated between novice, intermediate and expert participants, p = 0.01. Moderate agreement was found between raters, Krippendorff’s alpha = 0.4. The internal structure of ICARS was shown to be consistent and reliable (median Cronbach alpha = 0.92, range 0.85–0.94).

Conclusion

ICARS is the first NTS behavioural rating system developed for robotic surgery. Initial validation has shown it to be an effective and reliable tool. Implementation of ICARS will supported structured training and assessment of NTS within the robotic surgical curriculum.

Available only for authorised users

Wood TC, Raison N, Haldar S, Brunckhorst O, McIlhenny C, Dasgupta P, Ahmed K (2016) Training tools for nontechnical skills for surgeons—a systematic review. J Surg Educ. doi:10.1016/j.jsurg.2016.11.017

Ahmed K, Jawad M, Dasgupta P, Darzi A, Athanasiou T, Khan MS (2010) Assessment and maintenance of competence in urology. Nat Rev Urol 7(7):403–413. doi:10.1038/nrurol.2010.81 CrossRefPubMed

Robertson ER, Hadi M, Morgan LJ, Pickering SP, Collins G, New S, Griffin D, McCulloch P, Catchpole KC (2014) Oxford NOTECHS II: a modified theatre team non-technical skills scoring system. PLoS ONE 9(3):e90320. doi:10.1371/journal.pone.0090320 CrossRefPubMedPubMedCentral

Undre S, Sevdalis N, Healey AN, Darzi A, Vincent CA (2007) Observational teamwork assessment for surgery (OTAS): refinement and application in urological surgery. World J Surg 31(7):1373–1381. doi:10.1007/s00268-007-9053-z CrossRefPubMed

Mishra A, Catchpole K, McCulloch P (2009) The Oxford NOTECHS System: reliability and validity of a tool for measuring teamwork behaviour in the operating theatre. Qual Saf Health Care 18(2):104–108. doi:10.1136/qshc.2007.024760 CrossRefPubMed

Yule S, Flin R, Paterson-Brown S, Maran N, Rowley D (2006) Development of a rating system for surgeons’ non-technical skills. Med Educ 40(11):1098–1104. doi:10.1111/j.1365-2929.2006.02610.x CrossRefPubMed

Michinov E, Jamet E, Dodeler V, Haegelen C, Jannin P (2014) Assessing neurosurgical non-technical skills: an exploratory study of a new behavioural marker system. J Eval Clin Pract 20(5):582–588. doi:10.1111/jep.12152 CrossRefPubMed

Flin R, Patey R (2011) Non-technical skills for anaesthetists: developing and applying ANTS. Best Pract Res Clin Anaesthesiol 25(2):215–227. doi:10.1016/j.bpa.2011.02.005 CrossRefPubMed

Mitchell L, Flin R, Yule S, Mitchell J, Coutts K, Youngson G (2013) Development of a behavioural marker system for scrub practitioners’ non-technical skills (SPLINTS system). J Eval Clin Pract 19(2):317–323. doi:10.1111/j.1365-2753.2012.01825.x CrossRefPubMed

10.

Hull L, Bicknell C, Patel K, Vyas R, Van Herzeele I, Sevdalis N, Rudarakanchana N (2016) Content validation and evaluation of an endovascular teamwork assessment tool. Eur J Vasc Endovasc Surg 52(1):11–20. doi:10.1016/j.ejvs.2015.12.044 CrossRefPubMed

11.

Fisher RA, Dasgupta P, Mottrie A, Volpe A, Khan MS, Challacombe B, Ahmed K (2015) An over-view of robot assisted surgery curricula and the status of their validation. Int J Surg 13:115–123. doi:10.1016/j.ijsu.2014.11.033 CrossRefPubMed

12.

Challacombe B, Ahmed K, Soomro N, Dasgupta P, Khan MS, Cross W, Weston R, Sanger V, Joyce A, O’Flynn K, Speakman M (2015) British Association Of Urological Surgeons(Baus)Robotic Surgery Curriculum—Guidelines For Training

13.

Mottrie A, Novara G, van der Poel H, Volpe A (2017) ERUS Robotic Curriculum. http://uroweb.org/section/erus/education/. Accessed 9 Jan 2017

14.

Brewin J, Tang J, Dasgupta P, Khan MS, Ahmed K, Bello F, Kneebone R, Jaye P (2015) Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in urology. BJU Int 116(1):156–162. doi:10.1111/bju.12875 CrossRefPubMed

15.

Bland JM, Altman DG (2007) Agreement between methods of measurement with multiple observations per individual. J Biopharm Stat 17(4):571–582. doi:10.1080/10543400701329422 CrossRefPubMed

16.

Brunckhorst O, Shahid S, Aydin A, Khan S, McIlhenny C, Brewin J, Sahai A, Bello F, Kneebone R, Shamim Khan M, Dasgupta P, Ahmed K (2015) The relationship between technical and nontechnical skills within a simulation-based ureteroscopy training environment. J Surg Educ 72(5):1039–1044. doi:10.1016/j.jsurg.2015.04.002 CrossRefPubMed

17.

McHorney CA, Tarlov AR (1995) Individual-patient monitoring in clinical practice: are available health status surveys adequate? Qual Life Res 4(4):293–307CrossRefPubMed

18.

Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33(1):159–174CrossRefPubMed

19.

Khan M, Ahmed K, Gavazzi A, Gohil R, Thomas L, Poulsen J, Ahmed M, Jaye P, Dasgupta P (2013) Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity. BJU Int 111(3):518–523. doi:10.1111/j.1464-410X.2012.11204.x CrossRef

20.

Yule S, Flin R, Paterson-Brown S, Maran N, Rowley D (2006) Development of a rating system for surgeons’ non-technical skills. Med Educ 40(11):1098–1104CrossRefPubMed

21.

American Educational Research Association, American Psychological Association, National Council on Measurement in Education, Joint Committee on Standards for Educational and Psychological Testing (U.S.) (2014) Standards for educational and psychological testing

22.

Ahmed K, Miskovic D, Darzi A, Athanasiou T, Hanna GB (2011) Observational tools for assessment of procedural skills: a systematic review. Am J Surg 202(4):469.e6–480.e6. doi:10.1016/j.amjsurg.2010.10.020 CrossRef

23.

Dasgupta P, Rha KH, Stark M (2017) Robots on the horizon. Paper presented at the American Urological Association, San Diago

Title: Development and validation of a tool for non-technical skills evaluation in robotic surgery—the ICARS system
Authors: Nicholas Raison
Thomas Wood
Oliver Brunckhorst
Takashige Abe
Talisa Ross
Ben Challacombe
Mohammed Shamim Khan
Giacomo Novara
Nicolo Buffi
Henk Van Der Poel
Craig McIlhenny
Prokar Dasgupta
Kamran Ahmed
Publication date: 01-12-2017
Publisher: Springer US
Published in: Surgical Endoscopy / Issue 12/2017
Print ISSN: 0930-2794
Electronic ISSN: 1432-2218
DOI: https://doi.org/10.1007/s00464-017-5622-x

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Development and validation of a tool for non-technical skills evaluation in robotic surgery—the ICARS system

Abstract

Background

Methods

Results

Conclusion

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 12/2017

Prediction of surgical outcomes of laparoscopic liver resections for hepatocellular carcinoma by defining surgical difficulty

Laparoscopic splenopexy for wandering spleen, a video demonstration of technique by encircling the spleen with polyglactin 910 woven mesh

Comparison of two specimen retrieval techniques in laparoscopic sleeve gastrectomy: what is the role of endobag?

Virtual reality-based assessment of basic laparoscopic skills using the Leap Motion controller

Safety and efficacy of endoscopic submucosal dissection of large (≥3 cm) subepithelial tumors located in the cardia

Imaging of pancreatic cystic lesions with confocal laser endomicroscopy: an ex vivo pilot study