Top

International Journal of Computer Assisted Radiology and Surgery

25-09-2024 | Expert Opinion | Original Article

Objective performance indicators versus GEARS: an opportunity for more accurate assessment of surgical skill

Authors: Marzieh Ershad Langroodi, Xi Liu, Mark R. Tousignant, Anthony M. Jarc

Published in: International Journal of Computer Assisted Radiology and Surgery

Abstract

Purpose

Surgical skill evaluation that relies on subjective scoring of surgical videos can be time-consuming and inconsistent across raters. We demonstrate differentiated opportunities for objective evaluation to improve surgeon training and performance.

Methods

Subjective evaluation was performed using the Global evaluative assessment of robotic skills (GEARS) from both expert and crowd raters; whereas, objective evaluation used objective performance indicators (OPIs) derived from da Vinci surgical systems. Classifiers were trained for each evaluation method to distinguish between surgical expertise levels. This study includes one clinical task from a case series of robotic-assisted sleeve gastrectomy procedures performed by a single surgeon, and two training tasks performed by novice and expert surgeons, i.e., surgeons with no experience in robotic-assisted surgery (RAS) and those with more than 500 RAS procedures.

Results

When comparing expert and novice skill levels, OPI-based classifier showed significantly higher accuracy than GEARS-based classifier on the more complex dissection task (OPI 0.93 ± 0.08 vs. GEARS 0.67 ± 0.18; 95% CI, 0.16–0.37; p = 0.02), but no significant difference was shown on the simpler suturing task. For the single-surgeon case series, both classifiers performed well when differentiating between early and late group cases with smaller group sizes and larger intervals between groups (OPI 0.9 ± 0.08; GEARS 0.87 ± 0.12; 95% CI, 0.02–0.04; p = 0.67). When increasing the group size to include more cases, thereby having smaller intervals between groups, OPIs demonstrated significantly higher accuracy (OPI 0.97 ± 0.06; GEARS 0.76 ± 0.07; 95% CI, 0.12–0.28; p = 0.004) in differentiating between the early/late cases.

Conclusions

Objective methods for skill evaluation in RAS outperform subjective methods when (1) differentiating expertise in a technically challenging training task, and (2) identifying more granular differences along early versus late phases of a surgeon learning curve within a clinical task. Objective methods offer an opportunity for more accessible and scalable skill evaluation in RAS.

Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer MJO (2013) Surgical skill and complication rates after bariatric surgery. N Engl J Med 369:1434–1442. https://doi.org/10.1056/nejmsa1300625CrossRefPubMed

Schlomm T, Huland H, Graefen M (2014) Improving outcome of surgical procedures is not possible without adequate quality measurement. Eur Urol. https://doi.org/10.1016/j.eururo.2013.11.042CrossRefPubMed

Tam V, Zeh HJ, Hogg ME (2017) Incorporating metrics of surgical proficiency into credentialing and privileging pathways. JAMA Surg. https://doi.org/10.1001/jamasurg.2017.0025CrossRefPubMed

Reznick RK (1993) Teaching and testing technical skills. Am J Surg 165(3):358–361. https://doi.org/10.1016/S0002-9610(05)80843-8CrossRefPubMed

Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ (2012) Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol 187:247–252. https://doi.org/10.1016/j.juro.2011.09.032CrossRefPubMed

Martin JA, Regehr G, Reznick R, Macrae H, Murnaghan J, Hutchison C, Brown M (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84:273–278. https://doi.org/10.1002/bjs.1800840237CrossRefPubMed

Hussein AA, Ghani KR, Peabody J, Sarle R, Abaza R, Eun D, Hu J, Fumo M, Lane B, Montgomery JS, Hinata N, Rooney D, Comstock B, Chan HK, Mane SS, Mohler JL, Wilding G, Miller D, Guru KA (2017) Development and validation of an objective scoring tool for robot-assisted radical prostatectomy: prostatectomy assessment and competency evaluation. J Urol 197:1237–1244. https://doi.org/10.1016/j.juro.2016.11.100CrossRefPubMed

Raza SJ, Field E, Jay C, Eun D, Fumo M, Hu JC, Lee D, Mehboob Z, Nyquist J, Peabody JO, Sarle R, Stricker H, Yang Z, Wilding G, Mohler JL, Guru KA (2015) Surgical competency for urethrovesical anastomosis during robot-assisted radical prostatectomy: development and validation of the robotic anastomosis competency evaluation. Urology 85(1):27–32. https://doi.org/10.1016/j.urology.2014.09.017CrossRefPubMed

Wang C, Han L, Stein G, Day S, Bien-Gund C, Mathews A, Ong JJ, Zhao PZ, Wei SF, Walker J, Chou R, Lee A, Chen A, Bayus B, Tucker JD (2020) Crowdsourcing in health and medical research: a systematic review. Infect Dis Poverty 9:1–9. https://doi.org/10.1186/s40249-020-0622-9CrossRef

10.

White LW, Kowalewski TM, Dockter RL, Comstock B, Hannaford B, Lendvay TS (2015) Crowd-sourced assessment of technical skill: a valid method for discriminating basic robotic surgery skills. J Endourol 29(11):1295–1301. https://doi.org/10.1089/end.2015.0191CrossRefPubMed

11.

Vernez SL, Huynh V, Osann K, Okhunov Z, Landman J, Clayman RV (2017) C-SATS: assessing surgical skills among urology residency applicants. J Endourol 31:S95–S100. https://doi.org/10.1089/end.2016.0569CrossRefPubMed

12.

Dai JC, Lendvay TS, Sorensen MD (2017) Crowdsourcing in surgical skills acquisition: a developing technology in surgical education. J Grad Med Educ 9:697–705. https://doi.org/10.4300/JGME-D-17-00322.1CrossRefPubMedPubMedCentral

13.

Polin MR, Siddiqui NY, Comstock BA, Hesham H, Brown C, Lendvay TS, Martino MA (2016) Crowdsourcing: a valid alternative to expert evaluation of robotic surgery skills. Am J Obstet Gynecol 215:644.e1-644.e7. https://doi.org/10.1016/j.ajog.2016.06.033CrossRefPubMed

14.

Rhienmora P, Haddawy P, Suebnukarn S, Dailey MN (2011) Intelligent dental training simulator with objective skill assessment and feedback. Artif Intell Med 52(2):115–121. https://doi.org/10.1016/j.artmed.2011.04.003CrossRefPubMed

15.

Jarc AM, Curet MJ (2017) Viewpoint matters: objective performance metrics for surgeon endoscope control during robot-assisted surgery. Surg Endosc 31:1192–1202. https://doi.org/10.1007/s00464-016-5090-8CrossRefPubMed

16.

Jun SK, Narayanan MS, Agarwal P, Eddib, A, Singhal, P, Garimella, S, Krovi, V (2012) Robotic minimally invasive surgical skill assessment based on automated video-analysis motion studies. In: Proceedings of the IEEE RAS and EMBS International Conference on Biomedical Robotics and Biomechatronics, 25–31 https://doi.org/10.1109/BioRob.2012.6290869

17.

Brown KC, Bhattacharyya KD, Kulason S, Zia A, Jarc A (2020) How to bring surgery to the next level: Interpretable skills assessment in robotic-assisted surgery. Visc Med 36:463–470. https://doi.org/10.1159/000512437CrossRefPubMedPubMedCentral

18.

Hung AJ, Chen J, Jarc A, Hatcher D, Djaladat H, Gill IS (2018) Development and validation of objective performance metrics for robot-assisted radical prostatectomy: a pilot study. J Urol 199:296–304. https://doi.org/10.1016/j.juro.2017.07.081CrossRefPubMed

19.

Chen AB, Liang S, Nguyen JH, Liu Y, Hung AJ (2021) Machine learning analyses of automated performance metrics during granular sub-stitch phases predict surgeon experience. Surgery 169(5):1245–1249. https://doi.org/10.1016/j.surg.2020.09.020CrossRefPubMed

20.

Cacciamani GE, Anvar A, Chen A, Gill I, Hung AJ (2021) How the use of the artificial intelligence could improve surgical skills in urology: state of the art and future perspectives. Curr Opin Urol 31(4):378–384CrossRefPubMed

21.

Hung AJ, Oh PJ, Chen J, Ghodoussipour S, Lane C, Jarc A, Gill I (2019) Experts vs super-experts: differences in automated performance metrics and clinical outcomes for robot-assisted radical prostatectomy. BJU Int 123:861–868. https://doi.org/10.1111/bju.14599CrossRefPubMed

22.

Hung A, Chen J, Che Z, Nilanon T, Jarc A, Guo L, Oh P, Gill I, Liu Y (2018) PD58-01 utilization of machine learning and automated performance metrics to evaluate robot-assisted radical prostatectomy performance and predict patient outcomes. J Urol 199(4):e1133. https://doi.org/10.1016/j.juro.2018.02.2789CrossRef

23.

Hung AJ, Ma R, Cen S, Nguyen JH, Lei X, Wagner C (2021) Surgeon automated performance metrics as predictors of early urinary continence recovery after robotic radical prostatectomy—a prospective bi-institutional study. Eur Urol Open Sci 27:65–72. https://doi.org/10.1016/j.euros.2021.03.005CrossRefPubMedPubMedCentral

24.

Ghodoussipour S, Reddy SS, Ma R, Ma R, Huang D, Nguyen J, Hung AJ (2021) An objective assessment of performance during robotic partial nephrectomy: validation and correlation of automated performance metrics with intraoperative outcomes. J Urol 205(5):1294–1302. https://doi.org/10.1097/JU.0000000000001557CrossRefPubMed

25.

Trinh L, Mingo S, Vanstrum EB, Sanford DI, Ma R, Nguyen JH, Liu Y, Hung AJ (2022) Survival Analysis using surgeon skill metrics and patient factors to predict urinary continence recovery after robot-assisted radical prostatectomy. Eur Urol Focus 8(2):623–630. https://doi.org/10.1016/j.euf.2021.04.001CrossRefPubMed

26.

Tang B, Hanna GB, Joice P, Cuschieri A (2004) Identification and categorization of technical errors by observational clinical human reliability assessment (OCHRA) during laparoscopic cholecystectomy. Arch Surg 139(11):1215–1220CrossRefPubMed

27.

Ahmed K, Miskovic D, Darzi A, Athanasiou T, Hanna GB (2011) Observational tools for assessment of procedural skills: a systematic review. Am J Surg 202:469-480.e6CrossRefPubMed

28.

Biggs J (1996) Enhancing teaching through constructive alignment. High Educ 32(3):347–364CrossRef

29.

Nazari T, Bogomolova K, Ridderbos M, Dankbaar MEW, van Merriënboer JJG, Lange JF, Wiggers T, Wiggers JA (2021) Global versus task-specific postoperative feedback in surgical procedure learning. Surgery 170:81–87. https://doi.org/10.1016/j.surg.2020.12.038CrossRefPubMed

30.

Lavanchy JL, Zindel J, Kirtac K, Twick I, Hosgor E, Candinas D, Beldi G (2021) Automation of surgical skill assessment using a three-stage machine learning algorithm. Sci Rep 11:1–9. https://doi.org/10.1038/s41598-021-84295-6CrossRef

31.

Zia A, Hung A, Essa I, Jarc A (2018) Surgical activity recognition in robot-assisted radical prostatectomy using deep learning. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

32.

Hashimoto DA, Rosman G, Witkowski ER, Stafford C, Navarette-Welton AJ, Rattner DW, Lillemoe KD, Rus DL, Meireles OR (2019) Computer vision analysis of intraoperative video: automated recognition of operative steps in laparoscopic sleeve gastrectomy. Ann Surg 270(3):414–421. https://doi.org/10.1097/SLA.0000000000003460CrossRefPubMed

33.

Close MF, Mehta CH, Liu Y, Isaac MJ, Costello MS, Kulbarsh KD, Meyer TA (2020) Subjective vs computerized assessment of surgeon skill level during mastoidectomy. Otolaryngol-Head Neck Surg 163:1255–1257. https://doi.org/10.1177/0194599820933882CrossRefPubMedPubMedCentral

34.

Datta V, Chang A, Mackay S, Darzi A (2002) The relationship between motion analysis and surgical technical assessments. Am J Surg 184:70–73. https://doi.org/10.1016/S0002-9610(02)00891-7CrossRefPubMed

35.

Aggarwal R, Grantcharov T, Moorthy K, Milland T, Papasavas P, Dosis A, Bello F, Darzi A (2007) An evaluation of the feasibility, validity, and reliability of laparoscopic skills assessment in the operating room. Ann Surg 245:992–999. https://doi.org/10.1097/01.sla.0000262780.17950.e5CrossRefPubMedPubMedCentral

36.

van Empel PJ, van Rijssen LB, Commandeur JP, Verdam MGE, Huirne JA, Scheele F, Bonjer HJ, Meijerink WJ (2013) Objective versus subjective assessment of laparoscopic skill. ISRN Minim Invasive Surg 2013:1–5. https://doi.org/10.1155/2013/686494CrossRef

37.

Van Empel PJ, Van Rijssen LB, Commandeur JP, Verdam MGE, Huirne JA, Scheele F, Bonjer HJ, Jeroen Meijerin W (2012) Validation of a new box trainer-related tracking device: the TrEndo. Surg Endosc 26:2346–2352. https://doi.org/10.1007/s00464-012-2187-6CrossRefPubMedPubMedCentral

38.

Lefor AK, Harada K, Dosis A, Mitsuishi M (2020) Motion analysis of the JHU-ISI gesture and skill assessment working set using robotics video and motion assessment software. Int J Comput Assist Radiol Surg 15:2017–2025. https://doi.org/10.1007/s11548-020-02259-zCrossRefPubMedPubMedCentral

39.

Tousignant MR, Liu X, Ershad Langroodi M, Jarc AM (2022) Identification of main influencers of surgical efficiency and variability using task-level objective metrics: a five-year robotic sleeve gastrectomy case series. Front Surg 9:756522. https://doi.org/10.3389/fsurg.2022.756522CrossRefPubMedPubMedCentral

40.

Ma R, Ramaswamy A, Xu J, Trinh L, Kiyasseh D, Chu TN, Wong EY, Lee RS, Rodriguez I, DeMeo G, Desai A (2022) Surgical gestures as a method to quantify surgical performance and predict patient outcomes. NPJ Digit Med. 5(1):187. https://doi.org/10.1038/s41746-022-00738-yCrossRefPubMedPubMedCentral

Title: Objective performance indicators versus GEARS: an opportunity for more accurate assessment of surgical skill
Authors: Marzieh Ershad Langroodi
Xi Liu
Mark R. Tousignant
Anthony M. Jarc
Publication date: 25-09-2024
Publisher: Springer International Publishing
Keywords: Expert Opinion
Expert Opinion
Published in: International Journal of Computer Assisted Radiology and Surgery
Print ISSN: 1861-6410
Electronic ISSN: 1861-6429
DOI: https://doi.org/10.1007/s11548-024-03248-2

Keynote webinar | Spotlight on modern management of frailty

Springer Medicine

Objective performance indicators versus GEARS: an opportunity for more accurate assessment of surgical skill

Abstract

Purpose

Methods

Results

Conclusions

Keynote webinar | Spotlight on modern management of frailty

Springer Medicine

Abstract

Purpose

Methods

Results

Conclusions

Please log in to get access to this content