Top

International Journal of Computer Assisted Radiology and Surgery

Published in:

Open Access 01-12-2020 | Original Article

Motion analysis of the JHU-ISI Gesture and Skill Assessment Working Set using Robotics Video and Motion Assessment Software

Authors: Alan Kawarai Lefor, Kanako Harada, Aristotelis Dosis, Mamoru Mitsuishi

Published in: International Journal of Computer Assisted Radiology and Surgery | Issue 12/2020

Abstract

Purpose

The JIGSAWS dataset is a fixed dataset of robot-assisted surgery kinematic data used to develop predictive models of skill. The purpose of this study is to analyze the relationships of self-defined skill level with global rating scale scores and kinematic data (time, path length and movements) from three exercises (suturing, knot-tying and needle passing) (right and left hands) in the JIGSAWS dataset.

Methods

Global rating scale scores are reported in the JIGSAWS dataset and kinematic data were calculated using ROVIMAS software. Self-defined skill levels are in the dataset (novice, intermediate, expert). Correlation coefficients (global rating scale-skill level and global rating scale-kinematic parameters) were calculated. Kinematic parameters were compared among skill levels.

Results

Global rating scale scores correlated with skill in the knot-tying exercise (r = 0.55, p = 0.0005). In the suturing exercise, time, path length (left) and movements (left) were significantly different (p < 0.05) for novices and experts. For knot-tying, time, path length (right and left) and movements (right) differed significantly for novices and experts. For needle passing, no kinematic parameter was significantly different comparing novices and experts. The only kinematic parameter that correlated with global rating scale scores is time in the knot-tying exercise.

Conclusion

Global rating scale scores weakly correlate with skill level and kinematic parameters. The ability of kinematic parameters to differentiate among self-defined skill levels is inconsistent. Additional data are needed to enhance the dataset and facilitate subset analyses and future model development.

Kohn LT, Corrigan JM, Donaldson MS (eds) (2000) To err is human: building a safer health system. Institute of Medicine (US) Committee on Quality of Health Care in America. National Academies Press, Washington, DC

Zevin B, Levy JS, Satava RM, Grantcharov TP (2012) A consensus-based framework for design, validation, and implementation of simulation-based training curricula in surgery. J Am Coll Surg 215(4):580–586.e3CrossRef

Reiley CE, Lin HC, Yuh DD, Hager GD (2011) Review of methods for objective surgical skill evaluation. Surg Endosc 25(2):356–366. https://doi.org/10.1007/s00464-010-1190-z (Epub 2010 Jul 7)CrossRefPubMed

Moorthy K, Munz Y, Sarker SK, Darzi A (2003) Objective assessment of technical skills in surgery. BMJ 327(7422):1032–1037CrossRef

Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, Brown M (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84(2):273–278

Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ (2012) Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol 187(1):247–252. https://doi.org/10.1016/j.juro.2011.09.032 (Epub 2011 Nov 17)CrossRefPubMed

van Hove PD, Tuijthof GJ, Verdaasdonk EG, Stassen LP, Dankelman J (2010) Objective assessment of technical surgical skills. Br J Surg 97(7):972–987. https://doi.org/10.1002/bjs.7115CrossRefPubMed

Satava RM, Stefanidis D, Levy JS, Smith R, Martin JR, Monfared S, Timsina LR, Darzi AW, Moglia A, Brand TC, Dorin RP, Dumon KR, Francone TD, Georgiou E, Goh AC, Marcet JE, Martino MA, Sudan R, Vale J, Gallagher AG (2019) Proving the effectiveness of the fundamentals of robotic surgery (FRS) skills curriculum: a single-blinded, multispecialty, multi-institutional randomized control trial. Ann Surg. https://doi.org/10.1097/SLA.0000000000003220CrossRefPubMed

Uemura M, Tomikawa M, Kumashiro R, Miao T, Souzaki R, Ieiri S, Ohuchida K, Lefor AT, Hashizume M (2014) Analysis of hand motion differentiates expert and novice surgeons. J Surg Res 188(1):8–13. https://doi.org/10.1016/j.jss.2013.12.009 (Epub 2013 Dec 19)CrossRefPubMed

10.

Uemura M, Tomikawa M, Miao T, Souzaki R, Ieiri S, Akahoshi T, Lefor AK, Hashizume M (2018) Feasibility of an AI-based measure of the hand motions of expert and novice surgeons. Comput Math Methods Med 2018:9873273. https://doi.org/10.1155/2018/9873273 (eCollection 2018)CrossRefPubMedPubMedCentral

11.

Dosis A, Bello F, Rockall T, Munz Y, Moorthy K, Martin S, Darzi A (2003) ROVIMAS: a software package for assessing surgical skills using the da Vinci telemanipulator system. Paper presented at: the fourth international conference of information technology (ITAB 2003); April 24–27, Birmingham, England

12.

Aggarwal R, Grantcharov T, Moorthy K, Milland T, Papasavas P, Dosis A, Bello F, Darzi A (2007) An evaluation of the feasibility, validity, and reliability of laparoscopic skills assessment in the operating room. Ann Surg 245(6):992–999CrossRef

13.

Dosis A, Aggarwal R, Bello F, Moorthy K, Munz Y, Gillies D (2005) Darzi a synchronized video and motion analysis for the assessment of procedures in the operating theater. Arch Surg 140(3):293–299CrossRef

14.

Mason JD, Ansell J, Warren N (2013) Torkington is motion analysis a valid tool for assessing laparoscopic skill? J. Surg Endosc 27(5):1468–1477. https://doi.org/10.1007/s00464-012-2631-7 (Epub 2012 Dec 12)CrossRef

15.

Hung AJ, Chen J, Jarc A, Hatcher D, Djaladat H, Gill IS (2018) Development and validation of objective performance metrics for robot-assisted radical prostatectomy: a pilot study. J Urol 199(1):296–304. https://doi.org/10.1016/j.juro.2017.07.081 (Epub 2017 Jul 29)CrossRefPubMed

16.

Fard MJ, Ameri S, Darin Ellis R, Chinnam RB, Pandya AK, Klein MD (2018) Automated robot-assisted surgical skill evaluation: predictive analytics approach. Int J Med Robot. https://doi.org/10.1002/rcs.1850 (Epub 2017 Jun 29)CrossRefPubMed

17.

Dosis A (2005) Modeling and assessment of surgical dexterity in laparoscopic and robotically assisted surgery using synchronized video-motion analysis and hidden Markov models. Dissertation. Imperial College London, University of London

18.

Judkins TN, Oleynikov D, Stergiou N (2009) Objective evaluation of expert and novice performance during robotic surgical training tasks. Surg Endosc 23(3):590–597. https://doi.org/10.1007/s00464-008-9933-9 (Epub 2008 Apr 29)CrossRefPubMed

19.

DiMaio S, Hasser C (2008) The da Vinci research interface. In: MICCAI workshop on systems and arch. for computer assisted interventions. Midas Journal. http://hdl.handle.net/10380/1464

20.

Moorthy K, Munz Y, Dosis A, Hernandez J, Martin S, Bello F, Rockall T, Darzi A (2004) Dexterity enhancement with robotic surgery. Surg Endosc 18(5):790–795 (Epub 2004 Apr 6)CrossRef

21.

https://www.sages.org/meetings/annual-meeting/abstracts-archive/surgtrak-affordable-motion-tracking-and-video-capture-for-the-da-vinci-surgical-robot/. Accessed 27 Aug 2019

22.

Gao Y, Vedula SS, Reiley CE, Ahmidi N, Varadarajan B, Lin HC, Tao L, Zappella L, Béjar B, Yuh DD, Chen CCG, Vidal R, Khudanpur S, Hager GD (2014) JHU-ISI gesture and skill assessment working set (jigsaws): a surgical activity dataset for human motion modeling. In: MICCAI workshop: M2CAI, vol 3

23.

Forestier G, Petitjean F, Senin P, Despinoy F, Huaulmé A, Fawaz HI, Weber J, Idoumghar L, Muller PA, Jannin P (2018) Surgical motion analysis using discriminative interpretable patterns. Artif Intell Med 91:3–11. https://doi.org/10.1016/j.artmed.2018.08.002 (Epub 2018 Aug 30)CrossRefPubMed

24.

Funke I, Mees ST, Weitz J, Speidel S (2019) Video-based surgical skill assessment using 3D convolutional neural networks. Int J Comput Assist Radiol Surg 14(7):1217–1225. https://doi.org/10.1007/s11548-019-01995-1 (Epub 2019 May 18)CrossRefPubMed

25.

Zia A (2018) Essa Automated surgical skill assessment in RMIS training. I. Int J Comput Assist Radiol Surg 13(5):731–739. https://doi.org/10.1007/s11548-018-1735-5 (Epub 2018 Mar 16)CrossRefPubMed

26.

Ahmidi N, Tao L, Sefati S, Gao Y, Lea C, Haro BB, Zappella L, Khudanpur S, Vidal R, Hager GD (2017) A dataset and benchmarks for segmentation and recognition of gestures in robotic surgery. IEEE Trans Biomed Eng 64(9):2025–2041. https://doi.org/10.1109/TBME.2016.2647680 (Epub 2017 Jan 4)CrossRefPubMedPubMedCentral

27.

Wang Z, MajewiczFey A (2018) Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery. Int J Comput Assist Radiol Surg 13(12):1959–1970. https://doi.org/10.1007/s11548-018-1860-1 (Epub 2018 Sep 25)CrossRefPubMed

28.

Ismail Fawaz H, Forestier G, Weber J, Idoumghar L, Muller PA (2019) Accurate and interpretable evaluation of surgical skills from kinematic data using fully convolutional neural networks. Int J Comput Assist Radiol Surg. https://doi.org/10.1007/s11548-019-02039-4CrossRefPubMed

29.

Nguyen XA, Ljuhar D, Pacilli M, Nataraja RM, Chauhan S (2019) Surgical skill levels: classification and analysis using deep neural network model and motion signals. Comput Methods Programs Biomed 177:1–8. https://doi.org/10.1016/j.cmpb.2019.05.008 (Epub 2019 May 13)CrossRefPubMed

30.

https://cirl.lcsr.jhu.edu/research/hmm/datasets/jigsaws_release/. Accessed 10 July 2019

31.

Datta V, Chang A, Mackay S, Darzi A (2002) The relationship between motion analysis and surgical technical assessments. Am J Surg 184(1):70–73CrossRef

32.

https://www.socscistatistics.com/tests/spearman/default2.aspx. Accessed 2 Sept 2019

33.

https://www.dummies.com/education/math/statistics/how-to-interpret-a-correlation-coefficient-r/. Accessed 29 Aug 2019

34.

Mackay S, Datta V, Mandalia M, Bassett P, Darzi A (2002) Electromagnetic motion analysis in the assessment of surgical skill: relationship between time and movement. ANZ J Surg 72(9):632–642CrossRef

35.

Corvetto MA, Fuentes C, Araneda A, Achurra P, Miranda P, Viviani P, Altermatt FR (2017) Validation of the imperial college surgical assessment device for spinal anesthesia. BMC Anesthesiol 17(1):131. https://doi.org/10.1186/s12871-017-0422-3CrossRefPubMedPubMedCentral

36.

Fard MJ, Ameri S, Chinnam RB, Pandya AK, Klein MD, Ellis RD (2016) Machine learning approach for skill evaluation in robotic-assisted surgery. In: Lecture notes in engineering and computer science: proceedings of the world congress on engineering and computer science, San Francisco. http://arxiv.org/abs/1611.05136

37.

Moglia A, Ferrari V, Morelli L, Ferrari M, Mosca F, Cuschieri A (2016) A systematic review of virtual reality simulators for robot-assisted surgery. Eur Urol 69(6):1065–1080. https://doi.org/10.1016/j.eururo.2015.09.021 (Epub 2015 Oct 1)CrossRefPubMed

Title: Motion analysis of the JHU-ISI Gesture and Skill Assessment Working Set using Robotics Video and Motion Assessment Software
Authors: Alan Kawarai Lefor
Kanako Harada
Aristotelis Dosis
Mamoru Mitsuishi
Publication date: 01-12-2020
Publisher: Springer International Publishing
Published in: International Journal of Computer Assisted Radiology and Surgery / Issue 12/2020
Print ISSN: 1861-6410
Electronic ISSN: 1861-6429
DOI: https://doi.org/10.1007/s11548-020-02259-z

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Motion analysis of the JHU-ISI Gesture and Skill Assessment Working Set using Robotics Video and Motion Assessment Software

Abstract

Purpose

Methods

Results

Conclusion

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Purpose

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 12/2020

Enhanced registration of ultrasound volumes by segmentation of resection cavity in neurosurgical procedures

Assessing and mitigating the effects of class imbalance in machine learning with application to X-ray imaging

Robust endocytoscopic image classification based on higher-order symmetric tensor analysis and multi-scale topological statistics

Value of the surgeon’s sightline on hologram registration and targeting in mixed reality

Vision-based hand–eye calibration for robot-assisted minimally invasive surgery

Estimating 3-dimensional liver motion using deep learning and 2-dimensional ultrasound images