Top

International Journal of Computer Assisted Radiology and Surgery

Published in:

01-11-2014 | Original Article

Vision-based online recognition of surgical activities

Authors: Michael Unger, Claire Chalopin, Thomas Neumuth

Published in: International Journal of Computer Assisted Radiology and Surgery | Issue 6/2014

Abstract

Purpose

Surgical processes are complex entities characterized by expressive models and data. Recognizable activities define each surgical process. The principal limitation of current vision-based recognition methods is inefficiency due to the large amount of information captured during a surgical procedure. To overcome this technical challenge, we introduce a surgical gesture recognition system using temperature-based recognition.

Methods

An infrared thermal camera was combined with a hierarchical temporal memory and was used during surgical procedures. The recordings were analyzed for recognition of surgical activities. The image sequence information acquired included hand temperatures. This datum was analyzed to perform gesture extraction and recognition based on heat differences between the surgeon’s warm hands and the colder background of the environment.

Results

The system was validated by simulating a functional endoscopic sinus surgery, a common type of otolaryngologic surgery. The thermal camera was directed toward the hands of the surgeon while handling different instruments. The system achieved an online recognition accuracy of 96 % with high precision and recall rates of approximately 60 %.

Conclusion

Vision-based recognition methods are the current best practice approaches for monitoring surgical processes. Problems of information overflow and extended recognition times in vision-based approaches were overcome by changing the spectral range to infrared. This change enables the real-time recognition of surgical activities and provides online monitoring information to surgical assistance systems and workflow management systems.

http://www.optris.com/.

http://www.numenta.org/.

Archer T, Macario A (2006) The drive for operating room efficiency will increase quality of patient care. Curr Opin Anaesthesiol 19:171–176. doi:10.1097/01.aco.0000192796.02797.82 PubMedCrossRef

Sutherland J, van den Heuvel W-J (2006) Towards an intelligent hospital environment: adaptive workflow in the OR of the future. Proceedings of the 39th annual Hawaii international conference on system sciences, volume 05. IEEE Computer Society, Washington, DC, USA, p 100b

Cleary K, Kinsella A, Mun SK (2005) OR 2020 workshop report: operating room of the future. Int Congr Ser 1281:832–838. doi:10.1016/j.ics.2005.03.279 CrossRef

Neumuth T, Jannin P, Schlomberg J et al (2011) Analysis of surgical intervention populations using generic surgical process models. Int J Comput Assist Radiol Surg 6:59–71. doi:10.1007/s11548-010-0475-y PubMedCrossRefPubMedCentral

Neumuth T (2013) Surgical process modeling: theory, methods, and applications

Neumuth T, Liebmann P, Wiedemann P, Meixensberger J (2012) Surgical workflow management schemata for cataract procedures. Process model-based design and validation of workflow schemata. Methods Inf Med 51:371–382. doi:10.3414/ME11-01-0093 PubMedCrossRef

Franke S, Meixensberger J, Neumuth T (2013) Intervention time prediction from surgical low-level tasks. J Biomed Inform 46:152–159. doi:10.1016/j.jbi.2012.10.002 PubMedCrossRef

Tiwari V, Dexter F, Rothman BS et al (2013) Explanation for the near-constant mean time remaining in surgical cases exceeding their estimated duration, necessary for appropriate display on electronic white boards. Anesth Analg 117:487–493. doi:10.1213/ANE.0b013e31829772e9

Epstein RH, Dexter F (2012) Mediated interruptions of anaesthesia providers using predictions of workload from anaesthesia information management system data. Anaesth Intensive Care 40:803–812PubMed

10.

Lin HC, Shafran I, Yuh D, Hager GD (2006) Towards automatic skill evaluation: detection and segmentation of robot-assisted surgical motions. Comput Aided Surg 11:220–230. doi:10.3109/10929080600989189 PubMedCrossRef

11.

Judkins TN, Oleynikov D, Stergiou N (2009) Objective evaluation of expert and novice performance during robotic surgical training tasks. Surg Endosc 23:590–597. doi:10.1007/s00464-008-9933-9 PubMedCrossRef

12.

Reiley CE, Lin HC, Varadarajan B et al (2008) Automatic recognition of surgical motions using statistical modeling for capturing variability. Stud Health Technol Inform 132:396PubMed

13.

Vankipuram M, Kahol K, Cohen T, Patel VL (2009) Visualization and analysis of activities in critical care environments. AMIA Annu Symp Proc 2009:662–666PubMedPubMedCentral

14.

Bouarfa L, Jonker PP, Dankelman J (2009) Surgical context discovery by monitoring low-level activities in the OR

15.

Neumuth T, Meissner C (2012) Online recognition of surgical instruments by information fusion. Int J Comput Assist Radiol Surg 7:297–304. doi:10.1007/s11548-011-0662-5

16.

Meißner C, Neumuth T (2012) RFID-based surgical instrument detection using Hidden Markov models. Biomed Tech (Berl). doi:10.1515/bmt-2012-4047

17.

Kranzfelder M, Zywitza D, Jell T et al (2012) Real-time monitoring for detection of retained surgical sponges and team motion in the surgical operation room using radio-frequency-identification (RFID) technology: a preclinical evaluation. J Surg Res 175:191–198. doi:10.1016/j.jss.2011.03.029 PubMedCrossRef

18.

Ahmadi S-A, Padoy N, Heining SM et al (2008) Introducing wearable accelerometers in the surgery room for activity detection. 7. Jahrestagung der Deutschen Gesellschaft fuer Computer-und Roboter-Assistierte Chirurgie (CURAC 2008)

19.

Lester J, Choudhury T, Kern N et al (2005) A hybrid discriminative/generative approach for modeling human activities. In: Proceedings of the international joint conference on artificial intelligence (IJCAI), pp 766–772

20.

Klank U, Padoy N, Feussner H, Navab N (2008) Automatic feature generation in endoscopic images. Int J CARS 3:331–339. doi:10.1007/s11548-008-0223-8 CrossRef

21.

Blum T, Feußner H, Navab N (2010) Modeling and segmentation of surgical workflow from laparoscopic video. In: Jiang T, Navab N, Pluim J, Viergever M (eds) Medical image computing and computer-assisted intervention—MICCAI 2010. Springer, Berlin, pp 400–407CrossRef

22.

Lalys F, Riffaud L, Bouget D, Jannin P (2011) An application-dependent framework for the recognition of high-level surgical tasks in the OR. In: Fichtinger G, Martel A, Peters T (eds) Medical image computing and computer-assisted intervention—MICCAI 2011. Springer, Berlin, pp 331–338CrossRef

23.

Lalys F, Bouget D, Riffaud L, Jannin P (2013) Automatic knowledge-based recognition of low-level tasks in ophthalmological procedures. Int J Comput Assist Radiol Surg 8:39–49. doi:10.1007/s11548-012-0685-6 PubMedCrossRef

24.

Haro BB, Zappella L, Vidal R (2012) Surgical gesture classification from video data. In: Ayache N, Delingette H, Golland P, Mori K (eds) Medical image computing and computer-assisted intervention—MICCAI 2012. Springer, Berlin, pp 34–41CrossRef

25.

Appenrodt J, Al-Hamadi A, Michaelis B (2010) Data gathering for gesture recognition systems based on single color-, stereo color-and thermal cameras. Int J Signal Process Image Process Pattern Recognit 3:37–50

26.

Hawkins J, George D (2006) Hierarchical temporal memory: concepts, theory, and terminology

27.

Hawkins J, Blakeslee S (2007) On intelligence. Macmillan, New York

28.

Kapuscinski T (2010) Using hierarchical temporal memory for vision-based hand shape recognition under large variations in hand’s rotation. In: Rutkowski L, Scherer R, Tadeusiewicz R et al (eds) Artificial intelligence and soft computing. Springer, Berlin, pp 272–279

29.

Kapuscinski T, Wysocki M (2009) Using hierarchical temporal memory for recognition of signed polish words. In: Kurzynski M, Wozniak M (eds) Computer recognition systems 3. Springer, Berlin, pp 355–362CrossRef

30.

Bengio Y (2009) Learning deep architectures for AI. Found Trends Mach Learn 2:1–127. doi:10.1561/2200000006 CrossRef

31.

Bouvrie J, Rosasco L, Poggio T (2009) On invariance in hierarchical models. Adv Neural Inf Process Syst 22:162–170

Title: Vision-based online recognition of surgical activities
Authors: Michael Unger
Claire Chalopin
Thomas Neumuth
Publication date: 01-11-2014
Publisher: Springer Berlin Heidelberg
Published in: International Journal of Computer Assisted Radiology and Surgery / Issue 6/2014
Print ISSN: 1861-6410
Electronic ISSN: 1861-6429
DOI: https://doi.org/10.1007/s11548-014-0994-z

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Vision-based online recognition of surgical activities

Abstract

Purpose

Methods

Results

Conclusion

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Purpose

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 6/2014

Capturing large shape variations of liver using population-based statistical shape models

Experimental evaluation of ultrasound-guided 3D needle steering in biological tissue

Optimization of breast mass classification using sequential forward floating selection (SFFS) and a support vector machine (SVM) model

Temporal bone borehole accuracy for cochlear implantation influenced by drilling strategy: an in vitro study

Reliability of sensor-based real-time workflow recognition in laparoscopic cholecystectomy

Intraoperative visualization of cerebral oxygenation using hyperspectral image data: a two-dimensional mapping method