Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
Surgical Endoscopy
Published in: Surgical Endoscopy 9/2023

14-08-2023 | Pancreatic Surgery | Dynamic Manuscript

Application and evaluation of surgical tool and tool tip recognition based on Convolutional Neural Network in multiple endoscopic surgical scenarios

Authors: Lu Ping, Zhihong Wang, Jingjing Yao, Junyi Gao, Sen Yang, Jiayi Li, Jile Shi, Wenming Wu, Surong Hua, Huizhen Wang

Published in: Surgical Endoscopy | Issue 9/2023

Login to get access

Abstract

Background

In recent years, computer-assisted intervention and robot-assisted surgery are receiving increasing attention. The need for real-time identification and tracking of surgical tools and tool tips is constantly demanding. A series of researches focusing on surgical tool tracking and identification have been performed. However, the size of dataset, the sensitivity/precision, and the response time of these studies were limited. In this work, we developed and utilized an automated method based on Convolutional Neural Network (CNN) and You Only Look Once (YOLO) v3 algorithm to locate and identify surgical tools and tool tips covering five different surgical scenarios.

Materials and methods

An algorithm of object detection was applied to identify and locate the surgical tools and tool tips. DarkNet-19 was used as Backbone Network and YOLOv3 was modified and applied for the detection. We included a series of 181 endoscopy videos covering 5 different surgical scenarios: pancreatic surgery, thyroid surgery, colon surgery, gastric surgery, and external scenes. A total amount of 25,333 images containing 94,463 targets were collected. Training and test sets were divided in a proportion of 2.5:1. The data sets were openly stored at the Kaggle database.

Results

Under an Intersection over Union threshold of 0.5, the overall sensitivity and precision rate of the model were 93.02% and 89.61% for tool recognition and 87.05% and 83.57% for tool tip recognition, respectively. The model demonstrated the highest tool and tool tip recognition sensitivity and precision rate under external scenes. Among the four different internal surgical scenes, the network had better performances in pancreatic and colon surgeries and poorer performances in gastric and thyroid surgeries.

Conclusion

We developed a surgical tool and tool tip recognition model based on CNN and YOLOv3. Validation of our model demonstrated satisfactory precision, accuracy, and robustness across different surgical scenes.
Appendix
Available only for authorised users
Literature
1.
Baumhauer M, Feuerstein M, Meinzer HP, Rassweiler J (2008) Navigation in endoscopic soft tissue surgery: perspectives and limitations. J Endourol 22:751–766CrossRefPubMed
2.
Hua S, Gao J, Wang Z, Yeerkenbieke P, Li J, Wang J, He G, Jiang J, Lu Y, Yu Q, Han X, Liao Q, Wu W (2022) Automatic bleeding detection in laparoscopic surgery based on a faster region-based convolutional neural network. Ann Transl Med 10:546CrossRefPubMedPubMedCentral
3.
Bouget D, Allan M, Stoyanov D, Jannin P (2017) Vision-based and marker-less surgical tool detection and tracking: a review of the literature. Med Image Anal 35:633–654CrossRefPubMed
4.
Kanakatte A, Ramaswamy A, Gubbi J, Ghose A, Purushothaman B (2020) Surgical tool segmentation and localization using spatio-temporal deep network. Annu Int Conf IEEE Eng Med Biol Soc 2020:1658–1661PubMed
5.
Jo K, Choi Y, Choi J, Chung JW (2019) Robust real-time detection of laparoscopic instruments in robot surgery using convolutional neural networks with motion vector prediction. Appl Sci 9:2865CrossRef
6.
Ramesh A, Beniwal M, Uppar AM, Madhav Rao VV (2021) Microsurgical tool detection and characterization in intra-operative neurosurgical videos. Annu Int Conf IEEE Eng Med Biol Soc 2021:2676–2681PubMed
7.
Leppänen T, Vrzakova H, Bednarik R, Kanervisto A, Elomaa A-P, Huotarinen A, Bartczak P, Fraunberg M, Jääskeläinen JE (2018) Augmenting microsurgical training: microsurgical instrument detection using convolutional neural networks. In: 2018 IEEE 31st international symposium on computer-based medical systems (CBMS), IEEE, Karlstad, Sweden, pp 211–216
8.
Sarikaya D, Corso JJ, Guru KA (2017) Detection and localization of robotic tools in robot-assisted surgery videos using deep neural networks for region proposal and detection. IEEE Trans Med Imaging 36:1542–1549CrossRefPubMed
9.
Twinanda AP, Shehata S, Mutter D, Marescaux J, De Mathelin M, Padoy N (2016) Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging 36:86–97CrossRefPubMed
10.
Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, pp 779–788
11.
Choi B, Jo K, Choi S, Choi J (2017) Surgical-tools detection based on Convolutional Neural Network in laparoscopic robot-assisted surgery. In: 2017 39th annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, Jeju, South Korea, pp 1756–1759
12.
Hua S, Wang Z, Gao J, Wang J, He G, Han X, Chen G, Lia Q (2022) Exploration of deep learning to identify recurrent laryngeal nerve in endoscopic thyroidectomy via unilateral axillary approach. Chin J Endocr Surg 16:7
13.
Hua S, Wang Z, Li J, Gao J, Wang J, He G, Yeerkenbieke P, Han X, Chen G, Liao Q (2022) Application of deep learning to identify recurrent laryngeal nerve in endoscopic thyroidectomy via breast approach. Chin J Endocr Surg 16:6
14.
Hua S, Wang Z, Wang J, He G, Gao J, Yu Q, Han X, Liao Q, Wu W (2021) Application value of machine learning algorithms for gauze detection in laparoscopic pancreatic surgery. Chin J Digest Surg 12:1324–1330
15.
Zhang L, Ye M, Chan PL, Yang GZ (2017) Real-time surgical tool tracking and pose estimation using a hybrid cylindrical marker. Int J Comput Assist Radiol Surg 12:921–930CrossRefPubMedPubMedCentral
16.
Kranzfelder M, Schneider A, Fiolka A, Schwan E, Gillen S, Wilhelm D, Schirren R, Reiser S, Jensen B, Feussner H (2013) Real-time instrument detection in minimally invasive surgery using radiofrequency identification technology. J Surg Res 185:704–710CrossRefPubMed
17.
Koskinen J, Torkamani-Azar M, Hussein A, Huotarinen A, Bednarik R (2022) Automated tool detection with deep learning for monitoring kinematics and eye-hand coordination in microsurgery. Comput Biol Med 141:105121CrossRefPubMed
18.
Al Hajj H, Lamard M, Cochener B, Quellec G (2017) Smart data augmentation for surgical tool detection on the surgical tray. Annu Int Conf IEEE Eng Med Biol Soc 2017:4407–4410PubMed
19.
20.
Kitaguchi D, Takeshita N, Matsuzaki H, Oda T, Watanabe M, Mori K, Kobayashi E, Ito M (2020) Automated laparoscopic colorectal surgery workflow recognition using artificial intelligence: experimental research. Int J Surg 79:88–94CrossRefPubMed
21.
Choi J, Cho S, Chung JW, Kim N (2021) Video recognition of simple mastoidectomy using convolutional neural networks: detection and segmentation of surgical tools and anatomical regions. Comput Methods Programs Biomed 208:106251CrossRefPubMed
22.
Pose Díez de la Lastra A, García-Duarte Sáenz L, García-Mato D, Hernández-Álvarez L, Ochandiano S, Pascau J (2021) Real-time tool detection for workflow identification in open cranial vault remodeling. Entropy (Basel) 23(7):817CrossRefPubMed
Metadata
Title
Application and evaluation of surgical tool and tool tip recognition based on Convolutional Neural Network in multiple endoscopic surgical scenarios
Authors
Lu Ping
Zhihong Wang
Jingjing Yao
Junyi Gao
Sen Yang
Jiayi Li
Jile Shi
Wenming Wu
Surong Hua
Huizhen Wang
Publication date
14-08-2023
Publisher
Springer US
Published in
Surgical Endoscopy / Issue 9/2023
Print ISSN: 0930-2794
Electronic ISSN: 1432-2218
DOI
https://doi.org/10.1007/s00464-023-10323-3

Other articles of this Issue 9/2023

Surgical Endoscopy 9/2023 Go to the issue

2023 SAGES Oral

Not deep enough: Modeling the effect of shallow placement of the DaVinci Xi “bariatric” long trocar on the muscular abdominal wall

OriginalPaper

Endoscopic ultrasound-guided versus percutaneous drainage for the management of post-operative fluid collections after distal pancreatectomy

OriginalPaper

Antimigration versus conventional fully covered metal stents in the endoscopic treatment of anastomotic biliary strictures after deceased-donor liver transplantation

Original Article

Postoperative outcomes for sleeve gastrectomy patients with positive pH-defined GERD

2023 SAGES Poster

Gastric per-oral endoscopic myotomy versus pyloric injection of botulinum toxin for the treatment of gastroparesis: our institutional experience and a systematic review of the literature

OriginalPaper

Single-staged retromuscular abdominal wall reconstruction with mesh at the time of ostomy reversal: are we crossing the line? An ACHQC Analysis