Top

Published in:

15-03-2024 | Pancreaticojejunostomy

The development of a deep learning model for automated segmentation of the robotic pancreaticojejunostomy

Authors: Amr I. Al Abbas, Babak Namazi, Imad Radi, Rodrigo Alterio, Andres A. Abreu, Benjamin Rail, Patricio M. Polanco, Herbert J. Zeh III, Melissa E. Hogg, Amer H. Zureikat, Ganesh Sankaranarayanan

Published in: Surgical Endoscopy | Issue 5/2024

Abstract

Background

Minimally invasive surgery provides an unprecedented opportunity to review video for assessing surgical performance. Surgical video analysis is time-consuming and expensive. Deep learning provides an alternative for analysis. Robotic pancreaticoduodenectomy (RPD) is a complex and morbid operation. Surgeon technical performance of pancreaticojejunostomy (PJ) has been associated with postoperative pancreatic fistula. In this work, we aimed to utilize deep learning to automatically segment PJ RPD videos.

Methods

This was a retrospective review of prospectively collected videos from 2011 to 2022 that were in libraries at tertiary referral centers, including 111 PJ videos. Each frame of a robotic PJ video was categorized based on 6 tasks. A 3D convolutional neural network was trained for frame-level visual feature extraction and classification. All the videos were manually annotated for the start and end of each task.

Results

Of the 100 videos assessed, 60 videos were used for the training the model, 10 for hyperparameter optimization, and 30 for the testing of performance. All the frames were extracted (6 frames/second) and annotated. The accuracy and mean per-class F1 scores were 88.01% and 85.34% for tasks.

Conclusion

The deep learning model performed well for automated segmentation of PJ videos. Future work will focus on skills assessment and outcome prediction.

Gietelink L, Wouters MW, Bemelman WA, Dekker JW, Tollenaar RA, Tanis PJ et al (2016) Reduced 30-day mortality after laparoscopic colorectal cancer surgery: a population based study from the Dutch Surgical Colorectal Audit (DSCA). Ann Surg 264(1):135–140CrossRefPubMed

Semm K (1988) Pelviscopic appendectomy. Dtsch Med Wochenschr 113(1):3–5CrossRefPubMed

Aquina CT, Probst CP, Becerra AZ, Iannuzzi JC, Hensley BJ, Noyes K et al (2016) Missed opportunity: laparoscopic colorectal resection is associated with lower incidence of small bowel obstruction compared to an open approach. Ann Surg 264(1):127–134CrossRefPubMed

McCormack K, Scott NW, Go PM, Ross S, Grant AM (2003) Laparoscopic techniques versus open techniques for inguinal hernia repair. Cochrane Database Syst Rev 2003(1):CD001785PubMedPubMedCentral

Reynolds W Jr (2001) The first laparoscopic cholecystectomy. J Soc Laparoendosc Surg 5(1):89–94

Shaligram A, Unnirevi J, Simorov A, Kothari VM, Oleynikov D (2012) How does the robot affect outcomes? A retrospective review of open, laparoscopic, and robotic Heller myotomy for achalasia. Surg Endosc 26(4):1047–1050CrossRefPubMed

Zureikat AH, Postlewait LM, Liu Y, Gillespie TW, Weber SM, Abbott DE et al (2016) A multi-institutional comparison of perioperative outcomes of robotic and open pancreaticoduodenectomy. Ann Surg 264(4):640–649CrossRefPubMed

Cameron JL, He J (2015) Two thousand consecutive pancreaticoduodenectomies. J Am Coll Surg 220(4):530–536CrossRefPubMed

Cameron JL, Riall TS, Coleman J, Belcher KA (2006) One thousand consecutive pancreaticoduodenectomies. Ann Surg 244(1):10–15CrossRefPubMedPubMedCentral

10.

Al Abbas AI, Borrebach JD, Pitt HA, Bellon J, Hogg ME, Zeh HJ 3rd et al (2021) Development of a novel pancreatoduodenectomy-specific risk calculator: an analysis of 10,000 patients. J Gastrointest Surg 25(6):1503–1511CrossRefPubMed

11.

Cuschieri A (1994) Laparoscopic surgery of the pancreas. J R Coll Surg Edinb 39(3):178–184PubMed

12.

Gagner M, Pomp A (1994) Laparoscopic pylorus-preserving pancreatoduodenectomy. Surg Endosc 8(5):408–410CrossRefPubMed

13.

Kendrick ML, Cusati D (2010) Total laparoscopic pancreaticoduodenectomy: feasibility and outcome in an early experience. Arch Surg 145(1):19–23CrossRefPubMed

14.

Palanivelu C, Rajan PS, Rangarajan M, Vaithiswaran V, Senthilnathan P, Parthasarathi R et al (2009) Evolution in techniques of laparoscopic pancreaticoduodenectomy: a decade long experience from a tertiary center. J Hepatobiliary Pancreat Surg 16(6):731–740CrossRefPubMed

15.

Zeh HJ 3rd, Bartlett DL, Moser AJ (2011) Robotic-assisted major pancreatic resection. Adv Surg 45:323–340CrossRefPubMed

16.

Zureikat AH, Beane JD, Zenati MS, Al Abbas AI, Boone BA, Moser AJ et al (2021) 500 minimally invasive robotic pancreatoduodenectomies: one decade of optimizing performance. Ann Surg 273(5):966–972CrossRefPubMed

17.

Nassour I, Winters SB, Hoehn R, Tohme S, Adam MA, Bartlett DL et al (2020) Long-term oncologic outcomes of robotic and open pancreatectomy in a national cohort of pancreatic adenocarcinoma. J Surg Oncol 122(2):234–242CrossRefPubMed

18.

Zeh HJ, Zureikat AH, Secrest A, Dauoudi M, Bartlett D, Moser AJ (2012) Outcomes after robot-assisted pancreaticoduodenectomy for periampullary lesions. Ann Surg Oncol 19(3):864–870CrossRefPubMed

19.

Al Abbas AI, Jung JP, Rice MK, Zureikat AH, Zeh HJ 3rd, Hogg ME (2019) Methodology for developing an educational and research video library in minimally invasive surgery. J Surg Educ 76(3):745–755CrossRefPubMed

20.

Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Nunn AR et al (2013) Surgical skill and complication rates after bariatric surgery. N Engl J Med 369(15):1434–1442CrossRefPubMed

21.

Brown JA, Jung JP, Zenati MS, Simmons RL, Al Abbas AI, Hogg ME et al (2021) Video review reveals technical factors predictive of biliary stricture and cholangitis after robotic pancreaticoduodenectomy. HPB 23(1):144–153CrossRefPubMed

22.

Hogg ME, Zenati M, Novak S, Chen Y, Jun Y, Steve J et al (2016) Grading of surgeon technical performance predicts postoperative pancreatic fistula for pancreaticoduodenectomy independent of patient-related variables. Ann Surg 264(3):482–491CrossRefPubMed

23.

Jung JP, Zenati MS, Dhir M, Zureikat AH, Zeh HJ, Simmons RL et al (2018) Use of video review to investigate technical factors that may be associated with delayed gastric emptying after pancreaticoduodenectomy. JAMA Surg 153(10):918–927CrossRefPubMedPubMedCentral

24.

Rice MK, Zenati MS, Novak SM, Al Abbas AI, Zureikat AH, Zeh HJ 3rd et al (2019) Crowdsourced assessment of inanimate biotissue drills: a valid and cost-effective way to evaluate surgical trainees. J Surg Educ 76(3):814–823CrossRefPubMed

25.

Hashimoto DA, Rosman G, Rus D, Meireles OR (2018) Artificial intelligence in surgery: promises and perils. Ann Surg 268(1):70–76CrossRefPubMed

26.

Hung AJ, Liu Y, Anandkumar A (2021) Deep learning to automate technical skills assessment in robotic surgery. JAMA Surg 156(11):1059–1060CrossRefPubMed

27.

Luongo F, Hakim R, Nguyen JH, Anandkumar A, Hung AJ (2021) Deep learning-based computer vision to recognize and classify suturing gestures in robot-assisted surgery. Surgery 169(5):1240–1244CrossRefPubMed

28.

Feichtenhofer C (2020) X3D: expanding architectures for efficient video recognition. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 200–210

29.

Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN et al (2017) Attention is all you need. Adv Neur Inf Process Syst 30:5998–6008

30.

Pele O, Werman M (2009) Fast and robust earth mover's distances. In: 2009 IEEE 12th international conference on computer vision, pp 460–467

31.

Garrow CR, Kowalewski KF, Li L, Wagner M, Schmidt MW, Engelhardt S et al (2021) Machine learning for surgical phase recognition: a systematic review. Ann Surg 273(4):684–693CrossRefPubMed

32.

Ahmadi SA, Sielhorst T, Stauder R, Horn M, Feussner H, Navab N (2006) Recovery of surgical workflow without explicit models. Med Image Comput Comput Assist Interv 9(Pt 1):420–428PubMed

33.

Sodergren MH, Orihuela-Espina F, Clark J, Darzi A, Yang GZ (2010) A hidden Markov model-based analysis framework using eye-tracking data to characterise re-orientation strategies in minimally invasive surgery. Cogn Process 11(3):275–283CrossRefPubMed

34.

Kawka M, Gall TMH, Fang C, Liu R, Jiao LR (2022) Intraoperative video analysis and machine learning models will change the future of surgical training. Intell Surg 1:13–5CrossRef

35.

Jin Y et al (2018) SV-RCNet: workflow recognition from surgical videos using recurrent convolutional network. IEEE Trans Med Imaging 37(5):1114–25CrossRefPubMed

36.

Hashimoto DA, Rosman G, Witkowski ER, Stafford C, Navarette-Welton AJ, Rattner DW et al (2019) Computer vision analysis of intraoperative video: automated recognition of operative steps in laparoscopic sleeve gastrectomy. Ann Surg 270(3):414–421CrossRefPubMed

37.

Kitaguchi D, Takeshita N, Matsuzaki H, Takano H, Owada Y, Enomoto T et al (2020) Real-time automatic surgical phase recognition in laparoscopic sigmoidectomy using the convolutional neural network-based deep learning approach. Surg Endosc 34(11):4924–4931CrossRefPubMed

38.

Morita S, Tabuchi H, Masumoto H, Yamauchi T, Kamiura N (2019) Real-time extraction of important surgical phases in cataract surgery videos. Sci Rep 9(1):16590CrossRefPubMedPubMedCentral

39.

Ward TM, Hashimoto DA, Ban Y, Rattner DW, Inoue H, Lillemoe KD et al (2021) Automated operative phase identification in peroral endoscopic myotomy. Surg Endosc 35(7):4008–4015CrossRefPubMed

40.

Eppler MB, Sayegh AS, Maas M, Venkat A, Hemal S, Desai MM et al (2023) Automated capture of intraoperative adverse events using artificial intelligence: a systematic review and meta-analysis. J Clin Med 12(4):1687CrossRefPubMedPubMedCentral

41.

Shi P, Zhao Z, Liu K, Li F (2022) Attention-based spatial–temporal neural network for accurate phase recognition in minimally invasive surgery: feasibility and efficiency verification. J Comput Des Eng 9(2):406–416

42.

Namazi B, Sankaranarayanan G, Devarajan V (2018) Automatic detection of surgical phases in laparoscopic videos. In: International conference on artificial intelligence (ICAI'18), pp 124–130

Title: The development of a deep learning model for automated segmentation of the robotic pancreaticojejunostomy
Authors: Amr I. Al Abbas
Babak Namazi
Imad Radi
Rodrigo Alterio
Andres A. Abreu
Benjamin Rail
Patricio M. Polanco
Herbert J. Zeh III
Melissa E. Hogg
Amer H. Zureikat
Ganesh Sankaranarayanan
Publication date: 15-03-2024
Publisher: Springer US
Keyword: Pancreaticojejunostomy
Published in: Surgical Endoscopy / Issue 5/2024
Print ISSN: 0930-2794
Electronic ISSN: 1432-2218
DOI: https://doi.org/10.1007/s00464-024-10725-x

Keynote webinar | Spotlight on medication adherence

Springer Medicine

The development of a deep learning model for automated segmentation of the robotic pancreaticojejunostomy

Abstract

Background

Methods

Results

Conclusion

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 5/2024

Robotic-assisted surgery for left-sided colon and rectal resections is associated with reduction in the postoperative surgical stress response and improved short-term outcomes: a cohort study

Reasons for conversions in thoracoscopic repairs of neonatal congenital diaphragmatic hernias: a systematic review

Intensive pre-operative information course (IPIC) and pre-operative weight loss results in long-term sustained weight loss following bariatric surgery: 11 years results from a tertiary referral centre

Correction: Acute cholecystitis, obesity, and steatohepatitis constitute the lethal triad for bile duct injury (BDI) during laparoscopic cholecystectomy

Balloon endoscopy-assisted endoscopic retrograde cholangiopancreatography for hepatolithiasis in patients with hepaticojejunostomy

Analysis of factors influencing pancreatic fistula after minimally invasive pancreaticoduodenectomy and establishment of a new prediction model for clinically relevant pancreatic fistula