Top

International Journal of Computer Assisted Radiology and Surgery

Published in:

Open Access 01-05-2021 | Original Article

Real-to-virtual domain transfer-based depth estimation for real-time 3D annotation in transnasal surgery: a study of annotation accuracy and stability

Authors: Hon-Sing Tong, Yui-Lun Ng, Zhiyu Liu, Justin D. L. Ho, Po-Ling Chan, Jason Y. K. Chan, Ka-Wai Kwok

Published in: International Journal of Computer Assisted Radiology and Surgery | Issue 5/2021

Abstract

Purpose

Surgical annotation promotes effective communication between medical personnel during surgical procedures. However, existing approaches to 2D annotations are mostly static with respect to a display. In this work, we propose a method to achieve 3D annotations that anchor rigidly and stably to target structures upon camera movement in a transnasal endoscopic surgery setting.

Methods

This is accomplished through intra-operative endoscope tracking and monocular depth estimation. A virtual endoscopic environment is utilized to train a supervised depth estimation network. An adversarial network transfers the style from the real endoscopic view to a synthetic-like view for input into the depth estimation network, wherein framewise depth can be obtained in real time.

Results

(1) Accuracy: Framewise depth was predicted from images captured from within a nasal airway phantom and compared with ground truth, achieving a SSIM value of 0.8310 ± 0.0655. (2) Stability: mean absolute error (MAE) between reference and predicted depth of a target point was 1.1330 ± 0.9957 mm.

Conclusion

Both the accuracy and stability evaluations demonstrated the feasibility and practicality of our proposed method for achieving 3D annotations.

https://www.iris.mech.hku.hk/surgical-planning.

https://youtu.be/kCi1ux-Q1FQ.

Treter S, Perrier N, Sosa JA, Roman S (2013) Telementoring: a multi-institutional experience with the introduction of a novel surgical approach for adrenalectomy. Ann Surg Oncol 20(8):2754–2758CrossRef

Kwok KW, Sun LW, Mylonas GP, James DR, Orihuela-Espina F, Yang GZ (2012) Collaborative gaze channelling for improved cooperation during robotic assisted surgery. Ann Biomed Eng 40(10):2156–2167CrossRef

Bogen EM, Augestad KM, Patel HR, Lindsetmo RO (2014) Telementoring in education of laparoscopic surgeons: an emerging technology. World J Gastrointestinal Endosc 6(5):148CrossRef

Lee SL, Lerotic M, Vitiello V, Giannarou S, Kwok KW, Visentini-Scarzanella M, Yang GZ (2010) From medical images to minimally invasive intervention: computer assistance for robotic surgery. Comput Med Imaging Graph 34(1):33–45CrossRef

Bernhardt S, Nicolau SA, Soler L, Doignon C (2017) The status of augmented reality in laparoscopic surgery as of 2016. Med Image Anal 37:66–90CrossRef

Vávra P, Roman J, Zonča P, Ihnát P, Němec M, Kumar J, Habib N, El-Gendi A (2017) Recent development of augmented reality in surgery: a review. J Healthcare Eng 1–9

Stoyanov D, Scarzanella MV, Pratt P, Yang GZ (2010) Real-time stereo reconstruction in robotically assisted minimally invasive surgery. In: International conference on medical image computing and computer-assisted intervention. Springer

Mirota DJ, Wang H, Taylor RH, Ishii M, Gallia GL, Hager GD (2011) A system for video-based navigation for endoscopic endonasal skull base surgery. IEEE Trans Med Imaging 31(4):963–976CrossRef

Mur-Artal R, Montiel JMM, Tardos JD (2015) ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Trans Rob 31(5):1147–1163CrossRef

10.

Mahmoud N, Cirauqui I, Hostettler A, Doignon C, Soler L, Marescaux J, Montiel J (2016) ORBSLAM-based endoscope tracking and 3D reconstruction. International workshop on computer-assisted and robotic endoscopy. Springer

11.

Ma R, Wang R, Pizer S, Rosenman J, McGill SK, Frahm JM (2019) Real-time 3D reconstruction of colonoscopic surfaces for determining missing regions. In: International conference on medical image computing and computer-assisted intervention. Springer

12.

Fu H, Gong M, Wang C, Batmanghelich K, Tao D (2018) Deep ordinal regression network for monocular depth estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition

13.

Alhashim I, Wonka P (2018) High quality monocular depth estimation via transfer learning. arXiv preprint arXiv:1812.11941

14.

Liu X, Sinha A, Ishii M, Hager GD, Reiter A, Taylor RH, Unberath M (2019) Dense depth estimation in monocular endoscopy with self-supervised learning methods. IEEE Trans Med Imaging 39(5):1438–1447CrossRef

15.

Reiter A, Léonard S, Sinha A, Ishii M, Taylor RH, Hager GD (2016) Endoscopic-CT: learning-based photometric reconstruction for endoscopic sinus surgery. In: Medical imaging 2016: image processing. International Society for Optics and Photonics

16.

Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition

17.

Rau A, Edwards PE, Ahmad OF, Riordan P, Janatka M, Lovat LB, Stoyanov D (2019) Implicit domain adaptation with conditional generative adversarial networks for depth prediction in endoscopy. Int J Comput Assist Radiol Surg 14(7):1167–1176CrossRef

18.

Atapour-Abarghouei A, Breckon TP (2018) Real-time monocular depth estimation using synthetic data with domain adaptation via image style transfer. In: Proceedings of the IEEE conference on computer vision and pattern recognition

19.

Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision

20.

Zhao C, Shen M, Sun L, Yang GZ (2019) Generative localization with uncertainty estimation through video-CT data for bronchoscopic biopsy. IEEE Robot Autom Lett 5(1):258–265CrossRef

21.

Mahmood F, Chen R, Durr NJ (2018) Unsupervised reverse domain adaptation for synthetic medical images via adversarial training. IEEE Trans Med Imaging 37(12):2572–2581CrossRef

22.

Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems

23.

Tsai RY, Lenz RK (1989) A new technique for fully autonomous and efficient 3 D robotics hand/eye calibration. IEEE Trans Robot Autom 5(3):345–358CrossRef

24.

He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition

25.

Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980

26.

Wang L, Shen X, Zhang J, Wang O, Lin Z, Hsieh CY, Kong S, Lu H (2018) DeepLens: shallow depth of field from a single image. arXiv preprint arXiv:1810.08100

27.

Arun KS, Huang TS, Blostein SD (1987) Least-squares fitting of two 3-D point sets. IEEE Trans Pattern Anal Mach Intell 5:698–700CrossRef

28.

Luo X, Huang JB, Szeliski R, Matzen K, Kopf J (2020) Consistent video depth estimation. arXiv preprint arXiv:2004.15021

29.

Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612CrossRef

30.

Nadeem S, Kaufman A (2016) Computer-aided detection of polyps in optical colonoscopy images. In: Medical imaging 2016: computer-aided diagnosis. International Society for Optics and Photonics

Title: Real-to-virtual domain transfer-based depth estimation for real-time 3D annotation in transnasal surgery: a study of annotation accuracy and stability
Authors: Hon-Sing Tong
Yui-Lun Ng
Zhiyu Liu
Justin D. L. Ho
Po-Ling Chan
Jason Y. K. Chan
Ka-Wai Kwok
Publication date: 01-05-2021
Publisher: Springer International Publishing
Published in: International Journal of Computer Assisted Radiology and Surgery / Issue 5/2021
Print ISSN: 1861-6410
Electronic ISSN: 1861-6429
DOI: https://doi.org/10.1007/s11548-021-02346-9

At a glance: The STEP trials

Springer Medicine

Real-to-virtual domain transfer-based depth estimation for real-time 3D annotation in transnasal surgery: a study of annotation accuracy and stability

Abstract

Purpose

Methods

Results

Conclusion

At a glance: The STEP trials

Springer Medicine

Abstract

Purpose

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 5/2021

Fast and robust localization of surgical array using Kalman filter

Against spatial–temporal discrepancy: contrastive learning-based network for surgical workflow recognition

Domain adaptation and self-supervised learning for surgical margin detection

Automatic extraction of the mitral valve chordae geometry for biomechanical simulation

CycleGAN for interpretable online EMT compensation

Learning ultrasound rendering from cross-sectional model slices for simulated training