Skip to main content
Top
Published in: International Journal of Computer Assisted Radiology and Surgery 7/2020

Open Access 01-07-2020 | Vestibular Schwannoma | Original Article

i3PosNet: instrument pose estimation from X-ray in temporal bone surgery

Authors: David Kügler, Jannik Sehring, Andrei Stefanov, Igor Stenin, Julia Kristin, Thomas Klenzner, Jörg Schipper, Anirban Mukhopadhyay

Published in: International Journal of Computer Assisted Radiology and Surgery | Issue 7/2020

Login to get access

Abstract

Purpose

Accurate estimation of the position and orientation (pose) of surgical instruments is crucial for delicate minimally invasive temporal bone surgery. Current techniques lack in accuracy and/or line-of-sight constraints (conventional tracking systems) or expose the patient to prohibitive ionizing radiation (intra-operative CT). A possible solution is to capture the instrument with a c-arm at irregular intervals and recover the pose from the image.

Methods

i3PosNet infers the position and orientation of instruments from images using a pose estimation network. Said framework considers localized patches and outputs pseudo-landmarks. The pose is reconstructed from pseudo-landmarks by geometric considerations.

Results

We show i3PosNet reaches errors \(<\,0.05\) mm. It outperforms conventional image registration-based approaches reducing average and maximum errors by at least two thirds. i3PosNet trained on synthetic images generalizes to real X-rays without any further adaptation.

Conclusion

The translation of deep learning-based methods to surgical applications is difficult, because large representative datasets for training and testing are not available. This work empirically shows sub-millimeter pose estimation trained solely based on synthetic training data.
Footnotes
1
Find additional information at https://​i3posnet.​david-kuegler.​de/​
 
Literature
1.
go back to reference Ansó J, Dür C, Gavaghan K, Rohrbach H, Gerber N, Williamson T, Calvo EM, Balmer TW, Precht C, Ferrario D, Dettmer MS, Rösler KM, Caversaccio MD, Bell B, Weber S (2016) A neuromonitoring approach to facial nerve preservation during image-guided robotic cochlear implantation. Otol Neurotol 37(1):89–98CrossRef Ansó J, Dür C, Gavaghan K, Rohrbach H, Gerber N, Williamson T, Calvo EM, Balmer TW, Precht C, Ferrario D, Dettmer MS, Rösler KM, Caversaccio MD, Bell B, Weber S (2016) A neuromonitoring approach to facial nerve preservation during image-guided robotic cochlear implantation. Otol Neurotol 37(1):89–98CrossRef
2.
go back to reference Balachandran R, Mitchell JE, Blachon G, Noble JH, Dawant BM, Fitzpatrick JM, Labadie RF (2010) Percutaneous cochlear implant drilling via customized frames: an in vitro study. Otolaryngol Head Neck Surg 142(3):421–426CrossRef Balachandran R, Mitchell JE, Blachon G, Noble JH, Dawant BM, Fitzpatrick JM, Labadie RF (2010) Percutaneous cochlear implant drilling via customized frames: an in vitro study. Otolaryngol Head Neck Surg 142(3):421–426CrossRef
3.
go back to reference Bier B, Unberath M, Zaech JN, Fotouhi J, Armand M, Osgood G, Navab N, Maier A (2018) X-ray-transform invariant anatomical landmark detection for Pelvic trauma surgery. In: MICCAI 2018. Springer Bier B, Unberath M, Zaech JN, Fotouhi J, Armand M, Osgood G, Navab N, Maier A (2018) X-ray-transform invariant anatomical landmark detection for Pelvic trauma surgery. In: MICCAI 2018. Springer
4.
go back to reference Bui M, Albarqouni S, Schrapp M, Navab N, Ilic S (2017) X-Ray PoseNet: 6 DoF pose estimation for mobile x-ray devices. In: WACV 2017 Bui M, Albarqouni S, Schrapp M, Navab N, Ilic S (2017) X-Ray PoseNet: 6 DoF pose estimation for mobile x-ray devices. In: WACV 2017
5.
go back to reference Caversaccio M, Gavaghan K, Wimmer W, Williamson T, Ansò J, Mantokoudis G, Gerber N, Rathgeb C, Feldmann A, Wagner F, Scheidegger O, Kompis M, Weisstanner C, Zoka-Assadi M, Roesler K, Anschuetz L, Huth M, Weber S (2017) Robotic cochlear implantation: surgical procedure and first clinical experience. Acta Oto-laryngol 137(4):447–454CrossRef Caversaccio M, Gavaghan K, Wimmer W, Williamson T, Ansò J, Mantokoudis G, Gerber N, Rathgeb C, Feldmann A, Wagner F, Scheidegger O, Kompis M, Weisstanner C, Zoka-Assadi M, Roesler K, Anschuetz L, Huth M, Weber S (2017) Robotic cochlear implantation: surgical procedure and first clinical experience. Acta Oto-laryngol 137(4):447–454CrossRef
6.
go back to reference Esfandiari H, Newell R, Anglin C, Street J, Hodgson AJ (2018) A deep learning framework for segmentation and pose estimation of pedicle screw implants based on C-arm fluoroscopy. IJCARS 13(8):1269–1282 Esfandiari H, Newell R, Anglin C, Street J, Hodgson AJ (2018) A deep learning framework for segmentation and pose estimation of pedicle screw implants based on C-arm fluoroscopy. IJCARS 13(8):1269–1282
7.
go back to reference Fauser J, Sakas G, Mukhopadhyay A (2018) Planning nonlinear access paths for temporal bone surgery. IJCARS 13(5):637–646 Fauser J, Sakas G, Mukhopadhyay A (2018) Planning nonlinear access paths for temporal bone surgery. IJCARS 13(5):637–646
8.
go back to reference Gao G, Penney G, Gogin N, Cathier P, Arujuna A, Wright M, Caulfield D, Rinaldi A, Razavi R, Rhode K (2010) Rapid image registration of three-dimensional transesophageal echocardiography and x-ray fluoroscopy for the guidance of cardiac interventions. Lecture Notes in Computer Science. IPCAI. Springer, Berlin, pp 124–134 Gao G, Penney G, Gogin N, Cathier P, Arujuna A, Wright M, Caulfield D, Rinaldi A, Razavi R, Rhode K (2010) Rapid image registration of three-dimensional transesophageal echocardiography and x-ray fluoroscopy for the guidance of cardiac interventions. Lecture Notes in Computer Science. IPCAI. Springer, Berlin, pp 124–134
9.
go back to reference Hajj HA, Lamard M, Conze PH, Roychowdhury S, Hu X, Maršalkaite G, Zisimopoulos O, Dedmari MA, Zhao F, Prellberg J, Sahu M, Galdran A, Araújo T, Vo DM, Panda C, Dahiya N, Kondo S, Bian Z, Vahdat A, Bialopetravičius J, Flouty E, Qiu C, Dill S, Mukhopadhyay A, Costa P, Aresta G, Ramamurthy S, Lee SW, Campilho A, Zachow S, Xia S, Conjeti S, Stoyanov D, Armaitis J, Heng PA, Macready WG, Cochener B, Quellec G (2018) Cataracts: challenge on automatic tool annotation for cataract surgery. Medical image analysis Hajj HA, Lamard M, Conze PH, Roychowdhury S, Hu X, Maršalkaite G, Zisimopoulos O, Dedmari MA, Zhao F, Prellberg J, Sahu M, Galdran A, Araújo T, Vo DM, Panda C, Dahiya N, Kondo S, Bian Z, Vahdat A, Bialopetravičius J, Flouty E, Qiu C, Dill S, Mukhopadhyay A, Costa P, Aresta G, Ramamurthy S, Lee SW, Campilho A, Zachow S, Xia S, Conjeti S, Stoyanov D, Armaitis J, Heng PA, Macready WG, Cochener B, Quellec G (2018) Cataracts: challenge on automatic tool annotation for cataract surgery. Medical image analysis
10.
go back to reference Hatt CR, Speidel MA, Raval AN (2016) Real-time pose estimation of devices from x-ray images: application to x-ray/echo registration for cardiac interventions. Med Image Anal 34:101–108CrossRef Hatt CR, Speidel MA, Raval AN (2016) Real-time pose estimation of devices from x-ray images: application to x-ray/echo registration for cardiac interventions. Med Image Anal 34:101–108CrossRef
11.
go back to reference Jain AK, Mustafa T, Zhou Y, Burdette C, Chirikjian GS, Fichtinger G (2005) FTRAC-robust fluoroscope tracking fiducial. Med Phys 32(10):3185–3198CrossRef Jain AK, Mustafa T, Zhou Y, Burdette C, Chirikjian GS, Fichtinger G (2005) FTRAC-robust fluoroscope tracking fiducial. Med Phys 32(10):3185–3198CrossRef
12.
go back to reference Kügler D, Jastrzebski M, Mukhopadhyay A (2018) Instrument Pose estimation using registration for otobasis surgery. In: WBIR2018. Springer Kügler D, Jastrzebski M, Mukhopadhyay A (2018) Instrument Pose estimation using registration for otobasis surgery. In: WBIR2018. Springer
13.
go back to reference Kügler D, Krumb H, Bredemann J, Stenin I, Kristin J, Klenzner T, Schipper J, Schmitt R, Sakas G, Mukhopadhyay A (2019) High-precision evaluation of electromagnetic tracking. IJCARS Kügler D, Krumb H, Bredemann J, Stenin I, Kristin J, Klenzner T, Schipper J, Schmitt R, Sakas G, Mukhopadhyay A (2019) High-precision evaluation of electromagnetic tracking. IJCARS
14.
go back to reference Kurmann T, Marquez Neila P, Du X, Fua P, Stoyanov D, Wolf S, Sznitman R (2017) Simultaneous recognition and pose estimation of instruments in minimally invasive surgery. In: MICCAI 2017. Springer, pp 505–513 Kurmann T, Marquez Neila P, Du X, Fua P, Stoyanov D, Wolf S, Sznitman R (2017) Simultaneous recognition and pose estimation of instruments in minimally invasive surgery. In: MICCAI 2017. Springer, pp 505–513
15.
go back to reference Labadie RF, Balachandran R, Noble JH, Blachon GS, Mitchell JE, Reda FA, Dawant BM, Fitzpatrick JM (2014) Minimally invasive image-guided cochlear implantation surgery: first report of clinical implementation. The Laryngoscope 124(8):1915–1922CrossRef Labadie RF, Balachandran R, Noble JH, Blachon GS, Mitchell JE, Reda FA, Dawant BM, Fitzpatrick JM (2014) Minimally invasive image-guided cochlear implantation surgery: first report of clinical implementation. The Laryngoscope 124(8):1915–1922CrossRef
16.
go back to reference Miao S, Wang ZJ, Liao R (2016) A CNN regression approach for real-time 2D/3D registration. IEEE TMI 35(5):1352–1363 Miao S, Wang ZJ, Liao R (2016) A CNN regression approach for real-time 2D/3D registration. IEEE TMI 35(5):1352–1363
17.
go back to reference Nguyen Y, Miroir M, Vellin JF, Mazalaigue S, Bensimon JL, Bernardeschi D, Ferrary E, Sterkers O, Grayeli AB (2011) Minimally invasive computer-assisted approach for cochlear implantation: a human temporal bone study. Surg Innov 18(3):259–267CrossRef Nguyen Y, Miroir M, Vellin JF, Mazalaigue S, Bensimon JL, Bernardeschi D, Ferrary E, Sterkers O, Grayeli AB (2011) Minimally invasive computer-assisted approach for cochlear implantation: a human temporal bone study. Surg Innov 18(3):259–267CrossRef
18.
go back to reference Rathgeb C, Anschuetz L, Schneider D, Dür C, Caversaccio M, Weber S, Williamson T (2018) Accuracy and feasibility of a dedicated image guidance solution for endoscopic lateral skull base surgery. Eur Arch Oto-Rhino-Laryngol 275(4):905–911CrossRef Rathgeb C, Anschuetz L, Schneider D, Dür C, Caversaccio M, Weber S, Williamson T (2018) Accuracy and feasibility of a dedicated image guidance solution for endoscopic lateral skull base surgery. Eur Arch Oto-Rhino-Laryngol 275(4):905–911CrossRef
19.
go back to reference Schipper J, Aschendorff A, Arapakis I, Klenzner T, Teszler CB, Ridder GJ, Laszig R (2004) Navigation as a quality management tool in cochlear implant surgery. Laryngol Otol 118(10):764–770CrossRef Schipper J, Aschendorff A, Arapakis I, Klenzner T, Teszler CB, Ridder GJ, Laszig R (2004) Navigation as a quality management tool in cochlear implant surgery. Laryngol Otol 118(10):764–770CrossRef
21.
go back to reference Terunuma T, Tokui A, Sakae T (2018) Novel real-time tumor-contouring method using deep learning to prevent mistracking in x-ray fluoroscopy. Radiol Phys Technol 11(1):43–53CrossRef Terunuma T, Tokui A, Sakae T (2018) Novel real-time tumor-contouring method using deep learning to prevent mistracking in x-ray fluoroscopy. Radiol Phys Technol 11(1):43–53CrossRef
22.
go back to reference Unberath M, Zaech JN, Lee SC, Bier B, Fotouhi J, Armand M, Navab N (2018) Deepdrr–a catalyst for machine learning in fluoroscopy-guided procedures. In: Frangi A, Schnabel J, Davatsikos C, Alberola-López C, Fichtinger G, Frangi AF, Schnabel JA, Davatzikos C (eds) Medical image computing and computer assisted intervention–MICCAI 2018, Image processing, computer vision, pattern recognition, and graphics. Springer, Cham, pp 98–106. https://doi.org/10.1007/978-3-030-00937-3_12, http://arxiv.org/pdf/1803.08606 Unberath M, Zaech JN, Lee SC, Bier B, Fotouhi J, Armand M, Navab N (2018) Deepdrr–a catalyst for machine learning in fluoroscopy-guided procedures. In: Frangi A, Schnabel J, Davatsikos C, Alberola-López C, Fichtinger G, Frangi AF, Schnabel JA, Davatzikos C (eds) Medical image computing and computer assisted intervention–MICCAI 2018, Image processing, computer vision, pattern recognition, and graphics. Springer, Cham, pp 98–106. https://​doi.​org/​10.​1007/​978-3-030-00937-3_​12, http://​arxiv.​org/​pdf/​1803.​08606
24.
go back to reference Uneri A, Stayman JW, de Silva T, Wang AS, Kleinszig G, Vogt S, Khanna AJ, Wolinsky JP, Gokaslan ZL, Siewerdsen JH (2015) Known-component 3d–2d registration for image guidance and quality assurance in spine surgery pedicle screw placement. In: Proceedings of SPIE 9415 Uneri A, Stayman JW, de Silva T, Wang AS, Kleinszig G, Vogt S, Khanna AJ, Wolinsky JP, Gokaslan ZL, Siewerdsen JH (2015) Known-component 3d–2d registration for image guidance and quality assurance in spine surgery pedicle screw placement. In: Proceedings of SPIE 9415
25.
go back to reference van de Kraats EB, Penney GP, Tomazevic D, van Walsum T, Niessen WJ (2005) Standardized evaluation methodology for 2-D-3-D registration. IEEE TMI 24(9):1177–1189 van de Kraats EB, Penney GP, Tomazevic D, van Walsum T, Niessen WJ (2005) Standardized evaluation methodology for 2-D-3-D registration. IEEE TMI 24(9):1177–1189
26.
go back to reference Vandini A, Glocker B, Hamady M, Yang GZ (2017) Robust guidewire tracking under large deformations combining segment-like features (SEGlets). Med Image Anal 38:150–164CrossRef Vandini A, Glocker B, Hamady M, Yang GZ (2017) Robust guidewire tracking under large deformations combining segment-like features (SEGlets). Med Image Anal 38:150–164CrossRef
27.
go back to reference Vercauteren T, Unberath M, Padoy N, Navab N (2020) Cai4cai: the rise of contextual artificial intelligence in computer assisted interventions. Proc IEEE 108(1):198–214CrossRef Vercauteren T, Unberath M, Padoy N, Navab N (2020) Cai4cai: the rise of contextual artificial intelligence in computer assisted interventions. Proc IEEE 108(1):198–214CrossRef
28.
go back to reference Williamson TM, Bell BJ, Gerber N, Salas L, Zysset P, Caversaccio M, Weber S (2013) Estimation of tool pose based on force-density correlation during robotic drilling. IEEE TBME 60(4):969–976 Williamson TM, Bell BJ, Gerber N, Salas L, Zysset P, Caversaccio M, Weber S (2013) Estimation of tool pose based on force-density correlation during robotic drilling. IEEE TBME 60(4):969–976
29.
go back to reference Zhang L, Ye M, Chan PL, Yang GZ (2017) Real-time surgical tool tracking and pose estimation using a hybrid cylindrical marker. Int J Comput Assist Radiol Surg 12(6):921–930CrossRef Zhang L, Ye M, Chan PL, Yang GZ (2017) Real-time surgical tool tracking and pose estimation using a hybrid cylindrical marker. Int J Comput Assist Radiol Surg 12(6):921–930CrossRef
30.
go back to reference Zhang Y, Miao S, Mansi T, Liao R (2018) Task driven generative modeling for unsupervised domain adaptation: application to x-ray image segmentation. Medical image computing and computer assisted intervention—MICCAI 2018. Springer, Cham, pp 599–607CrossRef Zhang Y, Miao S, Mansi T, Liao R (2018) Task driven generative modeling for unsupervised domain adaptation: application to x-ray image segmentation. Medical image computing and computer assisted intervention—MICCAI 2018. Springer, Cham, pp 599–607CrossRef
Metadata
Title
i3PosNet: instrument pose estimation from X-ray in temporal bone surgery
Authors
David Kügler
Jannik Sehring
Andrei Stefanov
Igor Stenin
Julia Kristin
Thomas Klenzner
Jörg Schipper
Anirban Mukhopadhyay
Publication date
01-07-2020
Publisher
Springer International Publishing
Published in
International Journal of Computer Assisted Radiology and Surgery / Issue 7/2020
Print ISSN: 1861-6410
Electronic ISSN: 1861-6429
DOI
https://doi.org/10.1007/s11548-020-02157-4

Other articles of this Issue 7/2020

International Journal of Computer Assisted Radiology and Surgery 7/2020 Go to the issue