Skip to main content
Top
Published in: Journal of Robotic Surgery 3/2015

01-09-2015 | Original Article

Augmented reality and cone beam CT guidance for transoral robotic surgery

Authors: Wen P. Liu, Jeremy D. Richmon, Jonathan M. Sorger, Mahdi Azizian, Russell H. Taylor

Published in: Journal of Robotic Surgery | Issue 3/2015

Login to get access

Abstract

In transoral robotic surgery preoperative image data do not reflect large deformations of the operative workspace from perioperative setup. To address this challenge, in this study we explore image guidance with cone beam computed tomographic angiography to guide the dissection of critical vascular landmarks and resection of base-of-tongue neoplasms with adequate margins for transoral robotic surgery. We identify critical vascular landmarks from perioperative c-arm imaging to augment the stereoscopic view of a da Vinci si robot in addition to incorporating visual feedback from relative tool positions. Experiments resecting base-of-tongue mock tumors were conducted on a series of ex vivo and in vivo animal models comparing the proposed workflow for video augmentation to standard non-augmented practice and alternative, fluoroscopy-based image guidance. Accurate identification of registered augmented critical anatomy during controlled arterial dissection and en bloc mock tumor resection was possible with the augmented reality system. The proposed image-guided robotic system also achieved improved resection ratios of mock tumor margins (1.00) when compared to control scenarios (0.0) and alternative methods of image guidance (0.58). The experimental results show the feasibility of the proposed workflow and advantages of cone beam computed tomography image guidance through video augmentation of the primary stereo endoscopy as compared to control and alternative navigation methods.
Literature
1.
go back to reference Zhen W, Karnell LH, Hoffman HT, Funk GF, Buatti JM, Menck HR (2004) The National Cancer Data Base report on squamous cell carcinoma of the base of tongue. Head Neck 26:660–674CrossRefPubMed Zhen W, Karnell LH, Hoffman HT, Funk GF, Buatti JM, Menck HR (2004) The National Cancer Data Base report on squamous cell carcinoma of the base of tongue. Head Neck 26:660–674CrossRefPubMed
2.
go back to reference Weinstein GS, O’Malley BW Jr, Magnuson JS, Carroll WR, Olsen KD, Daio L, Moore EJ, Holsinger FC (2012) Transoral robotic surgery: a multicenter study to assess feasibility, safety, and surgical margins. Laryngoscope 122:1701–1707CrossRefPubMed Weinstein GS, O’Malley BW Jr, Magnuson JS, Carroll WR, Olsen KD, Daio L, Moore EJ, Holsinger FC (2012) Transoral robotic surgery: a multicenter study to assess feasibility, safety, and surgical margins. Laryngoscope 122:1701–1707CrossRefPubMed
3.
go back to reference Weinstein GS, Quon H, Newman HJ, Chalian JA, Malloy K, Lin A, Desai A, Livolsi VA, Montone KT, Cohen KR, O’Malley BW (2012) Transoral robotic surgery alone for oropharyngeal cancer: an analysis of local control. Arch otolaryngol-head and neck surg 138:628–634CrossRef Weinstein GS, Quon H, Newman HJ, Chalian JA, Malloy K, Lin A, Desai A, Livolsi VA, Montone KT, Cohen KR, O’Malley BW (2012) Transoral robotic surgery alone for oropharyngeal cancer: an analysis of local control. Arch otolaryngol-head and neck surg 138:628–634CrossRef
4.
go back to reference Van de Kelft E, Costa F, Van der Planken D, Schils F (2012) A prospective multicenter registry on the accuracy of pedicle screw placement in the thoracic, lumbar, and sacral levels with the use of the O-arm imaging system and StealthStation Navigation. Spine 37:E1580–E1587CrossRefPubMed Van de Kelft E, Costa F, Van der Planken D, Schils F (2012) A prospective multicenter registry on the accuracy of pedicle screw placement in the thoracic, lumbar, and sacral levels with the use of the O-arm imaging system and StealthStation Navigation. Spine 37:E1580–E1587CrossRefPubMed
5.
go back to reference Su LM, Vagvolgyi BP, Agarwal R, Reiley CE, Taylor RH, Hager GD (2009) Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. Urology 73:896–900CrossRefPubMed Su LM, Vagvolgyi BP, Agarwal R, Reiley CE, Taylor RH, Hager GD (2009) Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. Urology 73:896–900CrossRefPubMed
6.
go back to reference Hughes-Hallett A, Mayer EK, Marcus HJ, Cundy TP, Pratt PJ, Darzi AW, Vale JA (2013) Augmented reality partial nephrectomy: examining the current status and future perspectives. Urology 83:266–273CrossRefPubMed Hughes-Hallett A, Mayer EK, Marcus HJ, Cundy TP, Pratt PJ, Darzi AW, Vale JA (2013) Augmented reality partial nephrectomy: examining the current status and future perspectives. Urology 83:266–273CrossRefPubMed
7.
go back to reference Volonte F, Buchs NC, Pugin F, Spaltenstein J, Jung M, Ratib O, Morel P (2013) Stereoscopic augmented reality for da Vinci robotic biliary surgery. Int J Surg case rep 4:365–367PubMedCentralCrossRefPubMed Volonte F, Buchs NC, Pugin F, Spaltenstein J, Jung M, Ratib O, Morel P (2013) Stereoscopic augmented reality for da Vinci robotic biliary surgery. Int J Surg case rep 4:365–367PubMedCentralCrossRefPubMed
8.
go back to reference Cabrilo I, Sarrafzadeh A, Bijlenga P, Landis BN, Schaller K (2014) Augmented reality-assisted skull base surgery. Neurochirurgie 60(6):304–306CrossRefPubMed Cabrilo I, Sarrafzadeh A, Bijlenga P, Landis BN, Schaller K (2014) Augmented reality-assisted skull base surgery. Neurochirurgie 60(6):304–306CrossRefPubMed
9.
go back to reference Caversaccio M, Garcia Giraldez J, Thoranaghatte R, Zheng G, Eggli P, Nolte LP, Gonzalez Ballester MA (2008) Augmented reality endoscopic system (ARES): preliminary results. Rhinology 46(2):156–158PubMed Caversaccio M, Garcia Giraldez J, Thoranaghatte R, Zheng G, Eggli P, Nolte LP, Gonzalez Ballester MA (2008) Augmented reality endoscopic system (ARES): preliminary results. Rhinology 46(2):156–158PubMed
10.
go back to reference Caversaccio M, Langlotz F, Nolte LP, Häusler R (2007) Impact of a self-developed planning and self-constructed navigation system on skull base surgery: 10 years experience. Acta Otolaryngol 127(4):403–407CrossRefPubMed Caversaccio M, Langlotz F, Nolte LP, Häusler R (2007) Impact of a self-developed planning and self-constructed navigation system on skull base surgery: 10 years experience. Acta Otolaryngol 127(4):403–407CrossRefPubMed
11.
go back to reference Chen X, Wang L, Fallavollita P, Navab N (2013) Precise X-ray and video overlay for augmented reality fluoroscopy. Int J Comput Assist Radiol Surg 8:29–38CrossRefPubMed Chen X, Wang L, Fallavollita P, Navab N (2013) Precise X-ray and video overlay for augmented reality fluoroscopy. Int J Comput Assist Radiol Surg 8:29–38CrossRefPubMed
12.
go back to reference Volonte F, Buchs NC, Pugin F, Spaltenstein J, Schiltz B, Jung M, Hagen M, Ratib O, Morel P (2013) Augmented reality to the rescue of the minimally invasive surgeon. The usefulness of the interposition of stereoscopic images in the da Vinci robotic console. Int J Med Robotics + Comp Assisted Surg : MRCAS 9:e34–38CrossRef Volonte F, Buchs NC, Pugin F, Spaltenstein J, Schiltz B, Jung M, Hagen M, Ratib O, Morel P (2013) Augmented reality to the rescue of the minimally invasive surgeon. The usefulness of the interposition of stereoscopic images in the da Vinci robotic console. Int J Med Robotics + Comp Assisted Surg : MRCAS 9:e34–38CrossRef
13.
go back to reference Mirota DJ, Uneri A, Schafer, S, Nithiananthan S, Reh, DD, Gallia GL, Taylor RH, Hager GD, Siewerdsen JH (2011) High-accuracy 3D image-based registration of endoscopic video to C-arm cone-beam CT for image-guided skull base surgery. In: Wong, K.H., Holmes Iii, D.R. (eds.) SPIE Medical Imaging, vol 7964, pp 79640 J-79610. SPIE, Lake Buena Vista, FL Mirota DJ, Uneri A, Schafer, S, Nithiananthan S, Reh, DD, Gallia GL, Taylor RH, Hager GD, Siewerdsen JH (2011) High-accuracy 3D image-based registration of endoscopic video to C-arm cone-beam CT for image-guided skull base surgery. In: Wong, K.H., Holmes Iii, D.R. (eds.) SPIE Medical Imaging, vol 7964, pp 79640 J-79610. SPIE, Lake Buena Vista, FL
14.
go back to reference Liu WP, Mirota DJ, Uneri A, Otake Y, Hager GD, Reh DD, Ishii ML, Siewerdsen JH (2012) A clinical pilot study of a modular video-CT augmentation system for image-guided skull base surgery. In: SPIE Medical Imaging 2012: Image-Guided Procedures, Robotic Interventions, and Modeling, pp 8316–8112 Liu WP, Mirota DJ, Uneri A, Otake Y, Hager GD, Reh DD, Ishii ML, Siewerdsen JH (2012) A clinical pilot study of a modular video-CT augmentation system for image-guided skull base surgery. In: SPIE Medical Imaging 2012: Image-Guided Procedures, Robotic Interventions, and Modeling, pp 8316–8112
15.
go back to reference Pratt P, Edwards E, Arora A, Tolley N, Darzi A.W, Yang G.-Z (2012) Image-guided transoral robotic surgery for the treatment of oropharyngeal cancer. In: Hamlyn Symposium Pratt P, Edwards E, Arora A, Tolley N, Darzi A.W, Yang G.-Z (2012) Image-guided transoral robotic surgery for the treatment of oropharyngeal cancer. In: Hamlyn Symposium
16.
go back to reference Falk V, Mourgues F, Vieville T, Jacobs S, Holzhey D, Walther T, Mohr FW, Coste-Maniere E (2005) Augmented reality for intraoperative guidance in endoscopic coronary artery bypass grafting. Surgical technology international 14:231–235PubMed Falk V, Mourgues F, Vieville T, Jacobs S, Holzhey D, Walther T, Mohr FW, Coste-Maniere E (2005) Augmented reality for intraoperative guidance in endoscopic coronary artery bypass grafting. Surgical technology international 14:231–235PubMed
17.
go back to reference Pietrabissa A, Morelli L, Ferrari M, Peri A, Ferrari V, Moglia A, Pugliese L, Guarracino F, Mosca F (2010) Mixed reality for robotic treatment of a splenic artery aneurysm. Surg Endosc 24:1204CrossRefPubMed Pietrabissa A, Morelli L, Ferrari M, Peri A, Ferrari V, Moglia A, Pugliese L, Guarracino F, Mosca F (2010) Mixed reality for robotic treatment of a splenic artery aneurysm. Surg Endosc 24:1204CrossRefPubMed
18.
go back to reference Suzuki N, Hattori A, Suzuki S, Otake Y (2007) Development of a surgical robot system for endovascular surgery with augmented reality function. Stud Health Technol Inform 125:460–463PubMed Suzuki N, Hattori A, Suzuki S, Otake Y (2007) Development of a surgical robot system for endovascular surgery with augmented reality function. Stud Health Technol Inform 125:460–463PubMed
19.
go back to reference Herrell, SD, Kwartowitz, DM, Milhoua, PM, Galloway, RL (2009) Toward image guided robotic surgery: system validation. J Urol 181, 783–789; discussion 789–790 Herrell, SD, Kwartowitz, DM, Milhoua, PM, Galloway, RL (2009) Toward image guided robotic surgery: system validation. J Urol 181, 783–789; discussion 789–790
20.
go back to reference Edwards PJ, King AP, Hawkes DJ, Fleig O, Maurer CR Jr, Hill DL, Fenlon MR, de Cunha DA, Gaston RP, Chandra S, Mannss J, Strong AJ, Gleeson MJ, Cox TC (1999) Stereo augmented reality in the surgical microscope. Stud Health Technol Inform 62:102–108PubMed Edwards PJ, King AP, Hawkes DJ, Fleig O, Maurer CR Jr, Hill DL, Fenlon MR, de Cunha DA, Gaston RP, Chandra S, Mannss J, Strong AJ, Gleeson MJ, Cox TC (1999) Stereo augmented reality in the surgical microscope. Stud Health Technol Inform 62:102–108PubMed
21.
go back to reference Surgery Toward Intraoperative Image-Guided Transoral Robotic (2013) Liu, W.P., Reaungamornrat, S., A., D., Sorger, J.M., Siewerdsen, J.H., Richmon, J.D., Taylor, R.H. Robotic Surgery 7:217–225CrossRef Surgery Toward Intraoperative Image-Guided Transoral Robotic (2013) Liu, W.P., Reaungamornrat, S., A., D., Sorger, J.M., Siewerdsen, J.H., Richmon, J.D., Taylor, R.H. Robotic Surgery 7:217–225CrossRef
22.
go back to reference Badani KK, Shapiro EY, Berg WT, Kaufman S, Bergman A, Wambi C, Roychoudhury A, Patel T (2013) A Pilot Study of Laparoscopic Doppler Ultrasound Probe to Map Arterial Vascular Flow within the Neurovascular Bundle during Robot-Assisted Radical Prostatectomy. Prostate cancer 2013:810715PubMedCentralCrossRefPubMed Badani KK, Shapiro EY, Berg WT, Kaufman S, Bergman A, Wambi C, Roychoudhury A, Patel T (2013) A Pilot Study of Laparoscopic Doppler Ultrasound Probe to Map Arterial Vascular Flow within the Neurovascular Bundle during Robot-Assisted Radical Prostatectomy. Prostate cancer 2013:810715PubMedCentralCrossRefPubMed
23.
go back to reference Liu WP, Reaungamornrat SAD, Sorger JM, Siewerdsen JH, Richmon JD, Taylor RH (2013) Toward intraoperative image-guided transoral robotic surgery. Robotic Surg. 7:217–225CrossRef Liu WP, Reaungamornrat SAD, Sorger JM, Siewerdsen JH, Richmon JD, Taylor RH (2013) Toward intraoperative image-guided transoral robotic surgery. Robotic Surg. 7:217–225CrossRef
24.
go back to reference Reaungamornrat S, Liu WP, Wang AS, Otake Y, Nithiananthan S, Uneri A, Schafer S, Tryggestad E, Richmon J, Sorger JM, Siewerdsen JH, Taylor RH (2013) Deformable image registration for cone-beam CT guided transoral robotic base-of-tongue surgery. Phys Med Biol 58:4951–4979PubMedCentralCrossRefPubMed Reaungamornrat S, Liu WP, Wang AS, Otake Y, Nithiananthan S, Uneri A, Schafer S, Tryggestad E, Richmon J, Sorger JM, Siewerdsen JH, Taylor RH (2013) Deformable image registration for cone-beam CT guided transoral robotic base-of-tongue surgery. Phys Med Biol 58:4951–4979PubMedCentralCrossRefPubMed
25.
go back to reference Yushkevich Paul A, Piven Joseph, Hazlett Heather Cody, Smith Rachel Gimpel, Ho Sean, Gee James C, Gerig Guido (2006) User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability. Neuroimage 31(3):1116–1128CrossRefPubMed Yushkevich Paul A, Piven Joseph, Hazlett Heather Cody, Smith Rachel Gimpel, Ho Sean, Gee James C, Gerig Guido (2006) User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability. Neuroimage 31(3):1116–1128CrossRefPubMed
27.
go back to reference Jung MY, Balicki M., Deguet A, Taylor RH, Kazanzides P (2014) Lessons learned from the development of component-based medical robot systems. software engineering for robotics 5(2):25–41 Jung MY, Balicki M., Deguet A, Taylor RH, Kazanzides P (2014) Lessons learned from the development of component-based medical robot systems. software engineering for robotics 5(2):25–41
28.
go back to reference Pieper S, Lorenson B, Schroeder W, Kikinis R (2006) The NA-MIC kit: ITK, VTK, pipelines, grids, and 3D Slicer as an open platform for the medical image computing community. Proc. IEEE Intl. Symp. Biomed. Imag 698–701 Pieper S, Lorenson B, Schroeder W, Kikinis R (2006) The NA-MIC kit: ITK, VTK, pipelines, grids, and 3D Slicer as an open platform for the medical image computing community. Proc. IEEE Intl. Symp. Biomed. Imag 698–701
29.
go back to reference Reiter A, Allen PK, Zhao T (2012) Feature classification for tracking articulated surgical tools. Med Image Comput Comput Assist Interv 15:592–600PubMed Reiter A, Allen PK, Zhao T (2012) Feature classification for tracking articulated surgical tools. Med Image Comput Comput Assist Interv 15:592–600PubMed
30.
go back to reference Liu WP, Reaugamornrat S, Sorger JM, Siewerdsen JH, Taylor RH, Richmon JD (2014) Intraoperative image-guided transoral robotic surgery: pre-clinical studies. Int J Med Robot 11(2):256–267CrossRefPubMed Liu WP, Reaugamornrat S, Sorger JM, Siewerdsen JH, Taylor RH, Richmon JD (2014) Intraoperative image-guided transoral robotic surgery: pre-clinical studies. Int J Med Robot 11(2):256–267CrossRefPubMed
31.
go back to reference Rieger A, Blum T, Navab N, Friess H, Martignoni ME (2011) Augmented reality: merge of reality and virtuality in medicine. Dtsch Med Wochenschr 136:2427–2433CrossRefPubMed Rieger A, Blum T, Navab N, Friess H, Martignoni ME (2011) Augmented reality: merge of reality and virtuality in medicine. Dtsch Med Wochenschr 136:2427–2433CrossRefPubMed
32.
go back to reference Stayman JW, Otake Y, Prince JL, Khanna AJ, Siewerdsen JH (2012) Model-Based tomographic reconstruction of objects containing known components. IEEE Trans Med Imaging 31(10):1837–1848PubMedCentralCrossRefPubMed Stayman JW, Otake Y, Prince JL, Khanna AJ, Siewerdsen JH (2012) Model-Based tomographic reconstruction of objects containing known components. IEEE Trans Med Imaging 31(10):1837–1848PubMedCentralCrossRefPubMed
Metadata
Title
Augmented reality and cone beam CT guidance for transoral robotic surgery
Authors
Wen P. Liu
Jeremy D. Richmon
Jonathan M. Sorger
Mahdi Azizian
Russell H. Taylor
Publication date
01-09-2015
Publisher
Springer London
Published in
Journal of Robotic Surgery / Issue 3/2015
Print ISSN: 1863-2483
Electronic ISSN: 1863-2491
DOI
https://doi.org/10.1007/s11701-015-0520-5

Other articles of this Issue 3/2015

Journal of Robotic Surgery 3/2015 Go to the issue