Top

Published in:

01-07-2014 | Dynamic Manuscript

Stereoscopic augmented reality for laparoscopic surgery

Authors: Xin Kang, Mahdi Azizian, Emmanuel Wilson, Kyle Wu, Aaron D. Martin, Timothy D. Kane, Craig A. Peters, Kevin Cleary, Raj Shekhar

Published in: Surgical Endoscopy | Issue 7/2014

Abstract

Background

Conventional laparoscopes provide a flat representation of the three-dimensional (3D) operating field and are incapable of visualizing internal structures located beneath visible organ surfaces. Computed tomography (CT) and magnetic resonance (MR) images are difficult to fuse in real time with laparoscopic views due to the deformable nature of soft-tissue organs. Utilizing emerging camera technology, we have developed a real-time stereoscopic augmented-reality (AR) system for laparoscopic surgery by merging live laparoscopic ultrasound (LUS) with stereoscopic video. The system creates two new visual cues: (1) perception of true depth with improved understanding of 3D spatial relationships among anatomical structures, and (2) visualization of critical internal structures along with a more comprehensive visualization of the operating field.

Methods

The stereoscopic AR system has been designed for near-term clinical translation with seamless integration into the existing surgical workflow. It is composed of a stereoscopic vision system, a LUS system, and an optical tracker. Specialized software processes streams of imaging data from the tracked devices and registers those in real time. The resulting two ultrasound-augmented video streams (one for the left and one for the right eye) give a live stereoscopic AR view of the operating field. The team conducted a series of stereoscopic AR interrogations of the liver, gallbladder, biliary tree, and kidneys in two swine.

Results

The preclinical studies demonstrated the feasibility of the stereoscopic AR system during in vivo procedures. Major internal structures could be easily identified. The system exhibited unobservable latency with acceptable image-to-video registration accuracy.

Conclusions

We presented the first in vivo use of a complete system with stereoscopic AR visualization capability. This new capability introduces new visual cues and enhances visualization of the surgical anatomy. The system shows promise to improve the precision and expand the capacity of minimally invasive laparoscopic surgeries.

Available only for authorised users

Himal HS (2002) Minimally invasive (laparoscopic) surgery. Surg Endosc 16(12):1647–1652PubMedCrossRef

Rosen M, Ponsky J (2001) Minimally invasive surgery. Endoscopy 33(4):358–366PubMedCrossRef

Storz P, Buess GF, Kunert W, Kirschniak A (2012) 3D HD versus 2D HD: surgical task efficiency in standardised phantom tasks. Surg Endosc 26(5):1454–1460PubMedCrossRef

Smith R, Day A, Rockall T, Ballard K, Bailey M, Jourdan I (2012) Advanced stereoscopic projection technology significantly improves novice performance of minimally invasive surgical skills. Surg Endosc 26(6):1522–1527PubMedCrossRef

Nakamoto M, Ukimura O, Faber K, Gill IS (2012) Current progress on augmented reality visualization in endoscopic surgery. Curr Opin Urol 22(2):121–126PubMedCrossRef

Linte CA, Wiles A, Moore J, Wedlake C, Peters TM (2008) Virtual reality-enhanced ultrasound guidance for atrial ablation: in vitro epicardial study. Med Image Comput Comput Assist Interv 11(Pt 2):644–651PubMed

Leven J, Burschka D, Kumar R, Zhang G, Blumenkranz S, Dai XD, Awad M, Hager GD, Marohn M, Choti M, Hasser C, Taylor RH (2005) DaVinci canvas: a telerobotic surgical system with integrated, robot-assisted, laparoscopic ultrasound capability. Med Image Comput Comput Assist Interv 8:811–818PubMed

Su LM, Vagvolgyi BP, Agarwal R, Reiley CE, Taylor RH, Hager GD (2009) Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. Urology 73(4):896–900PubMedCrossRef

Cheung CL, Wedlake C, Moore J, Pautler SE, Ahmad A, Peters TM (2009) Fusion of stereoscopic video and laparoscopic ultrasound for minimally invasive partial nephrectomy. Proc SPIE 7261:726109–726110CrossRef

10.

Cheung CL, Wedlake C, Moore J, Pautler SE, Peters TM (2010) Fused video and ultrasound images for minimally invasive partial nephrectomy: a phantom study. Med Image Comput Comput Assist Interv 13(Pt 3):408–415PubMed

11.

Ieiri S, Uemura M, Konishi K, Souzaki R, Nagao Y, Tsutsumi N, Akahoshi T, Ohuchida K, Ohdaira T, Tomikawa M, Tanoue K, Hashizume M, Taguchi T (2012) Augmented reality navigation system for laparoscopic splenectomy in children based on preoperative CT image using optical tracking device. Pediatr Surg Int 28(4):341–346PubMedCrossRef

12.

Teber D, Guven S, Simpfendörfer T, Baumhauer M, Güven EO, Yencilek F, Gözen AS, Rassweiler J (2009) Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. Eur Urol 56(2):332–338PubMedCrossRef

13.

Simpfendörfer T, Baumhauer M, Müller M, Gutt CN, Meinzer HP, Rassweiler JJ, Guven S, Teber D (2011) Augmented reality visualization during laparoscopic radical prostatectomy. J Endourol 25(12):1841–1845PubMedCrossRef

14.

Yaron A, Bar-Zohar M, Horesh N (2007) Miniature stereoscopic video system provides real-time 3D registration and image fusion for minimally invasive surgery. Proc SPIE 6490, Stereoscopic Displays and Virtual Reality Systems XIV 649009

15.

Hostettler A, George D, Rémond Y, Nicolau SA, Soler L, Marescaux J (2010) Bulk modulus and volume variation measurement of the liver and the kidneys in vivo using abdominal kinetics during free breathing. Comput Methods Progr Biomed 100(2):149–157CrossRef

16.

Hostettler A, Nicolau SA, Rémond Y, Marescaux J, Soler L (2010) A real-time predictive simulation of abdominal viscera positions during quiet free breathing. Prog Biophys Mol Biol 103(2–3):169–184PubMedCrossRef

17.

Shekhar R, Dandekar O, Bhat V, Philip M, Lei P, Godinez C, Sutton E, George I, Kavic S, Mezrich R, Park A (2010) Live augmented reality: a new visualization method for laparoscopic surgery using continuous volumetric computed tomography. Surg Endosc 24(8):1976–1985PubMedCrossRef

18.

Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334CrossRef

19.

Yaniv Z, Foroughi P, Kang HJ, Boctor E (2011) Ultrasound calibration framework for the image-guided surgery toolkit. Medical Imaging 2011: visualization, image-guided procedures, and modeling. Proc SPIE 7964:79641N–1–79641N–11CrossRef

20.

Lu F, Hartley R (2007) A fast optimal algorithm for L₂ triangulation. ACCV 2:279–288

21.

Chen TK, Heffter T, Lasso A, Pinter C (2011) Automated intraoperative calibration for prostate cancer brachytherapy. Med Phys 38(11):6285–6299CrossRef

Title: Stereoscopic augmented reality for laparoscopic surgery
Authors: Xin Kang
Mahdi Azizian
Emmanuel Wilson
Kyle Wu
Aaron D. Martin
Timothy D. Kane
Craig A. Peters
Kevin Cleary
Raj Shekhar
Publication date: 01-07-2014
Publisher: Springer US
Published in: Surgical Endoscopy / Issue 7/2014
Print ISSN: 0930-2794
Electronic ISSN: 1432-2218
DOI: https://doi.org/10.1007/s00464-014-3433-x

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Stereoscopic augmented reality for laparoscopic surgery

Abstract

Background

Methods

Results

Conclusions

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 7/2014

Effects of robotic manipulators on movements of novices and surgeons

Development of force-based metrics for skills assessment in minimally invasive surgery

Day-case versus inpatient laparoscopic fundoplication: outcomes, quality of life and cost-analysis

Size discrepancy between endoscopic size and pathologic size is not negligible in endoscopic resection for early gastric cancer

What should be the correct indications for laparoscopic inguinal hernia repair?

Peripancreatic pseudoaneurysms: a management-based classification system