Skip to main content
Top
Published in: Surgical Endoscopy 6/2018

01-06-2018 | Dynamic Manuscript

Mobile, real-time, and point-of-care augmented reality is robust, accurate, and feasible: a prospective pilot study

Authors: Hannes Götz Kenngott, Anas Amin Preukschas, Martin Wagner, Felix Nickel, Michael Müller, Nadine Bellemann, Christian Stock, Markus Fangerau, Boris Radeleff, Hans-Ulrich Kauczor, Hans-Peter Meinzer, Lena Maier-Hein, Beat Peter Müller-Stich

Published in: Surgical Endoscopy | Issue 6/2018

Login to get access

Abstract

Background

Augmented reality (AR) systems are currently being explored by a broad spectrum of industries, mainly for improving point-of-care access to data and images. Especially in surgery and especially for timely decisions in emergency cases, a fast and comprehensive access to images at the patient bedside is mandatory. Currently, imaging data are accessed at a distance from the patient both in time and space, i.e., at a specific workstation. Mobile technology and 3-dimensional (3D) visualization of radiological imaging data promise to overcome these restrictions by making bedside AR feasible.

Methods

In this project, AR was realized in a surgical setting by fusing a 3D-representation of structures of interest with live camera images on a tablet computer using marker-based registration. The intent of this study was to focus on a thorough evaluation of AR. Feasibility, robustness, and accuracy were thus evaluated consecutively in a phantom model and a porcine model. Additionally feasibility was evaluated in one male volunteer.

Results

In the phantom model (n = 10), AR visualization was feasible in 84% of the visualization space with high accuracy (mean reprojection error ± standard deviation (SD): 2.8 ± 2.7 mm; 95th percentile = 6.7 mm). In a porcine model (n = 5), AR visualization was feasible in 79% with high accuracy (mean reprojection error ± SD: 3.5 ± 3.0 mm; 95th percentile = 9.5 mm). Furthermore, AR was successfully used and proved feasible within a male volunteer.

Conclusions

Mobile, real-time, and point-of-care AR for clinical purposes proved feasible, robust, and accurate in the phantom, animal, and single-trial human model shown in this study. Consequently, AR following similar implementation proved robust and accurate enough to be evaluated in clinical trials assessing accuracy, robustness in clinical reality, as well as integration into the clinical workflow. If these further studies prove successful, AR might revolutionize data access at patient bedside.
Appendix
Available only for authorised users
Literature
1.
go back to reference Bellini HC, Sugiyama M, Shin M, Alam S, Takayama D (2016) Virtual & augmented reality understanding the race for the next computing platform, Jan 13, 2016 edn. The Goldman Sachs Group, Inc., New York City Bellini HC, Sugiyama M, Shin M, Alam S, Takayama D (2016) Virtual & augmented reality understanding the race for the next computing platform, Jan 13, 2016 edn. The Goldman Sachs Group, Inc., New York City
2.
go back to reference Wurmb T, Balling H, Frühwald P, Keil T, Kredel M, Meffert R, Roewer N, Brederlau J (2009) Polytrauma management in a period of change: time analysis of new strategies for emergency room treatment. Der Unfallchirurg 112:390–399CrossRefPubMed Wurmb T, Balling H, Frühwald P, Keil T, Kredel M, Meffert R, Roewer N, Brederlau J (2009) Polytrauma management in a period of change: time analysis of new strategies for emergency room treatment. Der Unfallchirurg 112:390–399CrossRefPubMed
3.
go back to reference Wurmb TE, Quaisser C, Balling H, Kredel M, Muellenbach R, Kenn W, Roewer N, Brederlau J (2011) Whole-body multislice computed tomography (MSCT) improves trauma care in patients requiring surgery after multiple trauma. Emerg Med J 28:300–304CrossRefPubMed Wurmb TE, Quaisser C, Balling H, Kredel M, Muellenbach R, Kenn W, Roewer N, Brederlau J (2011) Whole-body multislice computed tomography (MSCT) improves trauma care in patients requiring surgery after multiple trauma. Emerg Med J 28:300–304CrossRefPubMed
4.
go back to reference Hilbert P, zur Nieden K, Hofmann GO, Hoeller I, Koch R, Stuttmann R (2007) New aspects in the emergency room management of critically injured patients: a multi-slice CT-oriented care algorithm. Injury 38:552–558CrossRefPubMed Hilbert P, zur Nieden K, Hofmann GO, Hoeller I, Koch R, Stuttmann R (2007) New aspects in the emergency room management of critically injured patients: a multi-slice CT-oriented care algorithm. Injury 38:552–558CrossRefPubMed
5.
go back to reference Kay M, Santos J, Takane M (2011) mHealth: new horizons for health through mobile technologies. World Health Organization, Geneva Kay M, Santos J, Takane M (2011) mHealth: new horizons for health through mobile technologies. World Health Organization, Geneva
6.
go back to reference Mecheal PSD (2008) Towards the development of an mHealth strategy: a literature review. In: World Health Organization TMVPaTEIaCU. World Health Organization, Geneva Mecheal PSD (2008) Towards the development of an mHealth strategy: a literature review. In: World Health Organization TMVPaTEIaCU. World Health Organization, Geneva
7.
go back to reference John S, Poh AC, Lim TC, Chan EH (2012) The iPad tablet computer for mobile on-call radiology diagnosis? Auditing discrepancy in CT and MRI reporting. J Digit Imaging 25:628–634CrossRefPubMedPubMedCentral John S, Poh AC, Lim TC, Chan EH (2012) The iPad tablet computer for mobile on-call radiology diagnosis? Auditing discrepancy in CT and MRI reporting. J Digit Imaging 25:628–634CrossRefPubMedPubMedCentral
8.
go back to reference Johnson PT, Zimmerman SL, Heath D, Eng J, Horton KM, Scott WW, Fishman EK (2012) The iPad as a mobile device for CT display and interpretation: diagnostic accuracy for identification of pulmonary embolism. Emerg Radiol 19:323–327CrossRefPubMed Johnson PT, Zimmerman SL, Heath D, Eng J, Horton KM, Scott WW, Fishman EK (2012) The iPad as a mobile device for CT display and interpretation: diagnostic accuracy for identification of pulmonary embolism. Emerg Radiol 19:323–327CrossRefPubMed
9.
go back to reference Simpfendörfer T, Baumhauer M, Müller M, Gutt CN, Meinzer H-P, Rassweiler JJ, Guven S, Teber D (2011) Augmented reality visualization during laparoscopic radical prostatectomy. J Endourol 25:1841–1845CrossRefPubMed Simpfendörfer T, Baumhauer M, Müller M, Gutt CN, Meinzer H-P, Rassweiler JJ, Guven S, Teber D (2011) Augmented reality visualization during laparoscopic radical prostatectomy. J Endourol 25:1841–1845CrossRefPubMed
10.
go back to reference Teber D, Simpfendörfer T, Guven S, Baumhauer M, Gözen AS, Rassweiler J (2010) In-vitro evaluation of a soft-tissue navigation system for laparoscopic prostatectomy. J Endourol 24:1487–1491CrossRefPubMed Teber D, Simpfendörfer T, Guven S, Baumhauer M, Gözen AS, Rassweiler J (2010) In-vitro evaluation of a soft-tissue navigation system for laparoscopic prostatectomy. J Endourol 24:1487–1491CrossRefPubMed
11.
go back to reference Rassweiler JJ, Muller M, Fangerau M, Klein J, Goezen AS, Pereira P, Meinzer HP, Teber D (2012) iPad-assisted percutaneous access to the kidney using marker-based navigation: initial clinical experience. Eur Urol 61:628–631CrossRefPubMed Rassweiler JJ, Muller M, Fangerau M, Klein J, Goezen AS, Pereira P, Meinzer HP, Teber D (2012) iPad-assisted percutaneous access to the kidney using marker-based navigation: initial clinical experience. Eur Urol 61:628–631CrossRefPubMed
12.
go back to reference Baumhauer M, Simpfendörfer T, Müller-Stich BP, Teber D, Gutt CN, Rassweiler J, Meinzer HP, Wolf I (2008) Soft tissue navigation for laparoscopic partial nephrectomy. Int J Comput Assist Radiol Surg 3:307–314CrossRef Baumhauer M, Simpfendörfer T, Müller-Stich BP, Teber D, Gutt CN, Rassweiler J, Meinzer HP, Wolf I (2008) Soft tissue navigation for laparoscopic partial nephrectomy. Int J Comput Assist Radiol Surg 3:307–314CrossRef
13.
go back to reference Muller M, Rassweiler MC, Klein J, Seitel A, Gondan M, Baumhauer M, Teber D, Rassweiler JJ, Meinzer HP, Maier-Hein L (2013) Mobile augmented reality for computer-assisted percutaneous nephrolithotomy. Int J Comput Assist Radiol Surg 8:663–675CrossRefPubMed Muller M, Rassweiler MC, Klein J, Seitel A, Gondan M, Baumhauer M, Teber D, Rassweiler JJ, Meinzer HP, Maier-Hein L (2013) Mobile augmented reality for computer-assisted percutaneous nephrolithotomy. Int J Comput Assist Radiol Surg 8:663–675CrossRefPubMed
14.
go back to reference Kenngott HG, Wunscher JJ, Wagner M, Preukschas A, Wekerle AL, Neher P, Suwelack S, Speidel S, Nickel F, Oladokun D, Maier-Hein L, Dillmann R, Meinzer HP, Muller-Stich BP (2015) OpenHELP (Heidelberg laparoscopy phantom): development of an open-source surgical evaluation and training tool. Surg Endosc 29:3338–3347CrossRefPubMedPubMedCentral Kenngott HG, Wunscher JJ, Wagner M, Preukschas A, Wekerle AL, Neher P, Suwelack S, Speidel S, Nickel F, Oladokun D, Maier-Hein L, Dillmann R, Meinzer HP, Muller-Stich BP (2015) OpenHELP (Heidelberg laparoscopy phantom): development of an open-source surgical evaluation and training tool. Surg Endosc 29:3338–3347CrossRefPubMedPubMedCentral
15.
go back to reference Nolden M, Zelzer S, Seitel A, Wald D, Muller M, Franz AM, Maleike D, Fangerau M, Baumhauer M, Maier-Hein L, Maier-Hein KH, Meinzer HP, Wolf I (2013) The medical imaging interaction toolkit: challenges and advances: 10 years of open-source development. Int J Comput Assist Radiol Surg 8:607–620CrossRefPubMed Nolden M, Zelzer S, Seitel A, Wald D, Muller M, Franz AM, Maleike D, Fangerau M, Baumhauer M, Maier-Hein L, Maier-Hein KH, Meinzer HP, Wolf I (2013) The medical imaging interaction toolkit: challenges and advances: 10 years of open-source development. Int J Comput Assist Radiol Surg 8:607–620CrossRefPubMed
16.
go back to reference Clutton RE, Blissitt KJ, Bradley AA, Camburn MA (1997) Comparison of three injectable anaesthetic techniques in pigs. Vet Rec 141:140–146CrossRefPubMed Clutton RE, Blissitt KJ, Bradley AA, Camburn MA (1997) Comparison of three injectable anaesthetic techniques in pigs. Vet Rec 141:140–146CrossRefPubMed
17.
go back to reference Deng W, Li F, Wang M, Song Z (2013) Easy-to-use augmented reality neuronavigation using a wireless tablet PC. Stereotact Funct Neurosurg 92:17–24CrossRefPubMed Deng W, Li F, Wang M, Song Z (2013) Easy-to-use augmented reality neuronavigation using a wireless tablet PC. Stereotact Funct Neurosurg 92:17–24CrossRefPubMed
18.
go back to reference Kenngott H, Neuhaus J, Müller-Stich B, Wolf I, Vetter M, Meinzer H-P, Köninger J, Büchler M, Gutt C (2008) Development of a navigation system for minimally invasive esophagectomy. Surg Endosc 22:1858–1865CrossRefPubMed Kenngott H, Neuhaus J, Müller-Stich B, Wolf I, Vetter M, Meinzer H-P, Köninger J, Büchler M, Gutt C (2008) Development of a navigation system for minimally invasive esophagectomy. Surg Endosc 22:1858–1865CrossRefPubMed
19.
go back to reference Nickel F, Kenngott HG, Neuhaus J, Sommer CM, Gehrig T, Kolb A, Gondan M, Radeleff BA, Schaible A, Meinzer H-P (2013) Navigation system for minimally invasive esophagectomy: experimental study in a porcine model. Surg Endosc 27:3663–3670CrossRefPubMed Nickel F, Kenngott HG, Neuhaus J, Sommer CM, Gehrig T, Kolb A, Gondan M, Radeleff BA, Schaible A, Meinzer H-P (2013) Navigation system for minimally invasive esophagectomy: experimental study in a porcine model. Surg Endosc 27:3663–3670CrossRefPubMed
20.
go back to reference Cash DM, Miga MI, Glasgow SC, Dawant BM, Clements LW, Cao Z, Galloway RL, Chapman WC (2007) Concepts and preliminary data toward the realization of image-guided liver surgery. J Gastrointest Surg 11:844–859CrossRefPubMed Cash DM, Miga MI, Glasgow SC, Dawant BM, Clements LW, Cao Z, Galloway RL, Chapman WC (2007) Concepts and preliminary data toward the realization of image-guided liver surgery. J Gastrointest Surg 11:844–859CrossRefPubMed
21.
go back to reference Carter TJ, Sermesant M, Cash DM, Barratt DC, Tanner C, Hawkes DJ (2005) Application of soft tissue modelling to image-guided surgery. Med Eng Phys 27:893–909CrossRefPubMed Carter TJ, Sermesant M, Cash DM, Barratt DC, Tanner C, Hawkes DJ (2005) Application of soft tissue modelling to image-guided surgery. Med Eng Phys 27:893–909CrossRefPubMed
22.
go back to reference Ukimura O, Gill IS (2008) Imaging-assisted endoscopic surgery: Cleveland clinic experience. J Endourol 22:803–810CrossRefPubMed Ukimura O, Gill IS (2008) Imaging-assisted endoscopic surgery: Cleveland clinic experience. J Endourol 22:803–810CrossRefPubMed
23.
go back to reference Kenngott HG, Wagner M, Gondan M, Nickel F, Nolden M, Fetzer A, Weitz J, Fischer L, Speidel S, Meinzer H-P, Böckler D, Büchler MW, Müller-Stich BP (2013) Real-time image guidance in laparoscopic liver surgery: first clinical experience with a guidance system based on intraoperative CT imaging. Surg Endosc 28:933–940CrossRefPubMed Kenngott HG, Wagner M, Gondan M, Nickel F, Nolden M, Fetzer A, Weitz J, Fischer L, Speidel S, Meinzer H-P, Böckler D, Büchler MW, Müller-Stich BP (2013) Real-time image guidance in laparoscopic liver surgery: first clinical experience with a guidance system based on intraoperative CT imaging. Surg Endosc 28:933–940CrossRefPubMed
24.
go back to reference Weidert S, Wang L, von der Heide A, Navab N, Euler E (2012) Intraoperative augmented reality visualization. Current state of development and initial experiences with the CamC. Der Unfallchirurg 115:209–213CrossRefPubMed Weidert S, Wang L, von der Heide A, Navab N, Euler E (2012) Intraoperative augmented reality visualization. Current state of development and initial experiences with the CamC. Der Unfallchirurg 115:209–213CrossRefPubMed
25.
go back to reference Shekhar R, Dandekar O, Bhat V, Philip M, Lei P, Godinez C, Sutton E, George I, Kavic S, Mezrich R (2010) Live augmented reality: a new visualization method for laparoscopic surgery using continuous volumetric computed tomography. Surg Endosc 24:1976–1985CrossRefPubMed Shekhar R, Dandekar O, Bhat V, Philip M, Lei P, Godinez C, Sutton E, George I, Kavic S, Mezrich R (2010) Live augmented reality: a new visualization method for laparoscopic surgery using continuous volumetric computed tomography. Surg Endosc 24:1976–1985CrossRefPubMed
26.
go back to reference Hirai N, Kosaka A, Kawamata T, Hori T, Iseki H (2005) Image-guided neurosurgery system integrating AR-based navigation and open-MRI monitoring. Comput Aided Surg 10:59–72CrossRefPubMed Hirai N, Kosaka A, Kawamata T, Hori T, Iseki H (2005) Image-guided neurosurgery system integrating AR-based navigation and open-MRI monitoring. Comput Aided Surg 10:59–72CrossRefPubMed
27.
go back to reference Nicolau S, Soler L, Mutter D, Marescaux J (2011) Augmented reality in laparoscopic surgical oncology. Surg Oncol 20:189–201CrossRefPubMed Nicolau S, Soler L, Mutter D, Marescaux J (2011) Augmented reality in laparoscopic surgical oncology. Surg Oncol 20:189–201CrossRefPubMed
28.
go back to reference Baumhauer M, Feuerstein M, Meinzer H-P, Rassweiler J (2008) Navigation in endoscopic soft tissue surgery: perspectives and limitations. J Endourol 22:751–766CrossRefPubMed Baumhauer M, Feuerstein M, Meinzer H-P, Rassweiler J (2008) Navigation in endoscopic soft tissue surgery: perspectives and limitations. J Endourol 22:751–766CrossRefPubMed
29.
go back to reference Oizumi H, Kato H, Watarai H, Sadahiro M (2013) Three-dimensional computed tomography image overlay facilitates thoracoscopic trocar placement. J Thorac Cardiovasc Surg 146:720–721CrossRefPubMed Oizumi H, Kato H, Watarai H, Sadahiro M (2013) Three-dimensional computed tomography image overlay facilitates thoracoscopic trocar placement. J Thorac Cardiovasc Surg 146:720–721CrossRefPubMed
Metadata
Title
Mobile, real-time, and point-of-care augmented reality is robust, accurate, and feasible: a prospective pilot study
Authors
Hannes Götz Kenngott
Anas Amin Preukschas
Martin Wagner
Felix Nickel
Michael Müller
Nadine Bellemann
Christian Stock
Markus Fangerau
Boris Radeleff
Hans-Ulrich Kauczor
Hans-Peter Meinzer
Lena Maier-Hein
Beat Peter Müller-Stich
Publication date
01-06-2018
Publisher
Springer US
Published in
Surgical Endoscopy / Issue 6/2018
Print ISSN: 0930-2794
Electronic ISSN: 1432-2218
DOI
https://doi.org/10.1007/s00464-018-6151-y

Other articles of this Issue 6/2018

Surgical Endoscopy 6/2018 Go to the issue