Skip to main content
SpringerLink
Log in

Combining Visual Cues with Interactions for 3D–2D Registration in Liver Laparoscopy

  • Original Article
  • Published:
Annals of Biomedical Engineering Aims and scope Submit manuscript

Abstract

Augmented Reality (AR) in monocular liver laparoscopy requires one to register a preoperative 3D liver model to a laparoscopy image. This is a difficult problem because the preoperative shape may significantly differ from the unknown intraoperative shape and the liver is only partially visible in the laparoscopy image. Previous approaches are either manual, using a rigid model, or automatic, using visual cues and a biomechanical model. We propose a new approach called the hybrid approach combining the best of both worlds. The visual cues allow us to capture the machine perception while user interaction allows us to take advantage of the surgeon’s prior knowledge and spatial understanding of the patient anatomy. The registration accuracy and repeatability were evaluated on phantom, animal ex vivo and patient data respectively. The proposed registration outperforms the state of the art methods both in terms of accuracy and repeatability. An average registration error below the 1 cm oncologic margin advised in the literature for tumour resection in laparoscopy hepatectomy was obtained.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 13

Similar content being viewed by others

References

  1. Adagolodjo, Y., R. Trivisonne, N. Haouchine, S. Cotin, and H. Courtecuisse. Silhouette-based pose estimation for deformable organs application to surgical augmented reality. In: IROS, 2017

  2. Adams, F., T. Qiu, A. Mark, B. Fritz, L. Kramer, D. Schlager, U. Wetterauer, A. Miernik, and P. Fischer. Soft 3D-printed phantom of the human kidney with collecting system. Ann. Biomed. Eng. 45:963–972, 2017.

    Article  Google Scholar 

  3. AgiSoft PhotoScan Professional (Version 1.2.6) (Software), 2016

  4. Barr, A. H. Global and local deformations of solid primitives. Comput. Graph. 18(3):21–30, 1984.

    Article  Google Scholar 

  5. Bechmann, D., and D. Gerber. Arbitrary shaped deformations with dogme. Vis. Comput. 19:175–186, 2003.

    Article  Google Scholar 

  6. Bender, J., D. Koschier, P. Charrier, and D. Weber. Position-based simulation of continuous materials. Comput. Graph. 44:1–10, 2014.

    Article  Google Scholar 

  7. Bernhardt, S., S. Nicolau, A. Bartoli, V. Agnus, L. Soler, and C. Doignon. Using shading to register an intraoperative CT scan to a laparoscopic image. Computer-Assisted Robotic Endoscopy. CARE, 2015

  8. Clements, L., J. Collins, J. Weis, A. Simpson, T. Kingham, W. Jarnagin, and M. Miga. Deformation correction for image guided liver surgery: an intraoperative fidelity assessment. Surgery 162(3):537–547, 2017.

    Article  Google Scholar 

  9. Feng, J., J. Shao, X. Jin, Q. Peng, and R. Forrest. Multiresolution free-form deformations with subdivision surfaces of arbitrary topology. Vis. Comput. 22(1):28–42, 2005.

    Article  Google Scholar 

  10. Haouchine, N., F. Roy, L. Untereiner, and S. Cotin. Using contours as boundary conditions for elastic registration during minimally invasive hepatic surgery. In: IROS, 2016.

  11. Koo, B., E. Ozgur, B. Le Roy, E. Buc, and A. Bartoli Deformable registration of a preoperative 3D liver volume to a laparoscopy image using contour and shading cues. In: MICCAI, 2017

  12. Lamata, P., F. Lamata, V. Sojar, P. Makowski, L. Massoptier, S. Casciaro, W. Ali, T. Studeli, J. Declerck, O. Elle, and B. Edwin. Use of the Resection Map system as guidance during hepatectomy. Surg. Endosc. 24:2327–2337, 2010.

    Article  Google Scholar 

  13. Margonis, G., T. Sergentanis, I. Ntanasis-Stathopoulos, N. Andreatos, I. Tzanninis, K. Sasaki, T. Psaltopoulou, J. Wang, S. Buettner, A. Papalois, J. He, C. Wolfgang, T. Pawlik, and M. Weiss. Impact of surgical margin width on recurrence and overall survival following R0 hepatic resection of colorectal metastases: a systematic review and meta-analysis. Ann. Surg. 267(6):1047–1055, 2018.

    Article  Google Scholar 

  14. Moghari, M., B. Ma, and P. Abolmaesumi. A theoretical comparison of different target registration error estimators. In: MICCAI, 2008.

  15. Nicolau, S., L. Soler, D. Mutter, and J. Marescaux. Augmented reality in laparoscopic surgical oncology. Surg. Oncol. 20(3):189–201, 2011.

    Article  Google Scholar 

  16. Oktay, O., L. Zhang, T. Mansi, P. Mountney, P. Mewes, S. Nicolau, L. Soler, and C. Chefd'hotel. Biomechanically driven registration of pre- to intra-operative 3d images for laparoscopic surgery. In: MICCAI, 2013

  17. Ozgur, E., B. Koo, B. Le Roy, E. Buc, and A. Bartoli. Preoperative liver registration for augmented monocular laparoscopy using backward-forward biomechanical simulation. Int. J. Comput. Assist. Radiol. Surg. 13:1629–1640, 2018.

    Article  Google Scholar 

  18. Robu, M. R., J. Ramalhinho, S. Thompson, K. Gurusamy, B. Davidson, D. Hawkes, D. Stoyanov, and M. Clarkson. Global rigid registration of CT to video in laparoscopic liver surgery. Int. J. Comput. Assist. Radiol. Surg. 13:947–956, 2018.

    Article  Google Scholar 

  19. Sacht, L., E. Vouga, and A. Jacobson. Nested cages. ACM Trans. Graph. 34(6) Article No. 170, 2015.

  20. Sederberg, T. W., and S. R. Parry. Free-form deformation of solid geometric models. Comput. Graph. 20(4):151–160, 1986.

    Article  Google Scholar 

  21. Thompson, S., J. Totz, Y. Song, S. Johnsen, D. Stoyanov, S. Ourselin, K. Gurusamy, C. Schneider, B. Davidson, D. Hawkes, and M. Clarkson. Accuracy validation of an image guided laparoscopy system for liver resection. Proc. SPIE 9415(09):1–12, 2015.

    Google Scholar 

  22. Zhong, F. P., Y. J. Zhang, Y. Liu, and S. B. Zou. Prognostic impact of surgical margin in patients with hepatocellular carcinoma: a meta-analysis. Medicine 96(37):e8043, 2017.

    Article  Google Scholar 

Download references

Conflict of interest

The authors of this article declare no potential conflicts of interest.

Ethical Approval

All procedures involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This study is also supported by an ethical approval with ID IRB00008526-2019-CE58 issued by CPP Sud-Est VI in Clermont-Ferrand, France.

Informed Consent

Informed consent was obtained from the patients included in the study.

Author information

Authors and Affiliations

  1. EnCoV, Institut Pascal, UMR 6602 CNRS/Université Clermont-Auvergne, Clermont-Ferrand, France

    Yamid Espinel, Erol Özgür, Lilian Calvet, Bertrand Le Roy, Emmanuel Buc & Adrien Bartoli

  2. University Hospital of Clermont-Ferrand, Clermont-Ferrand, France

    Lilian Calvet & Emmanuel Buc

  3. University Hospital of Saint-Étienne, Saint-Étienne, France

    Bertrand Le Roy

  4. Faculté de Médecine, Batiment 3C, 28 place Henri Dunant, 63001, Clermont-Ferrand, France

    Yamid Espinel

Authors
  1. Yamid Espinel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Erol Özgür
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lilian Calvet
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bertrand Le Roy
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Emmanuel Buc
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Adrien Bartoli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yamid Espinel.

Additional information

Associate Editor Elena S. Di Martino oversaw the review of this article.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Espinel, Y., Özgür, E., Calvet, L. et al. Combining Visual Cues with Interactions for 3D–2D Registration in Liver Laparoscopy. Ann Biomed Eng 48, 1712–1727 (2020). https://doi.org/10.1007/s10439-020-02479-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10439-020-02479-z

Keywords

Search

Navigation