Skip to main content
Top

Open Access 17-06-2024 | Thrombectomy | Original Article

Autonomous navigation of catheters and guidewires in mechanical thrombectomy using inverse reinforcement learning

Authors: Harry Robertshaw, Lennart Karstensen, Benjamin Jackson, Alejandro Granados, Thomas C. Booth

Published in: International Journal of Computer Assisted Radiology and Surgery

Login to get access

Abstract

Purpose

Autonomous navigation of catheters and guidewires can enhance endovascular surgery safety and efficacy, reducing procedure times and operator radiation exposure. Integrating tele-operated robotics could widen access to time-sensitive emergency procedures like mechanical thrombectomy (MT). Reinforcement learning (RL) shows potential in endovascular navigation, yet its application encounters challenges without a reward signal. This study explores the viability of autonomous guidewire navigation in MT vasculature using inverse reinforcement learning (IRL) to leverage expert demonstrations.

Methods

Employing the Simulation Open Framework Architecture (SOFA), this study established a simulation-based training and evaluation environment for MT navigation. We used IRL to infer reward functions from expert behaviour when navigating a guidewire and catheter. We utilized the soft actor-critic algorithm to train models with various reward functions and compared their performance in silico.

Results

We demonstrated feasibility of navigation using IRL. When evaluating single- versus dual-device (i.e. guidewire versus catheter and guidewire) tracking, both methods achieved high success rates of 95% and 96%, respectively. Dual tracking, however, utilized both devices mimicking an expert. A success rate of 100% and procedure time of 22.6 s were obtained when training with a reward function obtained through ‘reward shaping’. This outperformed a dense reward function (96%, 24.9 s) and an IRL-derived reward function (48%, 59.2 s).

Conclusions

We have contributed to the advancement of autonomous endovascular intervention navigation, particularly MT, by effectively employing IRL based on demonstrator expertise. The results underscore the potential of using reward shaping to efficiently train models, offering a promising avenue for enhancing the accessibility and precision of MT procedures. We envisage that future research can extend our methodology to diverse anatomical structures to enhance generalizability.
Literature
1.
go back to reference Townsend N, Wilson L, Bhatnagar P, Wickramasinghe K, Rayner M, Nichols M (2016) Cardiovascular disease in europe: epidemiological update 2016. Eur Heart J 37(3232–3245):11 Townsend N, Wilson L, Bhatnagar P, Wickramasinghe K, Rayner M, Nichols M (2016) Cardiovascular disease in europe: epidemiological update 2016. Eur Heart J 37(3232–3245):11
2.
go back to reference Goyal M, Menon BK, Zwam WHV, Dippel DW, Mitchell PJ, Demchuk AM, Dávalos A, Majoie CB, Lugt AVD, Miquel MAD, Donnan GA, Roos YB, Bonafe A, Jahan R, Diener HC, Berg LAVD, Levy EI, Berkhemer OA, Pereira VM, Rempel J, Millán M, Davis SM, Roy D, Thornton J, Román LS, Ribó M, Beumer D, Stouch B, Brown S, Campbell BC, Oostenbrugge RJV, Saver JL, Hill MD, Jovin TG (2016) Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomised trials. The Lancet 387:1723–1731CrossRef Goyal M, Menon BK, Zwam WHV, Dippel DW, Mitchell PJ, Demchuk AM, Dávalos A, Majoie CB, Lugt AVD, Miquel MAD, Donnan GA, Roos YB, Bonafe A, Jahan R, Diener HC, Berg LAVD, Levy EI, Berkhemer OA, Pereira VM, Rempel J, Millán M, Davis SM, Roy D, Thornton J, Román LS, Ribó M, Beumer D, Stouch B, Brown S, Campbell BC, Oostenbrugge RJV, Saver JL, Hill MD, Jovin TG (2016) Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomised trials. The Lancet 387:1723–1731CrossRef
3.
go back to reference Vidale S, Agostoni E (2017) Endovascular treatment of ischemic stroke: an updated meta-analysis of efficacy and safety. Vasc Endovasc Surg 51:215–219CrossRef Vidale S, Agostoni E (2017) Endovascular treatment of ischemic stroke: an updated meta-analysis of efficacy and safety. Vasc Endovasc Surg 51:215–219CrossRef
4.
go back to reference Rha JH, Saver JL (2007) The impact of recanalization on ischemic stroke outcome: a meta-analysis. Stroke 38:967–973CrossRefPubMed Rha JH, Saver JL (2007) The impact of recanalization on ischemic stroke outcome: a meta-analysis. Stroke 38:967–973CrossRefPubMed
5.
go back to reference Berkhemer OA, Fransen PS, Beumer D, van den Berg LA, Lingsma HF, Yoo AJ, Schonewille WJ, Vos JA, Nederkoorn PJ, Wermer MJ, van Walderveen MA, Staals J, Hofmeijer J, van Oostayen JA, Nijeholt GJL, Boiten J, Brouwer PA, Emmer BJ, de Bruijn SF, van Dijk LC, Kappelle LJ, Lo RH, van Dijk EJ, de Vries J, de Kort PL, van Rooij WJJ, van den Berg JS, van Hasselt BA, Aerden LA, Dallinga RJ, Visser MC, Bot JC, Vroomen PC, Eshghi O, Schreuder TH, Heijboer RJ, Keizer K, Tielbeek AV, den Hertog HM, Gerrits DG, van den Berg-Vos RM, Karas GB, Steyerberg EW, Flach HZ, Marquering HA, Sprengers ME, Jenniskens SF, Beenen LF, van den Berg R, Koudstaal PJ, van Zwam WH, Roos YB, van der Lugt A, van Oostenbrugge RJ, Majoie CB, Dippel DW (2015) A randomized trial of intraarterial treatment for acute ischemic stroke. N Engl J Med 372(11–20):1 Berkhemer OA, Fransen PS, Beumer D, van den Berg LA, Lingsma HF, Yoo AJ, Schonewille WJ, Vos JA, Nederkoorn PJ, Wermer MJ, van Walderveen MA, Staals J, Hofmeijer J, van Oostayen JA, Nijeholt GJL, Boiten J, Brouwer PA, Emmer BJ, de Bruijn SF, van Dijk LC, Kappelle LJ, Lo RH, van Dijk EJ, de Vries J, de Kort PL, van Rooij WJJ, van den Berg JS, van Hasselt BA, Aerden LA, Dallinga RJ, Visser MC, Bot JC, Vroomen PC, Eshghi O, Schreuder TH, Heijboer RJ, Keizer K, Tielbeek AV, den Hertog HM, Gerrits DG, van den Berg-Vos RM, Karas GB, Steyerberg EW, Flach HZ, Marquering HA, Sprengers ME, Jenniskens SF, Beenen LF, van den Berg R, Koudstaal PJ, van Zwam WH, Roos YB, van der Lugt A, van Oostenbrugge RJ, Majoie CB, Dippel DW (2015) A randomized trial of intraarterial treatment for acute ischemic stroke. N Engl J Med 372(11–20):1
6.
go back to reference Saver JL, Goyal Lugt AVD, Menon BK, Majoie CB, Dippel DW, Campbell BC, Nogueira RG, Demchuk AM, Tomasello A, Cardona P, Devlin TG, Frei DF, Rochemont RDMD, Berkhemer OA, Jovin TG, Siddiqui AH, Zwam WHV, Davis SM, Castaño C, Sapkota BL, Fransen PS, Molina C, Oostenbrugge RJV, Ángel Chamorro, Lingsma H, Silver FL, Donnan GA, Shuaib A, Brown S, Stouch B, Mitchell PJ, Davalos A, Roos YB, Hill MD (2016) Time to treatment with endovascular thrombectomy and outcomes from ischemic stroke: a meta-analysis. JAMA J Am Med Assoc 316:1279–1288CrossRef Saver JL, Goyal Lugt AVD, Menon BK, Majoie CB, Dippel DW, Campbell BC, Nogueira RG, Demchuk AM, Tomasello A, Cardona P, Devlin TG, Frei DF, Rochemont RDMD, Berkhemer OA, Jovin TG, Siddiqui AH, Zwam WHV, Davis SM, Castaño C, Sapkota BL, Fransen PS, Molina C, Oostenbrugge RJV, Ángel Chamorro, Lingsma H, Silver FL, Donnan GA, Shuaib A, Brown S, Stouch B, Mitchell PJ, Davalos A, Roos YB, Hill MD (2016) Time to treatment with endovascular thrombectomy and outcomes from ischemic stroke: a meta-analysis. JAMA J Am Med Assoc 316:1279–1288CrossRef
7.
go back to reference McMeekin P, White P, James MA, Price CI, Flynn D, Ford GA (2017) Estimating the number of UK stroke patients eligible for endovascular thrombectomy. Eur Stroke J 2:319–326CrossRefPubMedPubMedCentral McMeekin P, White P, James MA, Price CI, Flynn D, Ford GA (2017) Estimating the number of UK stroke patients eligible for endovascular thrombectomy. Eur Stroke J 2:319–326CrossRefPubMedPubMedCentral
8.
go back to reference Programme SSNA (2023) Ssnap annual report 2023 [Online]. Available: www.hqip.org.uk/national-programmes Programme SSNA (2023) Ssnap annual report 2023 [Online]. Available: www.hqip.org.uk/national-programmes
9.
go back to reference Hausegger KA, Schedlbauer P, Deutschmann HA, Tiesenhausen K (2001) Complications in endoluminal repair of abdominal aortic aneurysms. Eur J Radiol 39:22–33CrossRefPubMed Hausegger KA, Schedlbauer P, Deutschmann HA, Tiesenhausen K (2001) Complications in endoluminal repair of abdominal aortic aneurysms. Eur J Radiol 39:22–33CrossRefPubMed
10.
go back to reference Rudnick MR, Goldfarb S, Wexler L, Ludbrook PA, Murphy MJ, Halpern EF, Hill JA, Winniford M, Cohen MB, VanFossen DB (1995) Nephrotoxicity of ionic and nonionic contrast media in 1196 patients: a randomized trial. Kidney Int 47:254–261CrossRefPubMed Rudnick MR, Goldfarb S, Wexler L, Ludbrook PA, Murphy MJ, Halpern EF, Hill JA, Winniford M, Cohen MB, VanFossen DB (1995) Nephrotoxicity of ionic and nonionic contrast media in 1196 patients: a randomized trial. Kidney Int 47:254–261CrossRefPubMed
11.
go back to reference Klein LW, Miller DL, Balter S, Laskey W, Haines D, Norbash A, Mauro MA, Goldstein JA (2009) Occupational health hazards in the interventional laboratory: time for a safer environment. Soc Interv Radiol 250:538–544 Klein LW, Miller DL, Balter S, Laskey W, Haines D, Norbash A, Mauro MA, Goldstein JA (2009) Occupational health hazards in the interventional laboratory: time for a safer environment. Soc Interv Radiol 250:538–544
12.
go back to reference Ho P, Cheng SW, Wu PM, Ting AC, Poon JT, Cheng CK, Mok JH, Tsang MS (2007) Ionizing radiation absorption of vascular surgeons during endovascular procedures. J Vasc Surg 46:455–459CrossRefPubMed Ho P, Cheng SW, Wu PM, Ting AC, Poon JT, Cheng CK, Mok JH, Tsang MS (2007) Ionizing radiation absorption of vascular surgeons during endovascular procedures. J Vasc Surg 46:455–459CrossRefPubMed
13.
go back to reference Madder RD, VanOosterhout S, Mulder A, Elmore M, Campbell J, Borgman A, Parker J, Wohns D (2017) Impact of robotics and a suspended lead suit on physician radiation exposure during percutaneous coronary intervention. Cardiovasc Revascularization Med 18:190–196CrossRef Madder RD, VanOosterhout S, Mulder A, Elmore M, Campbell J, Borgman A, Parker J, Wohns D (2017) Impact of robotics and a suspended lead suit on physician radiation exposure during percutaneous coronary intervention. Cardiovasc Revascularization Med 18:190–196CrossRef
14.
go back to reference Crinnion W, Jackson B, Sood A, Lynch J, Bergeles C, Liu H, Rhode K, Pereira VM, Booth TC (2022) Robotics in neurointerventional surgery: a systematic review of the literature. J Neurointerventional Surg 14:539–545CrossRef Crinnion W, Jackson B, Sood A, Lynch J, Bergeles C, Liu H, Rhode K, Pereira VM, Booth TC (2022) Robotics in neurointerventional surgery: a systematic review of the literature. J Neurointerventional Surg 14:539–545CrossRef
15.
go back to reference Riga CV, Cheshire NJ, Hamady MS, Bicknell CD (2010) The role of robotic endovascular catheters in fenestrated stent grafting. J Vasc Surg 51:810–820CrossRefPubMed Riga CV, Cheshire NJ, Hamady MS, Bicknell CD (2010) The role of robotic endovascular catheters in fenestrated stent grafting. J Vasc Surg 51:810–820CrossRefPubMed
17.
go back to reference Jackson B, Crinnion W, Reyzabal MDI, Robertshaw H, Bergeles C, Rhode K, Booth T (2023) Comparative verification of control methodology for robotic interventional neuroradiology procedures, International Journal of Computer Assisted Radiology and Surgery, 7. [Online]. Available: https://link.springer.com/10.1007/s11548-023-02991-2 Jackson B, Crinnion W, Reyzabal MDI, Robertshaw H, Bergeles C, Rhode K, Booth T (2023) Comparative verification of control methodology for robotic interventional neuroradiology procedures, International Journal of Computer Assisted Radiology and Surgery, 7. [Online]. Available: https://​link.​springer.​com/​10.​1007/​s11548-023-02991-2
18.
go back to reference Sarker IH (2021) Machine learning: algorithms, real-world applications and research directions. SN Comput Sci 2:5CrossRef Sarker IH (2021) Machine learning: algorithms, real-world applications and research directions. SN Comput Sci 2:5CrossRef
19.
go back to reference Fatima M, Pasha M (2017) Survey of machine learning algorithms for disease diagnostic. J Intell Learn Syst Appl 09:1–16 Fatima M, Pasha M (2017) Survey of machine learning algorithms for disease diagnostic. J Intell Learn Syst Appl 09:1–16
20.
go back to reference Silahtaroğlu G, Yılmaztürk N (2021) Data analysis in health and big data: a machine learning medical diagnosis model based on patients’ complaints. Commun Stat Theory Methods 50:1547–1556CrossRef Silahtaroğlu G, Yılmaztürk N (2021) Data analysis in health and big data: a machine learning medical diagnosis model based on patients’ complaints. Commun Stat Theory Methods 50:1547–1556CrossRef
21.
go back to reference Robertshaw H, Karstensen L, Jackson B, Sadati H, Rhode K, Ourselin S, Granados A, Booth TC (2023) Artificial intelligence in the autonomous navigation of endovascular interventions: a systematic review. Front Hum Neurosci 17:1239374CrossRefPubMedPubMedCentral Robertshaw H, Karstensen L, Jackson B, Sadati H, Rhode K, Ourselin S, Granados A, Booth TC (2023) Artificial intelligence in the autonomous navigation of endovascular interventions: a systematic review. Front Hum Neurosci 17:1239374CrossRefPubMedPubMedCentral
22.
go back to reference Sutton RS, Barto AG (1998) Introduction to reinforcement learning, 1st ed. MIT Press Sutton RS, Barto AG (1998) Introduction to reinforcement learning, 1st ed. MIT Press
23.
go back to reference Adams S, Cody T, Beling PA (2022) A survey of inverse reinforcement learning. Artif Intelli Rev 55:4307–4346CrossRef Adams S, Cody T, Beling PA (2022) A survey of inverse reinforcement learning. Artif Intelli Rev 55:4307–4346CrossRef
24.
go back to reference Chi W, Liu J, Abdelaziz MEMK, Dagnino G, Riga C, Bicknell C, Yang GZ (2018) Trajectory optimization of robot-assisted endovascularcatheterization with reinforcement learning. In: 2018 IEEE/RSJ International conference on intelligent robots and systems (IROS). IEEE, vol 8, pp 3875–3881 Chi W, Liu J, Abdelaziz MEMK, Dagnino G, Riga C, Bicknell C, Yang GZ (2018) Trajectory optimization of robot-assisted endovascularcatheterization with reinforcement learning. In: 2018 IEEE/RSJ International conference on intelligent robots and systems (IROS). IEEE, vol 8, pp 3875–3881
25.
go back to reference Behr T, Pusch TP, Siegfarth M, Hüsener D, Mörschel T, Karstensen L (2019) Deep reinforcement learning for the navigation of neurovascular catheters. Curr Dir Biomed Eng 5:5–8CrossRef Behr T, Pusch TP, Siegfarth M, Hüsener D, Mörschel T, Karstensen L (2019) Deep reinforcement learning for the navigation of neurovascular catheters. Curr Dir Biomed Eng 5:5–8CrossRef
26.
go back to reference Kweon J, Kim K, Lee C, Kwon H, Park J, Song K, Kim YI, Park J, Back I, Roh JH, Moon Y, Choi J, Kim YH (2021) Deep reinforcement learning for guidewire navigation in coronary artery phantom. IEEE Access 9:166409–166422CrossRef Kweon J, Kim K, Lee C, Kwon H, Park J, Song K, Kim YI, Park J, Back I, Roh JH, Moon Y, Choi J, Kim YH (2021) Deep reinforcement learning for guidewire navigation in coronary artery phantom. IEEE Access 9:166409–166422CrossRef
27.
go back to reference Ng AY, Russel S (2000) Algorithms for inverse reinforcement learning. Morgan Kaufmann Publishers Inc., pp 663–670 Ng AY, Russel S (2000) Algorithms for inverse reinforcement learning. Morgan Kaufmann Publishers Inc., pp 663–670
28.
go back to reference Haarnoja T, Zhou A, Abbeel P, Levine S (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: Dy J, Krause A (Eds.), Proceedings of the 35th international conference on machine learning, ser. Proceedings of machine learning research, vol. 80. PMLR, pp 1861–1870. [Online]. Available: https://proceedings.mlr.press/v80/haarnoja18b.html Haarnoja T, Zhou A, Abbeel P, Levine S (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: Dy J, Krause A (Eds.), Proceedings of the 35th international conference on machine learning, ser. Proceedings of machine learning research, vol. 80. PMLR, pp 1861–1870. [Online]. Available: https://​proceedings.​mlr.​press/​v80/​haarnoja18b.​html
29.
go back to reference Karstensen L, Ritter J, Hatzl J, Ernst F, Langejürgen J, Uhl C, Mathis-Ullrich F (2023) Recurrent neural networks for generalization towards the vessel geometry in autonomous endovascular guidewire navigation in the aortic arch. Int J Comput Assist Radiol Surg 18:1735–1744CrossRefPubMedPubMedCentral Karstensen L, Ritter J, Hatzl J, Ernst F, Langejürgen J, Uhl C, Mathis-Ullrich F (2023) Recurrent neural networks for generalization towards the vessel geometry in autonomous endovascular guidewire navigation in the aortic arch. Int J Comput Assist Radiol Surg 18:1735–1744CrossRefPubMedPubMedCentral
30.
go back to reference Faure F, Duriez C, Delingette H, Allard J, Gilles B, Marchesseau S, Talbot H, Courtecuisse H, Bousquet G, Peterlik I, Cotin S (2012) SOFA, a multi-model framework for interactive physical simulation. Springer Berlin Heidelberg, 2012. [Online]. Available: https://doi.org/10.1007/8415_2012_125 Faure F, Duriez C, Delingette H, Allard J, Gilles B, Marchesseau S, Talbot H, Courtecuisse H, Bousquet G, Peterlik I, Cotin S (2012) SOFA, a multi-model framework for interactive physical simulation. Springer Berlin Heidelberg, 2012. [Online]. Available: https://​doi.​org/​10.​1007/​8415_​2012_​125
31.
go back to reference Duriez C, Cotin S, Lenoir J, Neumann P (2006) New approaches to catheter navigation for interventional radiology simulation. Comput Aided Surg 11:300–308CrossRefPubMed Duriez C, Cotin S, Lenoir J, Neumann P (2006) New approaches to catheter navigation for interventional radiology simulation. Comput Aided Surg 11:300–308CrossRefPubMed
32.
go back to reference Community BO (2018) Blender—a 3D modelling and rendering package, Blender Foundation, Stichting Blender Foundation, Amsterdam. [Online]. Available: http://www.blender.org Community BO (2018) Blender—a 3D modelling and rendering package, Blender Foundation, Stichting Blender Foundation, Amsterdam. [Online]. Available: http://​www.​blender.​org
33.
go back to reference Karstensen L, Ritter J, Hatzl J, Pätz T, Langejürgen J, Uhl C, Mathis-Ullrich F (2022) Learning-based autonomous vascular guidewire navigation without human demonstration in the venous system of a porcine liver. Int J Comput Assist Radiol Surg 17:2033–2040CrossRefPubMedPubMedCentral Karstensen L, Ritter J, Hatzl J, Pätz T, Langejürgen J, Uhl C, Mathis-Ullrich F (2022) Learning-based autonomous vascular guidewire navigation without human demonstration in the venous system of a porcine liver. Int J Comput Assist Radiol Surg 17:2033–2040CrossRefPubMedPubMedCentral
34.
go back to reference Ziebart BD, Maas A, Bagnell JA, Dey AK (2008) Maximum entropy inverse reinforcement learning. AAAI Press, pp 1433–1438. [Online]. Available: www.aaai.org Ziebart BD, Maas A, Bagnell JA, Dey AK (2008) Maximum entropy inverse reinforcement learning. AAAI Press, pp 1433–1438. [Online]. Available: www.​aaai.​org
36.
go back to reference Beaman A Gautam, Peterson C, Kaneko N, Ponce L, Saber H, Khatibi K, Morales J, Kimball D, Lipovac JR, Narsinh KH, Baker A, Caton MT, Smith ER, Nour M, Szeder V, Jahan R, Colby GP, Cord BJ, Cooke DL, Tateshima S, Duckwiler G, Waldau B (2023) Robotic diagnostic cerebral angiography: A multicenter experience of 113 patients. J NeuroInterventional Surg Beaman A Gautam, Peterson C, Kaneko N, Ponce L, Saber H, Khatibi K, Morales J, Kimball D, Lipovac JR, Narsinh KH, Baker A, Caton MT, Smith ER, Nour M, Szeder V, Jahan R, Colby GP, Cord BJ, Cooke DL, Tateshima S, Duckwiler G, Waldau B (2023) Robotic diagnostic cerebral angiography: A multicenter experience of 113 patients. J NeuroInterventional Surg
Metadata
Title
Autonomous navigation of catheters and guidewires in mechanical thrombectomy using inverse reinforcement learning
Authors
Harry Robertshaw
Lennart Karstensen
Benjamin Jackson
Alejandro Granados
Thomas C. Booth
Publication date
17-06-2024
Publisher
Springer International Publishing
Published in
International Journal of Computer Assisted Radiology and Surgery
Print ISSN: 1861-6410
Electronic ISSN: 1861-6429
DOI
https://doi.org/10.1007/s11548-024-03208-w