Top

BMC Medical Informatics and Decision Making

Published in:

Open Access 01-12-2023 | Artificial Intelligence | Research

Ethics and governance of trustworthy medical artificial intelligence

Authors: Jie Zhang, Zong-ming Zhang

Published in: BMC Medical Informatics and Decision Making | Issue 1/2023

Abstract

Background

The growing application of artificial intelligence (AI) in healthcare has brought technological breakthroughs to traditional diagnosis and treatment, but it is accompanied by many risks and challenges. These adverse effects are also seen as ethical issues and affect trustworthiness in medical AI and need to be managed through identification, prognosis and monitoring.

Methods

We adopted a multidisciplinary approach and summarized five subjects that influence the trustworthiness of medical AI: data quality, algorithmic bias, opacity, safety and security, and responsibility attribution, and discussed these factors from the perspectives of technology, law, and healthcare stakeholders and institutions. The ethical framework of ethical values-ethical principles-ethical norms is used to propose corresponding ethical governance countermeasures for trustworthy medical AI from the ethical, legal, and regulatory aspects.

Results

Medical data are primarily unstructured, lacking uniform and standardized annotation, and data quality will directly affect the quality of medical AI algorithm models. Algorithmic bias can affect AI clinical predictions and exacerbate health disparities. The opacity of algorithms affects patients’ and doctors’ trust in medical AI, and algorithmic errors or security vulnerabilities can pose significant risks and harm to patients. The involvement of medical AI in clinical practices may threaten doctors ‘and patients’ autonomy and dignity. When accidents occur with medical AI, the responsibility attribution is not clear. All these factors affect people’s trust in medical AI.

Conclusions

In order to make medical AI trustworthy, at the ethical level, the ethical value orientation of promoting human health should first and foremost be considered as the top-level design. At the legal level, current medical AI does not have moral status and humans remain the duty bearers. At the regulatory level, strengthening data quality management, improving algorithm transparency and traceability to reduce algorithm bias, and regulating and reviewing the whole process of the AI industry to control risks are proposed. It is also necessary to encourage multiple parties to discuss and assess AI risks and social impacts, and to strengthen international cooperation and communication.

Schwab K. The Fourth Industrial Revolution: what it means and how to respond. World Economic Forum. 2016. https://www.weforum.org/agenda/2016/01/the-fourth-industrial-revolution-what-it-means-and-how-to-respond/. Accessed 15 Jul 2022.

Keskinbora KH. Medical ethics considerations on artificial intelligence. J Clin Neurosci. 2019;64:277–82. https://doi.org/10.1016/j.jocn.2019.03.001.CrossRef

Attia ZI, Kapa S, Lopez-Jimenez F, et al. Screening for cardiac contractile dysfunction using an artificial intelligence-enabled electrocardiogram. Nat Med. 2019;25(1):70–4. https://doi.org/10.1038/s41591-018-0240-2.CrossRef

Alvarez-Machancoses O, Fernandez-Martinez JL. Using artificial intelligence methods to speed up drug discovery. Expert Opin Drug Discov. 2019;14(8):769–77. https://doi.org/10.1080/17460441.2019.1621284.CrossRef

Watson DS, Krutzinna J, Bruce IN, et al. Clinical applications of machine learning algorithms: beyond the black box. BMJ. 2019;364:1886. https://doi.org/10.1136/bmj.l886.CrossRef

Markets and Markets.(2020). Artificial Intelligence in Healthcare Market with Covide-19 Impact Analysis by Offering (Hardware, Software, Services), Technology (Machine Learning, NLP, Context-Aware Computing, Computer Vision), End-Use Application, End User and Region-Global Forecast to 2026. 2021.https://www.marketsandmarkets.com/Market-Reports/artificial-intelligence-healthcare-market-54679303.html. Accessed 23 Jul 2022.

Katznelson G, Gerke S. The need for health AI ethics in medical school education. Adv Health Sci Educ. 2021;26:1447–58. https://doi.org/10.1007/s10459-021-10040-3.CrossRef

Ching T, Himmelstein DS, Beaulieu-Jones BK, et al. Opportunities and obstacles for deep learning in biology and medicine. J R Soc Interface. 2018;15(141):20170387. https://doi.org/10.1098/rsif.2017.0387.CrossRef

Mintz Y, Brodie R. Introduction to artificial intelligence in medicine. Minim Invasive Ther Allied Technol. 2019;28(2):73–81. https://doi.org/10.1080/13645706.2019.1575882.CrossRef

10.

AMA (American Medical Association). Report of the Council on Long Range Planning and Development. 2018. https://www.ama-assn.org/system/files/2018-11/a18-clrpd-reports.pdf. Accessed 23 Jul 2022.

11.

Fogel AL, Kvedar JC. Artificial intelligence powers digital medicine. NPJ Digit Med. 2018;1(1):5. https://doi.org/10.1038/s41746-017-0012-2.CrossRef

12.

Fraser H, Coiera E, Wong D. Safety of patient-facing digital symptom checkers. Lancet. 2018;392(10161):2263–4. https://doi.org/10.1016/S0140-6736(18)32819-8.CrossRef

13.

Collingridge D. The social control of technology. London: Frances Pinter; 1980.

14.

Guo R. The ethics and governance of artificial intelligence. Beijing: Law Press; 2020. p. 42.

15.

Floridi L. Translating principles into practices of digital ethics: five risks of being unethical. Philos Technol. 2019;32(2):185–93. https://doi.org/10.1007/s13347-019-00354-x.CrossRef

16.

Simpson TW. What is trust? Pac Philos Q. 2012;93(4):550–69. https://doi.org/10.1111/j.1468-0114.2012.01438.x.CrossRef

17.

Gerke S, Minssen T, Yu H, et al. Ethical and legal issues of ingestible electronic sensors. Nat Electron. 2019;2(8):329–34. https://doi.org/10.1038/s41928-019-0290-6.CrossRef

18.

Johnson SLJ. AI, machine learning, and ethics in health care. J Leg Med. 2020;39:427–41. https://doi.org/10.1080/01947648.2019.1690604.CrossRef

19.

IBM’s Watson supercomputer recommended ‘unsafe and incorrect’ cancer treatments, internal documents show. 2018. https://www.statnews.com/2018/07/25/ibm-watson-recommended-unsafe-incorrect-treatments/. Accessed 23 Jul 2022.

20.

Wang H, Meng XF, Wang Q, et al. Strategy for management and quality control of datasets in artificial intelligence medical device. China Med Devices. 2018;33(12):1–5.

21.

Xu Y, Ma XM, Yue Q, et al. Ethics of lung cancer image data and artificial intelligence. Med Soc. 2021;34(5):100–104

22.

Markowetz A, Błaszkiewicz K, Montag C, et al. Psycho-informatics: big data shaping modern psychometrics. Med Hypotheses. 2014;82(4):405–11. https://doi.org/10.1016/j.mehy.2013.11.030.CrossRef

23.

Char DS, Shah NH, Magnus D. Implementing machine learning in health care-addressing ethical challenges. N Engl J Med. 2018;378(11):981–3. https://doi.org/10.1056/NEJMp1714229.CrossRef

24.

Sharon T. Self-Tracking for health and the quantified self: re-articulating autonomy, solidarity, and authenticity in an age of personalized healthcare. Philos Technol. 2017;30:93–121. https://doi.org/10.1007/s13347-016-0215-5.CrossRef

25.

Monteith S, Glenn T. Automated decision-making and big data: concerns for people with mental illness. Curr Psychiatry Rep. 2016;18(12):112. https://doi.org/10.1007/s11920-016-0746-6.CrossRef

26.

Howard A, Borenstein J. The ugly truth about ourselves and our robot creations: the problem of bias and social inequity. Sci Eng Eth. 2018;24(5):1521–36. https://doi.org/10.1007/s11948-017-9975-2.CrossRef

27.

Mooney SJ, Pejaver V. Big data in public health: terminology, machine learning, and privacy. Annu Rev Public Health. 2018;39(1):95–112. https://doi.org/10.1146/annurev-publhealth-040617-014208.CrossRef

28.

Senders JT, Zaki MM, Karhade AV, et al. An introduction and overview of machine learning in neurosurgical care. Acta Neurochir (Wien). 2018;160(1):29–38. https://doi.org/10.1007/s00701-017-3385-8.CrossRef

29.

Adamson AS, Smith A. Machine learning and health care disparities in dermatology. JAMA Dermatol. 2018;154:1247–8. https://doi.org/10.1001/jamadermatol.2018.2348.CrossRef

30.

Obermeyer Z, Powers B, Vogeli C, et al. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366:447–53. https://doi.org/10.1126/science.aax2342.CrossRef

31.

Price W, Nicholson II. Medical AI and contextual bias. Harv J Law Technol. 2019;33:65–116.

32.

Burrell J. How the machine “thinks”: understanding opacity in machine learning algorithms. Big Data Soc. 2016. https://doi.org/10.1177/2053951715622512.CrossRef

33.

Tibshirani R. Regression shrinkage and selection via the lasso. J Royal Stat Soc Ser B (Methodol). 1996;58:267–88. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x.CrossRef

34.

AI in the UK: ready, willing and able? United Kingdom: authority of the house of lords. 2018. https://publications.parliament.uk/pa/ld201719/ldselect/ldai/100/100.pdf. Accessed 25 Jul 2022.

35.

Silver D, Schrittwieser J, Simonyan K, et al. Mastering the game of go without human knowledge. Nature. 2017;550(7676):354–9. https://doi.org/10.1038/nature24270.CrossRef

36.

Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542(7639):115–8. https://doi.org/10.1038/nature21056.CrossRef

37.

Walsh CG, Ribeiro JD, Franklin JC. Predicting risk of suicide attempts over time through machine learning. Clin Psychol Sci. 2017;5(3):457–69. https://doi.org/10.1177/2167702617691560.CrossRef

38.

Grote T, Berens P. On the ethics of algorithmic decision-making in healthcare. J Med Eth. 2020;46(3):205–11. https://doi.org/10.1136/medethics-2019-105586.CrossRef

39.

Liu LL, He YM, Liu XD. A survey on patients perceptions and trust in artificial intelligence healthcare. Chin Med Eth. 2019;32(8):986–90. https://doi.org/10.12026/j.issn.1001-8565.2019.08.07.CrossRef

40.

Feldman R, Aldana E, Stein K. Artificial intelligence in the health care space: how we can trust what we cannot know. Stanf Law Policy Rev. 2019;30:399–419.

41.

Kamensky S. Artificial intelligence and technology in health care: overview and possible legal implications. DePaul J Health Care Law. 2020;21(3):1–13.

42.

Adadi A, Berrada M. Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access. 2018;6:52138–60. https://doi.org/10.1109/ACCESS.2018.2870052.CrossRef

43.

Keane MT, Kenny EM. The twin-system approach as one generic solution for XAI: an overview of ANN-CBR twins for explaining deep learning. arXiv. 2019. https://doi.org/10.48550/arXiv.1905.08069

44.

Madumal P, Miller T, Vetere F, et al. Towards a grounded dialog model for explainable artificial intelligence. arXiv. 2018. https://arxiv.org/pdf/1806.08055.pdf

45.

Fauw JD, Ledsam JR, Romera-Paredes B, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med. 2018;24(9):1342–50. https://doi.org/10.1038/s41591-018-0107-6.CrossRef

46.

Holzinger A, Biemann C, Pattichis CS. What Do We Need to Build Explainable AI Systems for the Medical Domain?. arXiv, 2017. https://doi.org/10.48550/arXiv.1712.09923

47.

O’Sullivan S, Nevejans N, Allen C, et al. Legal, regulatory, and ethical frameworks for develop-ment of standards in artificial intelligence(AI) and autonomous robotic surgery. Int J Med Robot. 2019;15(1):e1968. https://doi.org/10.1002/rcs.1968.CrossRef

48.

Alemzadeh H, Raman J, Leveson N, et al. Adverse events in robotic surgery: a retrospective study of 14 years of FDA data. PLoS ONE. 2016;11(4):e0151470. https://doi.org/10.1371/journal.pone.0151470.CrossRef

49.

Xu QA. UK's first robotic heart valve surgery: “Machine storms away”, patient dies without treatment. 2018. https://www.guancha.cn/internation/2018_11_08_478891.shtml?s=zwyzxw. Accessed 26 Jul 2022.

50.

Macnish K. Unblinking eyes: the ethics of automating surveillance. Eth Inf Technol. 2012;14:151–67. https://doi.org/10.1007/s10676-012-9291-0.CrossRef

51.

U.S. Food and Drug Administration. Zimmer Biomet Recalls ROSA One 3.1 Brain Application Due to Error in Software.2021. https://www.fda.gov/medical-devices/medical-device-recalls/zimmer-biomet-recalls-rosa-one-31-brain-application-due-error-software. Accessed 26 Jul 2022.

52.

Kelly CJ, Karthikesalingam A, Suleyman M, et al. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 2019;17(1):195. https://doi.org/10.1186/s12916-019-1426-2.CrossRef

53.

Price WN, Gerke S, Cohen IG. Potential liability for physicians using artificial intelligence. JAMA. 2019;322:1765–6. https://doi.org/10.1001/jama.2019.15064.CrossRef

54.

Price WN, Gerke S, Cohen IG. How much can potential jurors tell us about liability for medical artificial intelligence? J Nucl Med. 2021;62(1):15–6. https://doi.org/10.2967/jnumed.120.257196.CrossRef

55.

Sharkey A. Robots and human dignity: a consideration of the effects of robot care on the dignity of older people. Eth Inf Technol. 2014;16(1):63–75. https://doi.org/10.1007/s10676-014-9338-5.CrossRef

56.

Challen R, Denny J, Pitt M, et al. Artificial intelligence, bias and clinical safety. BMJ Qual Saf. 2019;28(3):231–7. https://doi.org/10.1136/bmjqs-2018-008370.CrossRef

57.

Finlayson SG, Bowers JD, Ito J, et al. Adversarial attacks on medical machine learning. Science. 2019;363(6433):1287–9. https://doi.org/10.1126/science.aaw4399.CrossRef

58.

The Economist. The World's Most Valuable Resource is No Longer Oil, But Data. 2021. https://www.economist.com/leaders/2017/05/06/the-worlds-most-valuable-resource-is-no-longer-oil-but-data. Accessed 26 Jul 2022.

59.

Zhang F. Chinese imaging AI firm hacked, coronavirus research sold online. 2020. https://cntechpost.com/2020/04/27/chinese-imaging-ai-firm-hacked-coronavirus-research-sold-online/. Accessed 28 Jul 2022.

60.

McDougall RJ. Computer knows best? The need for value-flexibility in medical AI. J Med Eth. 2019;45(3):156–60. https://doi.org/10.1136/medethics-2018-105118.CrossRef

61.

Warwick K, Shah H. Passing the turing test does not mean the end of humanity. Cognit Comput. 2016;8(3):409–19.CrossRef

62.

European Commission. European Parliament, Committee on Legal Affairs: Draft Report with Recommendations to the Commission on Civil Law Rules on Robotics. 2016. http://www.europarl.europa.eu/sides/getDoc.do?pubRef=-//EP//NONSGML%2BCOMPARL%2BPE-582.443%2B01%2BDOC%2BPDF%2BV0//EN. Accessed 28 Jul 2022.

63.

Gan SP. How can robots have rights. Eth Res. 2017;3:126–30. https://doi.org/10.3969/j.issn.1671-9115.2017.03.025.CrossRef

64.

Sparrow R. Killer robots. J Appl Philos. 2007;24(1):62–77. https://doi.org/10.1111/j.1468-5930.2007.00346.x.CrossRef

65.

Coeckelbergh M. Artificial intelligence, responsibility attribution, and a relational justification of explainability. Sci Eng Eth. 2020;26(4):2051–68. https://doi.org/10.1007/s11948-019-00146-8.CrossRef

66.

Jessica SA. From Jeopardy to Jaundice: the medical liability implications of Dr. Watson and other artificial intelligence systems. La Law Rev. 2013;73:1049.

67.

Chung J, Zink A. Hey Watson, can i sue you for malpractice? Examining the liability of artificial intelligence in medicine. Asia Pac J Health Law Eth. 2018;11(2):51–80.

68.

Weaver JF, Zheng ZF. How to sue a robot: liability and AI. Law Econ. 2019;1:140–60.

69.

Zhang L, Chen P. Critique of the theory of robot “personality” and the legal regulation of artificial intelligent objects. Academia. 2018;12:53–75. https://doi.org/10.3969/j.issn.1002-1698.2018.12.006.CrossRef

70.

Gitlin JM. NHTSA data shows Teslas using Autopilot crashed 273 times in less than a year. 2022. https://arstechnica.com/cars/2022/06/teslas-using-autopilot-crashed-273-times-in-less-than-a-year/. Accessed 28 Jul 2022.

71.

IEEE. Ethically Aligned Design: A Vision for Prioritizing Human Well-being with Autonomous and Intelligent Systems, First Edition. 2016. https://ethicsinaction.ieee.org/. Accessed 2 Dec 2022.

72.

Future of life institute. Asilomar AI Principles. 2017. https://futureoflife.org/open-letter/asilomar-ai-principles/. Accessed 2 Dec 2022.

73.

The High-Level Expert Group on AI. Ethics guidelines for trustworthy AI. 2019. https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai. Accessed 2 Dec 2022.

74.

National Professional Committee on New Generation Artificial Intelligence Governance. Next Generation AI Governance Principles-Developing Responsible AI. 2019. http://www.gov.cn/xinwen/2019-06/17/content_5401006.htm. Accessed 2 Dec 2022.

75.

WHO. Ethics and Governance of Artificial Intelligence for Health. 2021. https://www.linking-ai-principles.org/cnprinciples. Accessed 2 Dec 2022.

76.

Beauchamp T, Childress J. Principles of biomedical ethics. New York: Oxford University Press; 2013.

77.

Wallach W, Allen C, Smit I. Machine morality: bottom-up and top-down approaches for modelling human moral faculties. AI Soc. 2008;22(4):565–82. https://doi.org/10.1007/s00146-007-0099-0.CrossRef

78.

Reddy S, Allan S, Coghlan S, et al. A governance model for the application of AI in health care. J Am Med Inform Assoc. 2019;27(3):491–7. https://doi.org/10.1093/jamia/ocz192.CrossRef

79.

Floridi L, Peng C. Faultless responsibility: on the nature and allocation of moral responsibility for distributed moral actions. Philos Trans R Soc A Math Phys Eng Sci. 2018;374(2083):0112. https://doi.org/10.2139/ssrn.3835211.CrossRef

80.

Calo R. Artificial intelligence policy: a primer and roadmap. UC Davis Law Rev. 2017;2:399–436. https://doi.org/10.2139/ssrn.3015350.CrossRef

81.

Deep Learning Assisted Decision-Making Medical Device Software Approval Points. China National Medical Products Administration [EB/OL] (2019–7–3). https://www.cmde.org.cn/xwdt/shpgzgg/gztg/20190703141714991.html

82.

Macnish K, Gauttier S. A pre-occupation with possession: the (non-) ownership of personal data. Big data and democracy. Edinburgh: Edinburgh University Press; 2020. p. 42–56.

83.

Vayena E, Blasimme A, Cohen IG. Machine learning in medicine: addressing ethical challenges. PLoS Med. 2018;15(11):e1002689. https://doi.org/10.1371/journal.pmed.1002689.CrossRef

84.

Cath C. Governing artificial intelligence: ethical, legal and technical opportunities and challenges. Philos Trans A Math Phys Eng Sci. 2018;376(2133):20180080. https://doi.org/10.1098/rsta.2018.0080.CrossRef

85.

Dawson D, Schlieger E, Horton J, et al. Artificial Intelligence: Australia’s Ethics Framework. Data 61 CSIRO, Australia. 2019. https://consult.industry.gov.au/strategic-policy/artificial-intelligence-ethics-framework/supporting_documents/ArtificialIntelligenceethicsframeworkdiscussionpaper.pdf.

86.

EU: General Data Protection Regulation. 2018. https://gdpr-info.eu. Accessed 28 Jul 2022.

87.

Manson NC, O’Neill O. Rethinking informed consent in bioethics. Cambridge: Cambridge University Press; 2007.CrossRef

88.

Virani AH, Longstaff H. Ethical considerations in biobanks: how a public health ethics perspective sheds new light on old controversies. J Genet Couns. 2015;24(3):428–32. https://doi.org/10.1007/s10897-014-9781-9.CrossRef

89.

Kaufman D, Bollinger J, Dvoskin R, et al. Preferences for opt-in and opt-out enrollment and consent models in biobank research: a national survey of Veterans Administration patients. Genet Med. 2012;14(9):787–94. https://doi.org/10.1038/gim.2012.45.CrossRef

90.

Faden RR, Kass NE, Goodman SN, et al. An ethics framework for a learning health care system: a departure from traditional research ethics and clinical ethics. Hastings Cent Rep. 2013;43:16–27. https://doi.org/10.1002/hast.134.CrossRef

91.

Larson DB, Magnus DC, Lungren MP, et al. Ethics of using and sharing clinical imaging data for artificial Intelligence: a proposed framework. Radiology. 2020;295(3):675–82. https://doi.org/10.1148/radiol.2020192536.CrossRef

92.

Papernot N, Abadi M, Erlingsson L, et al. Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data. 2016. https://doi.org/10.48550/arXiv.1610.05755

93.

London AJ. Artificial intelligence and black-box medical decisions: accuracy versus explainability. Hastings Cent Rep. 2019;49(1):15–21. https://doi.org/10.1002/hast.973.CrossRef

94.

Vane JR, Botting RM. The mechanism of action of aspirin. Thromb Res. 2003;110(5–6):255–8. https://doi.org/10.1016/S0049-3848(03)00379-7.CrossRef

95.

Babic B, Cohen IG, Evgeniou T, et al. Can AI fairly decide who gets an organ transplant? 2020–12–01. https://hbr.org/2020/12/can-ai-fairly-decide-who-gets-an-organ-transplant

96.

Price W, Nicholson II. Regulating black-box medicine. Mich Law Rev. 2017;116:421–74. https://doi.org/10.36644/mlr.116.3.regulating.CrossRef

97.

Ujm A, Pd B, Pknv B. Approval of artificial intelligence and machine learning-based medical devices in the USA and Europe (2015–20): a comparative analysis. Lancet Digits Health. 2021;3:195–203. https://doi.org/10.1016/S2589-7500(20)30292-2.CrossRef

98.

FDA. Artificial Intelligence and Machine Learning in Software as a Medical Device. 2021. https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device. Accessed 29 Jul 2022.

Title: Ethics and governance of trustworthy medical artificial intelligence
Authors: Jie Zhang
Zong-ming Zhang
Publication date: 01-12-2023
Publisher: BioMed Central
Keyword: Artificial Intelligence
Published in: BMC Medical Informatics and Decision Making / Issue 1/2023
Electronic ISSN: 1472-6947
DOI: https://doi.org/10.1186/s12911-023-02103-9

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Ethics and governance of trustworthy medical artificial intelligence

Abstract

Background

Methods

Results

Conclusions

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2023

Factors influencing the implementation of decision support systems for antibiotic prescription in hospitals: a systematic review

Prediction of inappropriate pre-hospital transfer of patients with suspected cardiovascular emergency diseases using machine learning: a retrospective observational study

Use of sentiment analysis for capturing hospitalized cancer patients' experience from free-text comments in the Persian language

Machine learning-driven clinical decision support system for concept-based searching: a field trial in a Norwegian hospital

Artificial intelligence of digital morphology analyzers improves the efficiency of manual leukocyte differentiation of peripheral blood

Comparison of causal forest and regression-based approaches to evaluate treatment effect heterogeneity: an application for type 2 diabetes precision medicine