Top

memo - Magazine of European Medical Oncology

26-01-2024 | Artificial Intelligence | original report

Can artificial intelligence provide accurate and reliable answers to cancer patients’ questions about cancer pain? Comparison of chatbots based on ESMO cancer pain guideline

Authors: Kadriye Bir Yücel, MD, Osman Sutcuoglu, MD, Ozan Yazıcı, MD, Ahmet Ozet, MD, Nuriye Ozdemir, MD

Published in: memo - Magazine of European Medical Oncology

Summary

Purpose

The purpose of this study was to assess the accuracy, readability, and stability of the answers given by chatbots to patients’ questions about cancer pain management.

Materials and methods

Twenty questions about cancer pain were constructed based on the European Society of Medical Oncology (ESMO) cancer pain patient guide. These questions were asked to five chatbots: Google Bard (Google AI, USA), ChatGPT‑4 (Chat Generative Pre-trained Transforme, OpenAI, USA) (premium version), ChatGPT‑3.5 (free version), Perplexity (Perplexity AI, USA), and Chatsonic (WriteSonic, USA). Three medical oncologists with at least 10 years of experience evaluated the chatbots’ responses for accuracy, readability, and stability.

Results

ChatGPT-4 had the highest accuracy rate of 96%. Perplexity had the highest readability but the lowest accuracy (86%). Google Bard and ChatGPT‑4 were the most stable (100%) chatbots. Both versions of ChatGPT appeared to provide extensive information, but the answers only included information before September 2021.

Conclusion

All chatbots are insufficient to obtain accurate information for cancer patients, and the resources are quite inadequate in acquiring accurate information for cancer patients and their families, and they need further development.

Available only for authorised users

Bahar-Ozdemir Y, Ozsoy-Unubol T, Akyuz G. Is YouTube a high-quality source of information on cancer rehabilitation? J Cancer Surviv. 2022;16(5):1016–22. https://doi.org/10.1007/s11764-021-01093-9.CrossRefPubMed

Sütcüoğlu O, Özay Zİ, Özet A, Yazıcı O, Özdemir N. Evaluation of scientific reliability and quality of YouTube videos on cancer and nutrition. Nutrition. 2023;108:111933. https://doi.org/10.1016/j.nut.2022.111933.CrossRefPubMed

Ali SR, Dobbs TD, Hutchings HA, Whitaker IS. Using ChatGPT to write patient clinic letters. Lancet Digit Health. 2023;5(4):e179–e81. https://doi.org/10.1016/S2589-7500(23)00048-1.CrossRefPubMed

Xue VW, Lei P, Cho WC. The potential impact of ChatGPT in clinical and translational medicine. Clin Transl Med. 2023;13(3):e1216. https://doi.org/10.1002/ctm2.1216.CrossRefPubMedPubMedCentral

Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, et al. How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023;9:e45312. https://doi.org/10.2196/45312.CrossRefPubMedPubMedCentral

Arif TB, Munaf U, Ul-Haque I. The future of medical education and research: is ChatGPT a blessing or blight in disguise? Med Educ Online. 2023;28(1):2181052. https://doi.org/10.1080/10872981.2023.2181052.CrossRefPubMedPubMedCentral

Homolak J. Opportunities and risks of ChatGPT in medicine, science, and academic publishing: a modern Promethean dilemma. Croat Med J. 2023;64(1):1–3. https://doi.org/10.3325/cmj.2023.64.1.CrossRefPubMedPubMedCentral

Baumgartner C. The potential impact of ChatGPT in clinical and translational medicine. Clin Transl Med. 2023;13(3):e1206. https://doi.org/10.1002/ctm2.1206.CrossRefPubMedPubMedCentral

Hopkins AM, Logan JM, Kichenadasse G, Sorich MJ. Artificial intelligence chatbots will revolutionize how cancer patients access information: ChatGPT represents a paradigm-shift. JNCI Cancer Spectr. 2023;7(2):pkad10. https://doi.org/10.1093/jncics/pkad010.CrossRefPubMedPubMedCentral

10.

Si L, Callan J. A statistical model for scientific readability. In proceedings of the tenth international conference on Information and knowledge management (CIKM ’01). New York, NY, USA: Association for Computing Machinery; 2001. pp. 574–6. https://doi.org/10.1145/502585.502695.CrossRef

11.

Haver HL, Ambinder EB, Bahl M, Oluyemi ET, Jeudy J, Yi PH. Appropriateness of breast cancer prevention and screening recommendations provided by ChatGPT. Radiology. 2023;307(4):e230424. https://doi.org/10.1148/radiol.230424.CrossRefPubMed

12.

Sarraju A, Bruemmer D, Van Iterson E, Cho L, Rodriguez F, Laffin L. Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chat-based artificial intelligence model. JAMA. 2023;329(10):842–4. https://doi.org/10.1001/jama.2023.1044.CrossRefPubMedPubMedCentral

13.

Di Bello F, Collà Ruvolo C, Cilio S, La Rocca R, Capece M, Creta M, et al. Testicular cancer and YouTube: what do you expect from a social media platform? Int J Urol. 2022;29(7):685–91. https://doi.org/10.1111/iju.14871.CrossRefPubMed

14.

Johnson SB, Parsons M, Dorff T, Moran MS, Ward JH, Cohen SA, et al. Cancer misinformation and harmful information on facebook and other social media: a brief report. J Natl Cancer Inst. 2022;114(7):1036–9. https://doi.org/10.1093/jnci/djab141.CrossRefPubMed

15.

Johnson SB, King AJ, Warner EL, Aneja S, Kann BH, Bylund CL. Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information. JNCI Cancer Spectr. 2023;7(2):pkad15. https://doi.org/10.1093/jncics/pkad015.CrossRefPubMedPubMedCentral

16.

Yang K, Ji S, Zhang T, Xie Q, Ananiadou S. On the evaluations of ChatGPT and emotion-enhanced prompting for mental health analysis. 2023. https://doi.org/10.48550/arXiv.2304.03347.CrossRef

17.

Singh OP. Artificial intelligence in the era of ChatGPT-Opportunities and challenges in mental health care. Indian J Psychiatry. 2023;65(3):297–8. https://doi.org/10.4103/indianjpsychiatry.indianjpsychiatry_112_23.CrossRefPubMedPubMedCentral

18.

Uludag K. Testing creativity of ChatGPT in psychology: Interview with ChatGPT. 2023. https://doi.org/10.2139/ssrn.4390872. SSRN 4390872.CrossRef

Title: Can artificial intelligence provide accurate and reliable answers to cancer patients’ questions about cancer pain? Comparison of chatbots based on ESMO cancer pain guideline
Authors: Kadriye Bir Yücel, MD
Osman Sutcuoglu, MD
Ozan Yazıcı, MD
Ahmet Ozet, MD
Nuriye Ozdemir, MD
Publication date: 26-01-2024
Publisher: Springer Vienna
Keyword: Artificial Intelligence
Published in: memo - Magazine of European Medical Oncology
Print ISSN: 1865-5041
Electronic ISSN: 1865-5076
DOI: https://doi.org/10.1007/s12254-023-00954-6

Webinar | 19-02-2024 | 17:30 (CET)

Keynote webinar | Spotlight on antibody–drug conjugates in cancer

Watch now

Antibody–drug conjugates (ADCs) are novel agents that have shown promise across multiple tumor types. Explore the current landscape of ADCs in breast and lung cancer with our experts, and gain insights into the mechanism of action, key clinical trials data, existing challenges, and future directions.

Dr. Véronique Diéras

Prof. Fabrice Barlesi

Developed by: Springer Medicine