Top

Published in:

Open Access 05-03-2024 | Artificial Intelligence | 2024 SAGES Poster

Dr. Google to Dr. ChatGPT: assessing the content and quality of artificial intelligence-generated medical information on appendicitis

Authors: Yazid K. Ghanem, Armaun D. Rouhi, Ammr Al-Houssan, Zena Saleh, Matthew C. Moccia, Hansa Joshi, Kristoffel R. Dumon, Young Hong, Francis Spitz, Amit R. Joshi, Michael Kwiatt

Published in: Surgical Endoscopy | Issue 5/2024

Abstract

Introduction

Generative artificial intelligence (AI) chatbots have recently been posited as potential sources of online medical information for patients making medical decisions. Existing online patient-oriented medical information has repeatedly been shown to be of variable quality and difficult readability. Therefore, we sought to evaluate the content and quality of AI-generated medical information on acute appendicitis.

Methods

A modified DISCERN assessment tool, comprising 16 distinct criteria each scored on a 5-point Likert scale (score range 16–80), was used to assess AI-generated content. Readability was determined using the Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL) scores. Four popular chatbots, ChatGPT-3.5 and ChatGPT-4, Bard, and Claude-2, were prompted to generate medical information about appendicitis. Three investigators independently scored the generated texts blinded to the identity of the AI platforms.

Results

ChatGPT-3.5, ChatGPT-4, Bard, and Claude-2 had overall mean (SD) quality scores of 60.7 (1.2), 62.0 (1.0), 62.3 (1.2), and 51.3 (2.3), respectively, on a scale of 16–80. Inter-rater reliability was 0.81, 0.75, 0.81, and 0.72, respectively, indicating substantial agreement. Claude-2 demonstrated a significantly lower mean quality score compared to ChatGPT-4 (p = 0.001), ChatGPT-3.5 (p = 0.005), and Bard (p = 0.001). Bard was the only AI platform that listed verifiable sources, while Claude-2 provided fabricated sources. All chatbots except for Claude-2 advised readers to consult a physician if experiencing symptoms. Regarding readability, FKGL and FRE scores of ChatGPT-3.5, ChatGPT-4, Bard, and Claude-2 were 14.6 and 23.8, 11.9 and 33.9, 8.6 and 52.8, 11.0 and 36.6, respectively, indicating difficulty readability at a college reading skill level.

Conclusion

AI-generated medical information on appendicitis scored favorably upon quality assessment, but most either fabricated sources or did not provide any altogether. Additionally, overall readability far exceeded recommended levels for the public. Generative AI platforms demonstrate measured potential for patient education and engagement about appendicitis.

Duarte F (2024) Number of ChatGPT users. Exploding Topics. https://explodingtopics.com/blog/chatgpt-users

Shah NH, Entwistle DA, Pfeffer M (2023) Creation and adoption of large language models in medicine. JAMA 330(9):866. https://doi.org/10.1001/jama.2023.14217CrossRefPubMed

Ron L, Kumar A, Chen J (2023) How chatbots and large language model artificial intelligence systems will reshape modern medicine. JAMA Intern Med 183(6):596. https://doi.org/10.1001/jamainternmed.2023.1835CrossRef

Kirchner GJ, Kim RY, Weddle J, Bible JE (2023) Can artificial intelligence improve the readability of patient education materials? Clin Orthop Relat Res 481(11):2260–2267. https://doi.org/10.1097/corr.0000000000002668CrossRefPubMed

Rouhi AD, Ghanem YK, Hoeltzel GD et al (2022) Online resources for patients considering hiatal hernia repair: a quality and readability analysis. J Gastrointest Surg 27(3):598–600. https://doi.org/10.1007/s11605-022-05460-4CrossRefPubMed

Rouhi AD, Ghanem YK, Hoeltzel GD et al (2022) Quality and readability of online patient information on adolescent bariatric surgery. Obes Surg 33(1):397–399. https://doi.org/10.1007/s11695-022-06385-2CrossRefPubMed

Rouhi AD, Ghanem YK, Bader E et al (2023) Online information for incisional hernia repair: what are patients reading? Surgeon 21(4):e195–e200. https://doi.org/10.1016/j.surge.2022.12.002CrossRefPubMed

Rouhi AD, Han JJ, Ghanem YK et al (2022) Quality and readability of online patient information on the left ventricular assist device. Artif Organs 47(6):1029–1037. https://doi.org/10.1111/aor.14479CrossRefPubMed

Rouhi AD, Ghanem YK, Hoeltzel GD et al (2022) Quality and readability assessment of online patient information on cytoreductive surgery and hyperthermic intraperitoneal chemotherapy. J Surg Oncol 127(4):699–705. https://doi.org/10.1002/jso.27143CrossRefPubMed

10.

Weiss BD (2003) Health literacy: a manual for clinicians. American Medical Association Foundation and American Medical Association, Chicago

11.

National Cancer Institute (1994) Clear and simple: developing effective print materials for low literate readers. National Institutes of Health, National Cancer Institute

12.

Trutner Z, Furlough K, Martinez AB et al (2023) Is health literacy associated with surgical outcomes? A systematic review. J Surg Res 291:720–733. https://doi.org/10.1016/j.jss.2023.06.044CrossRefPubMed

13.

Moris D, Paulson EK, Pappas TN (2021) Diagnosis and management of acute appendicitis in adults. JAMA 326(22):2299. https://doi.org/10.1001/jama.2021.20502CrossRefPubMed

14.

Charnock D, Shepperd S, Needham G, Gann R (1999) DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 53(2):105–111. https://doi.org/10.1136/jech.53.2.105CrossRefPubMedPubMedCentral

15.

American College of Surgeons Division of Education (2022) Appendectomy. https://www.facs.org/media/4molizpf/app.pdf. Accessed 10 Sep 2023.

16.

Smink D et al (2023) Management of acute appendicitis in adults. UpToDate. https://www.uptodate.com/contents/management-of-acute-appendicitis-in-adults. Accessed 10 Sep 2023.

17.

Flesch R (1948) A new readability yardstick. J Appl Psychol 32(3):221–233. https://doi.org/10.1037/h0057532CrossRefPubMed

18.

Kincaid JP, Fishburne Jr. RP, Rogers RL, Chissom BS (1975) Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel. Institute for Simulation and Training 56. https://stars.library.ucf.edu/istlibrary/56.

19.

Massie P, Arshad SA, Auyang ED (2024) Readability of American Society of Metabolic Surgery’s patient information publications. J Surg Res 293:727–732. https://doi.org/10.1016/j.jss.2023.09.018CrossRefPubMed

20.

Daraz L, Morrow AS, Ponce OJ et al (2018) Readability of online health information: a meta-narrative systematic review. Am J Med Qual 33(5):487–492. https://doi.org/10.1177/1062860617751639CrossRefPubMed

21.

Link E, Baumann E (2020) Nutzung von Gesundheitsinformationen im Internet: personenbezogene und motivationale Einflussfaktoren. Bundesgesundheitsblatt—Gesundheitsforschung—Gesundheitsschutz. 63(6):681–689. https://doi.org/10.1007/s00103-020-03144-5CrossRefPubMedPubMedCentral

22.

Baumann E, Czerwinski F, Rosset M, Seelig M, Suhr R (2020) Wie informieren sich die Menschen in Deutschland zum Thema Gesundheit? Erkenntnisse aus der ersten Welle von HINTS Germany. Bundesgesundheitsblatt—Gesundheitsforschung—Gesundheitsschutz. 63(9):1151–1160. https://doi.org/10.1007/s00103-020-03192-xCrossRefPubMed

23.

Yeo YH, Samaan JS, Ng WH et al (2023) Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clin Mol Hepatol 29(3):721–732. https://doi.org/10.3350/cmh.2023.0089CrossRefPubMedPubMedCentral

24.

Momenaei B, Wakabayashi T, Shahlaee A et al (2023) Appropriateness and readability of CHATGPT-4-generated responses for surgical treatment of retinal diseases. Ophthalmol Retina 7(10):862–868. https://doi.org/10.1016/j.oret.2023.05.022CrossRefPubMed

25.

Samaan JS, Yeo YH, Rajeev N et al (2023) Assessing the accuracy of responses by the language model ChatGPT to questions regarding bariatric surgery. Obes Surg 33(6):1790–1796. https://doi.org/10.1007/s11695-023-06603-5CrossRefPubMedPubMedCentral

26.

Rao A, Pang M, Kim J et al (2023) Assessing the utility of ChatGPT throughout the entire clinical workflow: development and usability study. J Med Internet Res 25:e48659. https://doi.org/10.2196/48659CrossRefPubMedPubMedCentral

27.

Agarwal M, Sharma P, Goswami A (2023) Analysing the applicability of ChatGPT, bard, and bing to generate reasoning-based multiple-choice questions in medical physiology. Cureus. https://doi.org/10.7759/cureus.40977CrossRefPubMedPubMedCentral

28.

Giray L (2023) Prompt engineering with ChatGPT: a guide for academic writers. Ann Biomed Eng 51(12):2629–2633. https://doi.org/10.1007/s10439-023-03272-4CrossRefPubMed

29.

Moons P, Van Bulck L (2023) Using ChatGPT and Google Bard to improve the readability of written patient information: a proof of concept. Eur J Cardiovasc Nurs. https://doi.org/10.1093/eurjcn/zvad087CrossRefPubMed

30.

Meskó B (2023) Prompt engineering as an important emerging skill for medical professionals: tutorial. J Med Internet Res 25:e50638. https://doi.org/10.2196/50638CrossRefPubMedPubMedCentral

31.

Emsley R (2023) ChatGPT: these are not hallucinations—they’re fabrications and falsifications. Schizophrenia. https://doi.org/10.1038/s41537-023-00379-4CrossRefPubMedPubMedCentral

32.

Gao C, Howard FM, Markov NS et al (2022) Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. BioRxiv. https://doi.org/10.1101/2022.12.23.521610CrossRefPubMedPubMedCentral

Title: Dr. Google to Dr. ChatGPT: assessing the content and quality of artificial intelligence-generated medical information on appendicitis
Authors: Yazid K. Ghanem
Armaun D. Rouhi
Ammr Al-Houssan
Zena Saleh
Matthew C. Moccia
Hansa Joshi
Kristoffel R. Dumon
Young Hong
Francis Spitz
Amit R. Joshi
Michael Kwiatt
Publication date: 05-03-2024
Publisher: Springer US
Keywords: Artificial Intelligence
Appendicitis
Appendicitis
Published in: Surgical Endoscopy / Issue 5/2024
Print ISSN: 0930-2794
Electronic ISSN: 1432-2218
DOI: https://doi.org/10.1007/s00464-024-10739-5

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Dr. Google to Dr. ChatGPT: assessing the content and quality of artificial intelligence-generated medical information on appendicitis

Abstract

Introduction

Methods

Results

Conclusion

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Introduction

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 5/2024

Large language models and bariatric surgery patient education: a comparative readability analysis of GPT-3.5, GPT-4, Bard, and online institutional resources

Society of American Gastrointestinal and Endoscopic Surgeons guidelines development: health equity update to standard operating procedure

A decade of experience with minimally invasive anti-reflux operations: robot vs. LESS

Evaluation of 30-day outcomes for open ventral hernia repair using self-gripping versus nonself-gripping mesh

The death of laparoscopy

The performance of artificial intelligence large language model-linked chatbots in surgical decision-making for gastroesophageal reflux disease