Top

European Archives of Oto-Rhino-Laryngology

Published in:

02-11-2023 | Artificial Intelligence | Miscellaneous

Artificial intelligence chatbots as sources of patient education material for obstructive sleep apnoea: ChatGPT versus Google Bard

Authors: Ryan Chin Taw Cheong, Samit Unadkat, Venkata Mcneillis, Andrew Williamson, Jonathan Joseph, Premjit Randhawa, Peter Andrews, Vinidh Paleri

Published in: European Archives of Oto-Rhino-Laryngology | Issue 2/2024

Abstract

Purpose

To perform the first head-to-head comparative evaluation of patient education material for obstructive sleep apnoea generated by two artificial intelligence chatbots, ChatGPT and its primary rival Google Bard.

Methods

Fifty frequently asked questions on obstructive sleep apnoea in English were extracted from the patient information webpages of four major sleep organizations and categorized as input prompts. ChatGPT and Google Bard responses were selected and independently rated using the Patient Education Materials Assessment Tool–Printable (PEMAT-P) Auto-Scoring Form by two otolaryngologists, with a Fellowship of the Royal College of Surgeons (FRCS) and a special interest in sleep medicine and surgery. Responses were subjectively screened for any incorrect or dangerous information as a secondary outcome. The Flesch-Kincaid Calculator was used to evaluate the readability of responses for both ChatGPT and Google Bard.

Results

A total of 46 questions were curated and categorized into three domains: condition (n = 14), investigation (n = 9) and treatment (n = 23). Understandability scores for ChatGPT versus Google Bard on the various domains were as follows: condition 90.86% vs.76.32% (p < 0.001); investigation 89.94% vs. 71.67% (p < 0.001); treatment 90.78% vs.73.74% (p < 0.001). Actionability scores for ChatGPT versus Google Bard on the various domains were as follows: condition 77.14% vs. 51.43% (p < 0.001); investigation 72.22% vs. 54.44% (p = 0.05); treatment 73.04% vs. 54.78% (p = 0.002). The mean Flesch–Kincaid Grade Level for ChatGPT was 9.0 and Google Bard was 5.9. No incorrect or dangerous information was identified in any of the generated responses from both ChatGPT and Google Bard.

Conclusion

Evaluation of ChatGPT and Google Bard patient education material for OSA indicates the former to offer superior information across several domains.

OpenAI (2022) Introducing ChatGPT [Internet]. OpenAI.com. pp 1–11. https://openai.com/blog/chatgpt. Accessed 6 July 2023

Google AI updates: Bard and new AI features in Search [Internet]. Google. (2023). https://blog.google/technology/ai/bard-google-ai-search-updates/. Accessed 6 July 2023

Meskó B, Topol EJ (2023) The imperative for regulatory oversight of large language models (or generative AI) in healthcare. NPJ Digit Med. https://doi.org/10.1038/s41746-023-00873-0CrossRefPubMedPubMedCentral

Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C et al (2023) Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health 2(2):e0000198CrossRefPubMedPubMedCentral

Patel SB, Lam K (2023) ChatGPT: the future of discharge summaries? Lancet Digit Heal [Internet] 5(3):e107–e108. https://doi.org/10.1016/S2589-7500(23)00021-3CrossRef

Ali SR, Dobbs TD, Hutchings HA, Whitaker IS (2023) Using ChatGPT to write patient clinic letters. Lancet Digit Heal [Internet]. 5(4):e179–e181. https://doi.org/10.1016/S2589-7500(23)00048-1CrossRef

Ayoub NF, Lee YJ, Grimm D, Balakrishnan K (2023) Comparison between ChatGPT and google search as sources of postoperative patient instructions. JAMA Otolaryngol Head Neck Surg [Internet] 149(6):556–558CrossRefPubMed

Benjafield AV, Ayas NT, Eastwood PR, Heinzer R, Ip MSM, Morrell MJ et al (2019) Estimation of the global prevalence and burden of obstructive sleep apnoea: a literature-based analysis. Lancet Respir Med 7(8):687–698CrossRefPubMedPubMedCentral

Marin JM, Carrizo SJ, Vicente E, Agusti AGN (2005) Long-term cardiovascular outcomes in men with obstructive sleep apnoea-hypopnoea with or without treatment with continuous positive airway pressure: an observational study. Lancet 365(9464):1046–1053CrossRefPubMed

10.

Watson NF (2016) Health care savings: the economic value of diagnostic and therapeutic care for obstructive sleep apnea. J Clin Sleep Med 12(8):1075–1077CrossRefPubMedPubMedCentral

11.

Reinventing search with a new AI-powered Microsoft Bing and Edge, your copilot for the web—the Official Microsoft Blog [Internet]. https://blogs.microsoft.com/blog/2023/02/07/reinventing-search-with-a-new-ai-powered-microsoft-bing-and-edge-your-copilot-for-the-web/. Accessed 8 July 2023

12.

Patient-Friendly Guides—American Academy of Sleep Medicine—Association for Sleep Clinicians and Researchers [Internet]. https://aasm.org/clinical-resources/patient-info/patient-friendly-guidelines/. Accessed 12 July 2023

13.

Obstructive Sleep Apnea (OSA) [Internet]. https://www.sleephealthfoundation.org.au/obstructive-sleep-apnea.html. Accessed 12 July 2023

14.

Snoring and Sleep Apnoea | ENT UK [Internet]. https://www.entuk.org/patients/conditions/34/snoring_and_sleep_apnoea. Accessed 12 July 2023

15.

For Patients | International Surgical Sleep Society [Internet]. https://www.surgicalsleep.org/for-patients. Accessed 12 July 2023.

16.

Flesch Kincaid Calculator | Good Calculators [Internet]. https://goodcalculators.com/flesch-kincaid-calculator/. Accessed 20 July 2023

17.

Shoemaker SJ, Wolf MS, Brach C. The patient education materials assessment tool (PEMAT) and user’s guide. Rockville, MD: Agency for Healthcare Research and Quality. 2013 Nov;11.

18.

Introduction | Agency for Healthcare Research and Quality [Internet]. https://www.ahrq.gov/health-literacy/patient-education/pemat1.html. Accessed 15 July 2023

19.

Lyons MM, Bhatt NY, Pack AI, Magalang UJ (2020) Global burden of sleep-disordered breathing and its implications. Respirology 25(7):690–702CrossRefPubMed

20.

Bing vs Google: Search Engine Comparison 2023 | Impression [Internet]. https://www.impressiondigital.com/blog/bing-differ-google/. Accessed 15 July 2023

21.

Internet and social media users in the world 2023 | Statista [Internet]. https://www.statista.com/statistics/617136/digital-population-worldwide/. Accessed 15 July 2023

22.

Alkaissi H, McFarlane SI (2023) Artificial hallucinations in ChatGPT: implications in scientific writing. Cureus 15(2):2–5

23.

ChatGPT and LLMs: what’s the risk—NCSC.GOV.UK [Internet]. https://www.ncsc.gov.uk/blog-post/chatgpt-and-large-language-models-whats-the-risk. Accessed 16 July 2023

24.

Oliffe M, Thompson E, Johnston J, Freeman D, Bagga H, Wong PKK (2019) Assessing the readability and patient comprehension of rheumatology medicine information sheets: a cross-sectional Health Literacy Study. BMJ Open 9(2):1–10CrossRef

25.

Epic and Microsoft Bring GPT-4 to EHRs [Internet]. https://www.epic.com/epic/post/epic-and-microsoft-bring-gpt-4-to-ehrs. Accessed 6 July 2023

26.

NHS Long Term Plan » The NHS Long Term Plan [Internet]. https://www.longtermplan.nhs.uk/publication/nhs-long-term-plan/. Accessed 7 July 2023

27.

Funding boost for artificial intelligence in NHS to speed up diagnosis of deadly diseases—GOV.UK [Internet]. https://www.gov.uk/government/news/funding-boost-for-artificial-intelligence-in-nhs-to-speed-up-diagnosis-of-deadly-diseases. Accessed 17 July 2023

28.

£21 million to roll out artificial intelligence across the NHS—GOV.UK [Internet]. https://www.gov.uk/government/news/21-million-to-roll-out-artificial-intelligence-across-the-nhs. Accessed 17 July 2023

Title: Artificial intelligence chatbots as sources of patient education material for obstructive sleep apnoea: ChatGPT versus Google Bard
Authors: Ryan Chin Taw Cheong
Samit Unadkat
Venkata Mcneillis
Andrew Williamson
Jonathan Joseph
Premjit Randhawa
Peter Andrews
Vinidh Paleri
Publication date: 02-11-2023
Publisher: Springer Berlin Heidelberg
Keyword: Artificial Intelligence
Published in: European Archives of Oto-Rhino-Laryngology / Issue 2/2024
Print ISSN: 0937-4477
Electronic ISSN: 1434-4726
DOI: https://doi.org/10.1007/s00405-023-08319-9

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Artificial intelligence chatbots as sources of patient education material for obstructive sleep apnoea: ChatGPT versus Google Bard

Abstract

Purpose

Methods

Results

Conclusion

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Purpose

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 2/2024

Endoscopic assisted microscopic posterior cordotomy for bilateral abductor vocal fold paralysis using radiofrequency versus coblation

The development of paranasal sinuses in patients with cystic fibrosis: sinuses volume analysis

Intralesional cidofovir vs. bevacizumab for recurrent respiratory papillomatosis: a systematic review and indirect meta-analysis

Preoperative tracheostomy is associated with thyroid gland invasion and poorer prognosis in laryngectomized patients

The transcervical–transparotid corridor for management of parapharyngeal space neoplasms: strengths and limits in a bi-institutional retrospective series

Neoadjuvant chemotherapy for organ preservation in sinonasal squamous cell carcinoma