Top

European Archives of Oto-Rhino-Laryngology

Published in:

08-09-2023 | Artificial Intelligence | Short Communication

Assessing the accuracy of ChatGPT references in head and neck and ENT disciplines

Authors: Andrea Frosolini, Leonardo Franz, Simone Benedetti, Luigi Angelo Vaira, Cosimo de Filippis, Paolo Gennaro, Gino Marioni, Guido Gabriele

Published in: European Archives of Oto-Rhino-Laryngology | Issue 11/2023

Abstract

Purpose

ChatGPT has gained popularity as a web application since its release in 2022. While artificial intelligence (AI) systems’ potential in scientific writing is widely discussed, their reliability in reviewing literature and providing accurate references remains unexplored. This study examines the reliability of references generated by ChatGPT language models in the Head and Neck field.

Methods

Twenty clinical questions were generated across different Head and Neck disciplines, to prompt ChatGPT versions 3.5 and 4.0 to produce texts on the assigned topics. The generated references were categorized as “true,” “erroneous,” or “inexistent” based on congruence with existing records in scientific databases.

Results

ChatGPT 4.0 outperformed version 3.5 in terms of reference reliability. However, both versions displayed a tendency to provide erroneous/non-existent references.

Conclusions

It is crucial to address this challenge to maintain the reliability of scientific literature. Journals and institutions should establish strategies and good-practice principles in the evolving landscape of AI-assisted scientific writing.

Available only for authorised users

Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, Chartash D (2023) How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ 9:e45312. https://doi.org/10.2196/45312CrossRefPubMedPubMedCentral

Number of ChatGPT Users (2023) Available online: https://explodingtopics.com/blog/chatgpt-users. Accessed on 9 Jun 2023

Dave T, Athaluri SA, Singh S (2023) ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell 6:1169595. https://doi.org/10.3389/frai.2023.1169595CrossRefPubMedPubMedCentral

Kim JK, Chua M, Rickard M, Lorenzo A (2023) ChatGPT and large language model (LLM) chatbots: the current state of acceptability and a proposal for guidelines on utilization in academic medicine. J Pediatr Urol S1477–5131(23):00224–00233. https://doi.org/10.1016/j.jpurol.2023.05.018CrossRef

Sallam M (2023) ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare (Basel) 19(11):887CrossRef

Bahadoran Z, Mirmiran P, Kashfi K, Ghasemi A (2020) The principles of biomedical scientific writing: citation. Int J Endocrinol Metab 18:e102622PubMedPubMedCentral

ChatGPT (2023) Available online: https://openai.com/blog/chatgpt. Accessed on 10 Jun 2023

Hill-Yardin EL, Hutchinson MR, Laycock R, Spencer SJ (2023) A Chat(GPT) about the future of scientific publishing. Brain Behav Immun 110:152–154. https://doi.org/10.1016/j.bbi.2023.02.022CrossRefPubMed

Frosolini A, Gennaro P, Cascino F, Gabriele G (2023) In reference to “role of Chat GPT in public health”, to highlight the AI’s incorrect reference generation. Ann Biomed Eng. https://doi.org/10.1007/s10439-023-03248-4CrossRefPubMed

10.

Balel Y (2023) Can ChatGPT be used in oral and maxillofacial surgery? J Stomatol Oral Maxillofac Surg. https://doi.org/10.1016/j.jormas.2023.101471CrossRefPubMed

11.

Park I, Joshi AS, Javan R (2023) Potential role of ChatGPT in clinical otolaryngology explained by ChatGPT. Am J Otolaryngol 29(44):103873CrossRef

12.

Hoch CC, Wollenberg B, Lüers JC, Knoedler S, Knoedler L, Frank K, Cotofana S, Alfertshofer M (2023) ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions. Eur Arch Otorhinolaryngol. https://doi.org/10.1007/s00405-023-08051-4CrossRefPubMedPubMedCentral

13.

Chiesa-Estomba CM, Lechien JR, Vaira LA, Brunet A, Cammaroto G, Mayo-Yanez M, Sanchez-Barrueco A, Saga-Gutierrez C (2023) Exploring the potential of Chat-GPT as a supportive tool for sialendoscopy clinical decision making and patient information support. Eur Arch Otorhinolaryngol. https://doi.org/10.1007/s00405-023-08104-8CrossRefPubMed

14.

Wu RT, Dang RR (2023) ChatGPT in head and neck scientific writing: a precautionary anecdote. Am J Otolaryngol 44(6):103980. https://doi.org/10.1016/j.amjoto.2023.103980CrossRefPubMed

Title: Assessing the accuracy of ChatGPT references in head and neck and ENT disciplines
Authors: Andrea Frosolini
Leonardo Franz
Simone Benedetti
Luigi Angelo Vaira
Cosimo de Filippis
Paolo Gennaro
Gino Marioni
Guido Gabriele
Publication date: 08-09-2023
Publisher: Springer Berlin Heidelberg
Keyword: Artificial Intelligence
Published in: European Archives of Oto-Rhino-Laryngology / Issue 11/2023
Print ISSN: 0937-4477
Electronic ISSN: 1434-4726
DOI: https://doi.org/10.1007/s00405-023-08205-4

2024 ASCO Annual Meeting

Springer Medicine

Assessing the accuracy of ChatGPT references in head and neck and ENT disciplines

Abstract

Purpose

Methods

Results

Conclusions

2024 ASCO Annual Meeting

Springer Medicine

Abstract

Purpose

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 11/2023

Prevalence and risk factors of hearing loss among the middle-aged and older population in China: a systematic review and meta-analysis

The effects of endoscope placement in the hypopharynx on swallowing-related measures in healthy adults

Letter to the Editor regarding “A comparative study of audiological outcomes and compliance between the Osia system and other bone conduction hearing implants” by Kim et al. European Archives of Otorhinolaryngology 2023 May; 280(5):2217–2224

Radiopathologic predictors of 1- and 2-year frontal sinusotomy outcomes in a southeast Asian chronic rhinosinusitis population

Local anesthesia with blue-dyed lidocaine for a better patient’s tolerance during office-based laryngology procedures: how I do it

General classification of rhinopaties: the need for standardization according to etiology and nasal cytology