Skip to main content
Top
Published in:

05-05-2024 | Artificial Intelligence | Short Communication

The utility of ChatGPT as a generative medical translator

Authors: David R. Grimm, Yu-Jin Lee, Katherine Hu, Longsha Liu, Omar Garcia, Karthik Balakrishnan, Noel F. Ayoub

Published in: European Archives of Oto-Rhino-Laryngology | Issue 11/2024

Login to get access

Abstract

Purpose

Large language models continue to dramatically change the medical landscape. We aimed to explore the utility of ChatGPT in providing accurate, actionable, and understandable generative medical translations in English, Spanish, and Mandarin pertaining to Otolaryngology.

Methods

Responses of GPT-4 to commonly asked patient questions listed on official otolaryngology clinical practice guidelines (CPG) were evaluated with the Patient Education materials Assessment Tool-printable (PEMAT-P.) Additional critical elements were identified a priori to evaluate ChatGPT’s accuracy and thoroughness in its responses. Multiple fluent speakers of English, Mandarin, and Spanish evaluated each response generated by ChatGPT.

Results

Total PEMAT-P scores differed between English, Mandarin, and Spanish GPT-4 generated responses depicting a moderate effect size of language, Eta-Square 0.07 with scores ranging from 73 to 77 (P-value = 0.03). Overall understandability scores did not differ between English, Mandarin, and Spanish depicting a small effect size of language, Eta-Square 0.02 scores ranging from 76 to 79 (P-value = 0.17), nor did overall actionability scores Eta-Square 0 score ranging 66–73 (P-value = 0.44). Overall a priori procedure-specific responses similarly did not differ between English, Spanish, and Mandarin Eta-Square 0.02 scores ranging 61–78 (P-value = 0.22).

Conclusion

GPT-4 produces accurate, understandable, and actionable outputs in English, Spanish, and Mandarin. Responses generated by GPT-4 in Spanish and Mandarin are comparable to English counterparts indicating a novel use for these models within Otolaryngology, and implications for bridging healthcare access and literacy gaps.

Level of evidence

IV.
Appendix
Available only for authorised users
Literature
Metadata
Title
The utility of ChatGPT as a generative medical translator
Authors
David R. Grimm
Yu-Jin Lee
Katherine Hu
Longsha Liu
Omar Garcia
Karthik Balakrishnan
Noel F. Ayoub
Publication date
05-05-2024
Publisher
Springer Berlin Heidelberg
Published in
European Archives of Oto-Rhino-Laryngology / Issue 11/2024
Print ISSN: 0937-4477
Electronic ISSN: 1434-4726
DOI
https://doi.org/10.1007/s00405-024-08708-8