Top

25-09-2024 | Artificial Intelligence

Artificial intelligence large language model scores highly on focused practice designation in metabolic and bariatric surgery board practice questions

Authors: A. Sanders, R. Lim, D. Jones, R. W. Vosburg

Abstract

Background

Artificial intelligence models such as ChatGPT (Open AI) have performed well on the exams of various medical and surgical fields. It is not yet known how ChatGPT performs on similar metabolic and bariatric surgery (MBS) questions.

Objective

Assess the performance of ChatGPT on Focused Practice Designation in Metabolic and Bariatric Surgery board-style questions.

Setting

United States.

Methods

Questions obtained from the largest commercially available bank of FPD-MBS practice questions were entered into ChatGPT-4, as is, without prior training. We assessed the overall percentage correct as well as the percentage correct within each of the five American Board of Surgery (ABS) question categories. One-way ANOVA was used to determine if the frequency of correct answers differed between categories.

Results

Out of 255 questions, ChatGPT-4 correctly answered 189 (74.1%). Between the five question categories there was no difference between the frequency of correct answers (p = 0.22). It did not matter if questions were entered individually or in groups of up to 10.

Conclusion

Without prior training, ChatGPT-4 scored highly when evaluated on the largest practice question bank for the FPD-MBS exam.

Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW (2023) Large language models in medicine. Nat Med 29:1930–1940. https://doi.org/10.1038/s41591-023-02448-8CrossRefPubMed

Hofmann HL, Guerra GA, Le JL, Wong AM, Hofmann GH, Mayfield CK et al (2024) The rapid development of artificial intelligence: GPT-4’s Performance on Orthopedic Surgery Board Questions. Orthopedics 47:e85–e89. https://doi.org/10.3928/01477447-20230922-05CrossRefPubMed

OpenAI, Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I et al (2024) GPT-4 technical report. arXiv. https://arxiv.org/abs/2303.08774

Beaulieu-Jones BR, Berrigan MT, Shah S, Marwaha JS, Lai S-L, Brat GA (2024) Evaluating capabilities of large language models: performance of GPT-4 on surgical knowledge assessments. Surgery 175:936–942. https://doi.org/10.1016/j.surg.2023.12.014CrossRefPubMed

Taloni A, Borselli M, Scarsi V, Rossi C, Coco G, Scorcia V et al (2023) Comparative performance of humans versus GPT-4.0 and GPT-3.5 in the self-assessment program of American Academy of Ophthalmology. Sci Rep 13:18562. https://doi.org/10.1038/s41598-023-45837-2CrossRefPubMedPubMedCentral

Hopkins BS, Nguyen VN, Dallas J, Texakalidis P, Yang M, Renn A et al (2023) ChatGPT versus the neurosurgical written boards: a comparative analysis of artificial intelligence/machine learning performance on neurosurgical board-style questions. J Neurosurg 139:904–911. https://doi.org/10.3171/2023.2.JNS23419CrossRefPubMed

Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C et al (2023) Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digit Health 2:e0000198. https://doi.org/10.1371/journal.pdig.0000198CrossRefPubMedPubMedCentral

Elias ML, Burshtein J, Sharon VR (2024) OpenAI’s GPT-4 performs to a high degree on board-style dermatology questions. Int J Dermatol 63:73–78. https://doi.org/10.1111/ijd.16913CrossRefPubMed

Long C, Lowe K, Zhang J, Santos AD, Alanazi A, O’Brien D et al (2024) A novel evaluation model for assessing ChatGPT on Otolaryngology-Head and Neck Surgery Certification examinations: performance study. JMIR Med Educ 10:e49970. https://doi.org/10.2196/49970CrossRefPubMedPubMedCentral

10.

Takagi S, Watari T, Erabi A, Sakaguchi K (2023) Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: comparison study. JMIR Med Educ 9:e48002. https://doi.org/10.2196/48002CrossRefPubMedPubMedCentral

11.

Toyama Y, Harigai A, Abe M, Nagano M, Kawabata M, Seki Y et al (2024) Performance evaluation of ChatGPT, GPT-4, and Bard on the official board examination of the Japan Radiology Society. Jpn J Radiol 42:201–207. https://doi.org/10.1007/s11604-023-01491-2CrossRefPubMed

12.

Oh N, Choi G-S, Lee WY (2023) ChatGPT goes to the operating room: evaluating GPT-4 performance and its potential in surgical education and training in the era of large language models. Ann Surg Treat Res 104:269–273. https://doi.org/10.4174/astr.2023.104.5.269CrossRefPubMedPubMedCentral

13.

The American Board of Surgery (n.d.) Focused practice designations. https://www.absurgery.org/get-certified/focused-practice-designations/. Accessed 15 Apr 2024

14.

NMBE (n.d.) NBME item-writing guide. https://www.nbme.org/item-writing-guide. Accessed 15 Apr 2024

15.

The American Board of Surgery (n.d.) Metabolic and bariatric surgery focused practice designation. https://program.absurgery.org/default.jsp?certfpd_mbs. Accessed 15 Apr 2024

Title: Artificial intelligence large language model scores highly on focused practice designation in metabolic and bariatric surgery board practice questions
Authors: A. Sanders
R. Lim
D. Jones
R. W. Vosburg
Publication date: 25-09-2024
Publisher: Springer US
Keywords: Artificial Intelligence
Bariatric Surgery
Bariatric Surgery
Published in: Surgical Endoscopy
Print ISSN: 0930-2794
Electronic ISSN: 1432-2218
DOI: https://doi.org/10.1007/s00464-024-11267-y

Mastering chronic pancreatitis pain: A multidisciplinary approach and practical solutions

Webinar | 06-02-2024 | 20:00 (CET)

Severe pain is the most common symptom of chronic pancreatitis. In this webinar, experts share the latest insights in pain management for chronic pancreatitis patients. Experts from a range of disciplines discuss pertinent cases and provide practical suggestions for use within clinical practice.

Keynote webinar | Spotlight on modern management of frailty

Springer Medicine

Artificial intelligence large language model scores highly on focused practice designation in metabolic and bariatric surgery board practice questions

Abstract

Background

Objective

Setting

Methods

Results

Conclusion

Mastering chronic pancreatitis pain: A multidisciplinary approach and practical solutions

Keynote webinar | Spotlight on modern management of frailty

Springer Medicine

Abstract

Background

Objective

Setting

Methods

Results

Conclusion

Please log in to get access to this content

Mastering chronic pancreatitis pain: A multidisciplinary approach and practical solutions