Top

Published in:

28-09-2024 | Artificial Intelligence | ORIGINAL ARTICLE

Comparative performance of artificial intelligence models in rheumatology board-level questions: evaluating Google Gemini and ChatGPT-4o

Authors: Enes Efe Is, Ahmet Kivanc Menekseoglu

Published in: Clinical Rheumatology | Issue 11/2024

Abstract

Objectives

This study evaluates the performance of AI models, ChatGPT-4o and Google Gemini, in answering rheumatology board-level questions, comparing their effectiveness, reliability, and applicability in clinical practice.

Method

A cross-sectional study was conducted using 420 rheumatology questions from the BoardVitals question bank, excluding 27 visual data questions. Both artificial intelligence models categorized the questions according to difficulty (easy, medium, hard) and answered them. In addition, the reliability of the answers was assessed by asking the questions a second time. The accuracy, reliability, and difficulty categorization of the AI models’ response to the questions were analyzed.

Results

ChatGPT-4o answered 86.9% of the questions correctly, significantly outperforming Google Gemini’s 60.2% accuracy (p < 0.001). When the questions were asked a second time, the success rate was 86.7% for ChatGPT-4o and 60.5% for Google Gemini. Both models mainly categorized questions as medium difficulty. ChatGPT-4o showed higher accuracy in various rheumatology subfields, notably in Basic and Clinical Science (p = 0.028), Osteoarthritis (p = 0.023), and Rheumatoid Arthritis (p < 0.001).

Conclusions

ChatGPT-4o significantly outperformed Google Gemini in rheumatology board-level questions. This demonstrates the success of ChatGPT-4o in situations requiring complex and specialized knowledge related to rheumatological diseases. The performance of both AI models decreased as the question difficulty increased. This study demonstrates the potential of AI in clinical applications and suggests that its use as a tool to assist clinicians may improve healthcare efficiency in the future. Future studies using real clinical scenarios and real board questions are recommended.

Key Points

•ChatGPT-4o significantly outperformed Google Gemini in answering rheumatology board-level questions, achieving 86.9% accuracy compared to Google Gemini’s 60.2%.

•For both AI models, the correct answer rate decreased as the question difficulty increased.

•The study demonstrates the potential for AI models to be used in clinical practice as a tool to assist clinicians and improve healthcare efficiency.

Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA et al (2023) How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ 9:e45312. https://doi.org/10.2196/45312CrossRefPubMedPubMedCentral

Hopkins BS, Nguyen VN, Dallas J, Texakalidis P, Yang M, Renn A et al (2023) ChatGPT versus the neurosurgical written boards: a comparative analysis of artificial intelligence/machine learning performance on neurosurgical board-style questions. J Neurosurg 139(3):904–911. https://doi.org/10.3171/2023.2.JNS23419CrossRefPubMed

Toyama Y, Harigai A, Abe M, Nagano M, Kawabata M, Seki Y et al (2024) Performance evaluation of ChatGPT, GPT-4, and Bard on the official board examination of the Japan radiology society. Jpn J Radiol 42(2):201–207. https://doi.org/10.1007/s11604-023-01491-2CrossRefPubMed

Cheung BHH, Lau GKK, Wong GTC, Lee EYP, Kulkarni D, Seow CS et al (2023) ChatGPT versus human in generating medical graduate exam multiple choice questions - A multinational prospective study (Hong Kong S.A.R., Singapore, Ireland, and the United Kingdom). PLoS One 18(8):e0290691. https://doi.org/10.1371/journal.pone.0290691CrossRefPubMedPubMedCentral

Chen TC, Multala E, Kearns P, Delashaw J, Dumont A, Maraganore D et al (2023) Assessment of ChatGPT’s performance on neurology written board examination questions. BMJ Neurol Open 5(2):e000530. https://doi.org/10.1136/bmjno-2023-000530CrossRefPubMedPubMedCentral

Chandwar K, PrasannaMisra D (2024) What does artificial intelligence mean in rheumatology? Arch Rheumatol 39(1):1–9. https://doi.org/10.46497/ArchRheumatol.2024.10664CrossRefPubMedPubMedCentral

Stoel B (2020) Use of artificial intelligence in imaging in rheumatology - current status and future perspectives. RMD Open 6(1). https://doi.org/10.1136/rmdopen-2019-001063

McMaster C, Bird A, Liew DFL, Buchanan RR, Owen CE, Chapman WW et al (2022) Artificial intelligence and deep learning for rheumatologists. Arthritis Rheumatol 74(12):1893–1905. https://doi.org/10.1002/art.42296CrossRefPubMedPubMedCentral

Krusche M, Callhoff J, Knitza J, Ruffer N (2024) Diagnostic accuracy of a large language model in rheumatology: comparison of physician and ChatGPT-4. Rheumatol Int 44(2):303–306. https://doi.org/10.1007/s00296-023-05464-6CrossRefPubMed

10.

Miloslavsky EM, Marston B (2022) The challenge of addressing the rheumatology workforce shortage. J Rheumatol 49(6):555–557. https://doi.org/10.3899/jrheum.220300CrossRefPubMed

11.

Cipolletta E, Fiorentino MC, Vreju FA, Moccia S, Filippucci E (2024) Editorial: artificial intelligence in rheumatology and musculoskeletal diseases. Front Med (Lausanne) 11:1402871. https://doi.org/10.3389/fmed.2024.1402871CrossRefPubMed

12.

Madrid-García A, Rosales-Rosado Z, Freites-Nuñez D, Pérez-Sancristóbal I, Pato-Cour E, Plasencia-Rodríguez C et al (2023) Harnessing ChatGPT and GPT-4 for evaluating the rheumatology questions of the Spanish access exam to specialized medical training. Sci Rep 13(1):22129. https://doi.org/10.1038/s41598-023-49483-6CrossRefPubMedPubMedCentral

13.

Battafarano DF, Ditmyer M, Bolster MB, Fitzgerald JD, Deal C, Bass AR et al (2018) 2015 American College of Rheumatology Workforce Study: supply and demand projections of adult rheumatology workforce, 2015–2030. Arthritis Care Res (Hoboken) 70(4):617–626. https://doi.org/10.1002/acr.23518CrossRefPubMed

14.

Rudnicka E, Napierała P, Podfigurna A, Męczekalski B, Smolarczyk R, Grymowicz M (2020) The world health organization (WHO) approach to healthy ageing. Maturitas 139:6–11. https://doi.org/10.1016/j.maturitas.2020.05.018CrossRefPubMedPubMedCentral

15.

Sinsky C, Colligan L, Li L, Prgomet M, Reynolds S, Goeders L et al (2016) Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Ann Intern Med 165(11):753–760. https://doi.org/10.7326/M16-0961CrossRefPubMed

16.

Nagy G, Roodenrijs NM, Welsing PM, Kedves M, Hamar A, van der Goes MC et al (2022) EULAR points to consider for the management of difficult-to-treat rheumatoid arthritis. Ann Rheum Dis 81(1):20–33. https://doi.org/10.1136/annrheumdis-2021-220973CrossRefPubMed

17.

Solomon DH, Rudin RS (2020) Digital health technologies: opportunities and challenges in rheumatology. Nat Rev Rheumatol 16(9):525–535. https://doi.org/10.1038/s41584-020-0461-xCrossRefPubMed

18.

Lohr KM, Clauser A, Hess BJ, Gelber AC, Valeriano-Marcet J, Lipner RS et al (2015) Performance on the adult rheumatology in-training examination and relationship to outcomes on the rheumatology certification examination. Arthritis Rheumatol 67(11):3082–3090. https://doi.org/10.1002/art.39281CrossRefPubMed

19.

Caspi R, Karp PD (2024) An evaluation of ChatGPT and Bard (Gemini) in the context of biological knowledge retrieval. Access Microbiol 6(6). https://doi.org/10.1099/acmi.0.000790.v3

20.

Isleem UN, Zaidat B, Ren R, Geng EA, Burapachaisri A, Tang JE et al (2024) Can generative artificial intelligence pass the orthopaedic board examination? J Orthop 53:27–33. https://doi.org/10.1016/j.jor.2023.10.026CrossRefPubMed

21.

Jeong HK, Park C, Henao R, Kheterpal M (2023) Deep learning in dermatology: a systematic review of current approaches, outcomes, and limitations. JID Innov 3(1):100150. https://doi.org/10.1016/j.xjidi.2022.100150CrossRefPubMed

22.

Kelly BS, Judge C, Bollard SM, Clifford SM, Healy GM, Aziz A et al (2022) Radiology artificial intelligence: a systematic review and evaluation of methods (RAISE). Eur Radiol 32(11):7998–8007. https://doi.org/10.1007/s00330-022-08784-6CrossRefPubMedPubMedCentral

23.

Rodriguez JPM, Rodriguez R, Silva VWK, Kitamura FC, Corradi GCA, de Marchi ACB et al (2022) Artificial intelligence as a tool for diagnosis in digital pathology whole slide images: a systematic review. J Pathol Inform 13:100138. https://doi.org/10.1016/j.jpi.2022.100138CrossRefPubMedPubMedCentral

24.

Zinchenko V, Chetverikov S, Akhmad E, Arzamasov K, Vladzymyrskyy A, Andreychenko A et al (2022) Changes in software as a medical device based on artificial intelligence technologies. Int J Comput Assist Radiol Surg 17(10):1969–1977. https://doi.org/10.1007/s11548-022-02669-1CrossRefPubMedPubMedCentral

25.

Tang L, Li J, Fantus S (2023) Medical artificial intelligence ethics: A systematic review of empirical studies. Digit Health 9:20552076231186064. https://doi.org/10.1177/20552076231186064CrossRefPubMedPubMedCentral

26.

WHO Guidance (2021) Ethics and governance of artificial intelligence for health. World Health Organization. https://www.who.int/publications/i/item/9789240029200. Accessed August 2024

27.

Khan RA, Jawaid M, Khan AR, Sajjad M (2023) ChatGPT - Reshaping medical education and clinical management. Pak J Med Sci 39(2):605–607. https://doi.org/10.12669/pjms.39.2.7653CrossRefPubMedPubMedCentral

28.

Eggmann F, Blatz MB (2023) ChatGPT: chances and challenges for dentistry. Compend Contin Educ Dent 44:220–224. https://doi.org/10.2345/37075729CrossRefPubMed

29.

Sallam M (2023) ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare (Basel) 11:887. https://doi.org/10.3390/healthcare11060887CrossRefPubMed

30.

Giannakopoulos K, Kavadella A, Aaqel Salim A, Stamatopoulos V, Kaklamanos EG (2023) Evaluation of the performance of generative AI large language models ChatGPT, Google Bard, and Microsoft Bing Chat in supporting evidence-based dentistry: comparative mixed methods study. J Med Internet Res 25:e51580. https://doi.org/10.2196/51580CrossRefPubMedPubMedCentral

31.

Gemini Team Google, Anil R, Borgeaud S, Wu Y, Alayrac JB, Yu J, Soricut R, Schalkwyk J, Dai AM, Hauth A, Millican K et al (2023) Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. Accessed Aug 2024

Title: Comparative performance of artificial intelligence models in rheumatology board-level questions: evaluating Google Gemini and ChatGPT-4o
Authors: Enes Efe Is
Ahmet Kivanc Menekseoglu
Publication date: 28-09-2024
Publisher: Springer International Publishing
Keyword: Artificial Intelligence
Published in: Clinical Rheumatology / Issue 11/2024
Print ISSN: 0770-3198
Electronic ISSN: 1434-9949
DOI: https://doi.org/10.1007/s10067-024-07154-5

An atypical nail lesion in a patient with psoriatic disease

Psoriatic Arthritis
CLINICAL IMAGE

Elderly versus younger patients with microscopic polyangiitis vasculitis (MPA): a single-center retrospective study

Vasculitis
ORIGINAL ARTICLE

Dynamic change of lymphocytes associated with short-term prognosis in anti-MDA5-positive dermatomyositis with interstitial lung disease: a multicenter retrospective study

Open Access
Dermatomyositis
ORIGINAL ARTICLE

TLR2 reprograms glucose metabolism in CD4+ T cells of rheumatoid arthritis patients to mediate cell hyperactivation and TNF-α secretion

Rheumatoid Arthritis
ORIGINAL ARTICLE

In vitro silencing of RIP2 in naive CD4+ T cells from lupus-prone mice promotes pathogenic Th17 cell differentiation

Systemic Lupus Erythematosus
ORIGINAL ARTICLE

Correction: Biomarkers reflecting the pathogenesis, clinical manifestations, and guide therapeutic approach in systemic sclerosis: a narrative review

Open Access
Systemic Sclerosis
Correction

Keynote webinar | Spotlight on menopause

Menopause
Video Series

Menopause can have a significant impact on the body, with effects ranging beyond the endocrine and reproductive systems. Learn about the broader systemic effects of menopause, so you can help patients in your clinics through the transition.

Prof. Martha Hickey

Dr. Claudia Barth

Dr. Samar El Khoudary

Developed by: Springer Medicine

Watch now

Keynote webinar | Spotlight on adolescent vaping

Live
Webinar | 29-01-2025 | 18:00 (CET)

Growing numbers of young people are using e-cigarettes, despite warnings of respiratory effects and addiction. How can doctors tackle the epidemic, and what health effects should you prepare to manage in your clinics?

Watch it live: Wednesday 29th January, 18:00-19:30 CET

Prof. Ann McNeill

Dr. Debbie Robson

Benji Horwell

Developed by: Springer Medicine

Join the webinar

Keynote webinar | Spotlight on modern management of frailty

Frailty in Geriatric Patients
On-demand webinar

Frailty has a significant impact on health and wellbeing, especially in older adults. Our experts explain the factors that contribute to the development of frailty and how you can manage the condition and reduce the risk of disability, dependency, and mortality in your patients.

Prof. Alfonso Cruz-Jentoft

Prof. Barbara C. van Munster

Prof. Mirko Petrovic

Developed by: Springer Medicine

Watch now

A quick guide to ECGs

Electrocardiography
Training Course

Improve your ECG interpretation skills with this comprehensive, rapid, interactive course. Expert advice provides detailed feedback as you work through 50 ECGs covering the most common cardiac presentations to ensure your practice stays up to date.

PD Dr. Carsten W. Israel

Developed by: Springer Medizin

Start the cases

At a glance: The STEP trials

Obesity
Clinical Trial Summary

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.

Developed by: Springer Medicine

Keynote webinar | Spotlight on adolescent vaping

Springer Medicine

Abstract

Objectives

Method

Results

Conclusions

Please log in to get access to this content