Top

Knee Surgery, Sports Traumatology, Arthroscopy

Published in:

Open Access 08-08-2023 | Artificial Intelligence | SPORTS MEDICINE

Exploring the potential of ChatGPT as a supplementary tool for providing orthopaedic information

Authors: Janina Kaarre, Robert Feldt, Laura E. Keeling, Sahil Dadoo, Bálint Zsidai, Jonathan D. Hughes, Kristian Samuelsson, Volker Musahl

Published in: Knee Surgery, Sports Traumatology, Arthroscopy | Issue 11/2023

Abstract

Purpose

To investigate the potential use of large language models (LLMs) in orthopaedics by presenting queries pertinent to anterior cruciate ligament (ACL) surgery to generative pre-trained transformer (ChatGPT, specifically using its GPT-4 model of March 14th 2023). Additionally, this study aimed to evaluate the depth of the LLM’s knowledge and investigate its adaptability to different user groups. It was hypothesized that the ChatGPT would be able to adapt to different target groups due to its strong language understanding and processing capabilities.

Methods

ChatGPT was presented with 20 questions and response was requested for two distinct target audiences: patients and non-orthopaedic medical doctors. Two board-certified orthopaedic sports medicine surgeons and two expert orthopaedic sports medicine surgeons independently evaluated the responses generated by ChatGPT. Mean correctness, completeness, and adaptability to the target audiences (patients and non-orthopaedic medical doctors) were determined. A three-point response scale facilitated nuanced assessment.

Results

ChatGPT exhibited fair accuracy, with average correctness scores of 1.69 and 1.66 (on a scale from 0, incorrect, 1, partially correct, to 2, correct) for patients and medical doctors, respectively. Three of the 20 questions (15.0%) were deemed incorrect by any of the four orthopaedic sports medicine surgeon assessors. Moreover, overall completeness was calculated to be 1.51 and 1.64 for patients and medical doctors, respectively, while overall adaptiveness was determined to be 1.75 and 1.73 for patients and doctors, respectively.

Conclusion

Overall, ChatGPT was successful in generating correct responses in approximately 65% of the cases related to ACL surgery. The findings of this study imply that LLMs offer potential as a supplementary tool for acquiring orthopaedic knowledge. However, although ChatGPT can provide guidance and effectively adapt to diverse target audiences, it cannot supplant the expertise of orthopaedic sports medicine surgeons in diagnostic and treatment planning endeavours due to its limited understanding of orthopaedic domains and its potential for erroneous responses.

Level of evidence

Available only for authorised users

GPT-4. 2023; https://openai.com/research/gpt-4.

OpenAI. Introducing ChatGPT 2023; https://openai.com/blog/chatgpt.

WHO calls for safe and ethical AI for health 2023; https://www.who.int/news/item/16-05-2023-who-calls-for-safe-and-ethical-ai-for-health.

Ali R, Tang OY, Connolly ID, Sullivan PLZ, Shin JH, Fridley JS, et al. (2023) Performance of ChatGPT and GPT-4 on Neurosurgery Written Board Examinations. arXiv. Preprint posted online. https://doi.org/10.1101/2023.03.25.23287743

Ayers JW, Poliak A, Dredze M, Leas EC, Zhu Z, Kelley JB, et al. (2023) Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Intern Med; https://doi.org/10.1001/jamainternmed.2023.1838

Beltrami EJ, Grant-Kels JM, (2023) Consulting ChatGPT: Ethical dilemmas in language model artificial intelligence. J Am Acad Dermatol; https://doi.org/10.1016/j.jaad.2023.02.052

Borji A (2023) A Categorical Archive of ChatGPT Failures. arXiv. Preprint posted https://doi.org/10.48550/arXiv.2302.03494

Bubeck S, Chandrasekaran V, Eldan R, Gehrke R, Horvitz E, Kamar E, et al. (2023) Sparks of Artificial General Intelligence: Early experiments with GPT-4. arXiv preprint; https://doi.org/10.48550/arXiv.2303.12712

Diermeier T, Rothrauff BB, Engebretsen L, Lynch AD, Ayeni OR, Paterno MV et al (2020) Treatment after anterior cruciate ligament injury: panther symposium ACL treatment consensus group. Orthop J Sports Med 8:2325967120931097PubMedPubMedCentral

10.

Grünebaum A, Chervenak J, Pollet SL, Katz A, Chervenak FA (2023) The exciting potential for ChatGPT in obstetrics and gynecology. Am J Obstet Gynecol. https://doi.org/10.1016/j.ajog.2023.03.009CrossRefPubMedPubMedCentral

11.

Gupta R, Park JB, Bisht C, Herzog I, Weisberger J, Chao J, et al. (2023) Expanding Cosmetic Plastic Surgery Research Using ChatGPT. Aesthet Surg J https://doi.org/10.1093/asj/sjad069

12.

Harrer S, (2023) Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine eBioMedicine. https://doi.org/10.1016/j.ebiom.2023.104512:

13.

Liu P, Yuan W, Fu J, Jiang Z, Hayashi H, Neubig G (2021) Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. arXiv. Preprint posted online. https://doi.org/10.48550/arXiv.2107.13586

14.

Lum ZC (2023) Can Artificial Intelligence Pass the American Board of Orthopaedic Surgery Examination? Orthopaedic Residents Versus ChatGPT. Clin Orthop Relat Res; https://doi.org/10.1097/corr.0000000000002704

15.

Nori H, King N, McKinney SM, Carignan S, Horvitz E (2023) Capabilites of GPT-4 on Medical Challenge Problems. arXiv, Preprint posted online. https://doi.org/10.48550/arXiv.2303.13375

16.

Ollivier M, Pareek A, Dahmen J, Kayaalp ME, Winkler PW, Hirschmann MT et al (2023) A deeper dive into ChatGPT: history, use and future perspectives for orthopaedic research. Knee Surg Sports Traumatol Arthrosc 31:1190–1192CrossRefPubMed

17.

OpenAI (2023) GPT-4 Technical Report. arXiv. Preprint posted online. https://doi.org/10.48550/arXiv.2303.08774.

18.

Sherman SL, Calcei J, Ray T, Magnussen RA, Musahl V, Kaeding CC et al (2021) ACL Study group presents the global trends in ACL reconstruction: biennial survey of the ACL study group. JISAKOS 6:322–328PubMed

19.

Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, et al. (2022) Large Language Models Encode Clinical Knowledge. arXiv. Preprint posted online. https://doi.org/10.48550/arXiv.2212.13138

20.

Stokel-Walker C (2023) ChatGPT listed as author on research papers: many scientists disapprove. Nature 613:620–621. https://doi.org/10.1038/d41586-023-00107-zCrossRefPubMed

21.

Stokel-Walker C, Noorden VR, (2023) What ChatGPRT and generative AI mean for science nature 614:214-216. https://doi.org/10.1038/d41586-023-00340-6

22.

Vaishya R, Misra A, Vaish A (2023) ChatGPT: Is this version good for healthcare and research? Diabetes Metab Syndr 17:102744CrossRefPubMed

23.

White J, Fu Q, Hays S, Sandborn M, Olea C, Gilbert H, et al. (2023) A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arXiv. Preprint posted online. https://doi.org/10.48550/arXiv.2302.11382

24.

Yeo YH, Samaan JS, Ng WH, Ting PS, Trivedi H, Vipani A, et al. (2023) Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clin Mol Hepatol https://doi.org/10.3350/cmh.2023.0089

Title: Exploring the potential of ChatGPT as a supplementary tool for providing orthopaedic information
Authors: Janina Kaarre
Robert Feldt
Laura E. Keeling
Sahil Dadoo
Bálint Zsidai
Jonathan D. Hughes
Kristian Samuelsson
Volker Musahl
Publication date: 08-08-2023
Publisher: Springer Berlin Heidelberg
Keyword: Artificial Intelligence
Published in: Knee Surgery, Sports Traumatology, Arthroscopy / Issue 11/2023
Print ISSN: 0942-2056
Electronic ISSN: 1433-7347
DOI: https://doi.org/10.1007/s00167-023-07529-2

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Exploring the potential of ChatGPT as a supplementary tool for providing orthopaedic information

Abstract

Purpose

Methods

Results

Conclusion

Level of evidence

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Purpose

Methods

Results

Conclusion

Level of evidence

Please log in to get access to this content

Other articles of this Issue 11/2023

Freeze-dried noncoagulating platelet-derived factor concentrate is a safe and effective treatment for early knee osteoarthritis

Risk for total knee arthroplasty after tibial plateau fractures: a systematic review

Image-free handheld robotic-assisted technology improved the accuracy of implant positioning compared to conventional instrumentation in patients undergoing simultaneous bilateral total knee arthroplasty, without additional benefits in improvement of clinical outcomes

Kinesiophobia contributes to worse functional and patient-reported outcome measures in Achilles tendinopathy: a systematic review

Arthroscopic capsular release for frozen shoulder: when etiology matters

Increased subtalar rotational motion in patients with symptomatic ankle instability under load and stress conditions