Top

Published in:

Open Access 27-04-2023 | Bariatric Surgery | Original Contributions

Assessing the Accuracy of Responses by the Language Model ChatGPT to Questions Regarding Bariatric Surgery

Authors: Jamil S. Samaan, Yee Hui Yeo, Nithya Rajeev, Lauren Hawley, Stuart Abel, Wee Han Ng, Nitin Srinivasan, Justin Park, Miguel Burch, Rabindra Watson, Omer Liran, Kamran Samakar

Published in: Obesity Surgery | Issue 6/2023

Abstract

Purpose

ChatGPT is a large language model trained on a large dataset covering a broad range of topics, including the medical literature. We aim to examine its accuracy and reproducibility in answering patient questions regarding bariatric surgery.

Materials and methods

Questions were gathered from nationally regarded professional societies and health institutions as well as Facebook support groups. Board-certified bariatric surgeons graded the accuracy and reproducibility of responses. The grading scale included the following: (1) comprehensive, (2) correct but inadequate, (3) some correct and some incorrect, and (4) completely incorrect. Reproducibility was determined by asking the model each question twice and examining difference in grading category between the two responses.

Results

In total, 151 questions related to bariatric surgery were included. The model provided “comprehensive” responses to 131/151 (86.8%) of questions. When examined by category, the model provided “comprehensive” responses to 93.8% of questions related to “efficacy, eligibility and procedure options”; 93.3% related to “preoperative preparation”; 85.3% related to “recovery, risks, and complications”; 88.2% related to “lifestyle changes”; and 66.7% related to “other”. The model provided reproducible answers to 137 (90.7%) of questions.

Conclusion

The large language model ChatGPT often provided accurate and reproducible responses to common questions related to bariatric surgery. ChatGPT may serve as a helpful adjunct information resource for patients regarding bariatric surgery in addition to standard of care provided by licensed healthcare professionals. We encourage future studies to examine how to leverage this disruptive technology to improve patient outcomes and quality of life.

Graphical Abstract

Available only for authorised users

O'Connor S. ChatGPT. Open artificial intelligence platforms in nursing education: tools for academic progress or abuse? Nurse Educ Pract. 2023;66:103537. https://doi.org/10.1016/j.nepr.2022.103537.CrossRef

Graham F. Daily briefing: will ChatGPT kill the essay assignment? Nature. 2022; https://doi.org/10.1038/d41586-022-04437-2.

Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, Chartash D. How does ChatGPT perform on the United States Medical Licensing Examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023;8(9):e45312. https://doi.org/10.2196/45312.CrossRef

Sarraju A, Bruemmer D, Van Iterson E, Cho L, Rodriguez F, Laffin L. Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chat-based artificial intelligence model. JAMA. 2023;329(10):842. https://doi.org/10.1001/jama.2023.1044.CrossRefPubMed

Yeo YH, Samaan JS, Ng WH, Ting PS, Trivedi H, Vipani A, Ayoub W, Yang JD, Liran O, Spiegel B, Kuo A. Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clin Mol Hepatol. 2023; https://doi.org/10.3350/cmh.2023.0089.

Obesity is a common, serious, and costly disease. Centers for Disease Control and Prevention. Accessed January 21, 2023. https://www.cdc.gov/obesity/data/adult.html

Arterburn DE, Telem DA, Kushner RF, Courcoulas AP. Benefits and risks of bariatric surgery in adults: a review. JAMA. 2020 Sep;324(9):879–87. https://doi.org/10.1001/jama.2020.12567.CrossRefPubMed

New study finds most bariatric surgeries performed in northeast, and fewest in south where obesity rates are highest, and economies are weakest. American Society for Metabolic and Bariatric Surgery. Published online November 15, 2018. Accessed January 23, 2023. https://asmbs.org/articles/new-study-finds-most-bariatric-surgeries-performed-in-northeast-and-fewest-in-south-where-obesity-rates-are-highest-and-economies-are-weakest

Premkumar A, Samaan JS, Samakar K. Factors associated with bariatric surgery referral patterns: a systematic review. J Surg Res. 2022;276:54–75. https://doi.org/10.1016/j.jss.2022.01.023.CrossRef

10.

Rajeev ND, Samaan JS, Premkumar A, Srinivasan N, Yu E, Samakar K. Patient and the public’s perceptions of bariatric surgery: a systematic review. J Surg Res. 2023;283:385–406. https://doi.org/10.1016/j.jss.2022.10.061.CrossRefPubMed

11.

Scarano Pereira JP, Martinino A, Manicone F, Scarano Pereira ML, Iglesias Puzas Á, Pouwels S, Martínez JM. Bariatric surgery on social media: a cross-sectional study. Obes Res Clin Pract. 2022;16(2):158–62. https://doi.org/10.1016/j.orcp.2022.02.005.CrossRefPubMed

12.

openai. ChatGPT: optimizing language models for dialogue. 2023; https://openai.com/blog/chatgpt/. Accessed 1/1/2023, 2023.

13.

Ouyang L, Wu J, Jiang X, et al. Training language models to follow instructions with human feedback. Adv Neur Inform Proc Syst. 2022;35:27730. https://doi.org/10.48550/ARXIV.2203.02155.CrossRef

14.

Athanasiadis DI, Roper A, Hilgendorf W, Voss A, Zike T, Embry M, Banerjee A, Selzer D, Stefanidis D. Facebook groups provide effective social support to patients after bariatric surgery. Surg Endosc. 2021;35(8):4595–601. https://doi.org/10.1007/s00464-020-07884-y.CrossRefPubMed

15.

Koball AM, Jester DJ, Domoff SE, Kallies KJ, Grothe KB, Kothari SN. Examination of bariatric surgery Facebook support groups: a content analysis. Surg Obes Relat Dis. 2017;13(8):1369–75. https://doi.org/10.1016/j.soard.2017.04.025.CrossRefPubMed

16.

Batar N, Kermen S, Sevdin S, Yıldız N, Güçlü D. Assessment of the quality and reliability of information on nutrition after bariatric surgery on YouTube. Obes Surg. 2020;30:4905–10. https://doi.org/10.1007/s11695-020-05015-z.CrossRef

17.

Corcelles R, Daigle CR, Talamas HR, Brethauer SA, Schauer PR. Assessment of the quality of Internet information on sleeve gastrectomy. Surg Obes Relat Dis. 2015;11(3):539–44. https://doi.org/10.1016/j.soard.2014.08.014.CrossRefPubMed

18.

Koball AM, Jester DJ, Pruitt MA, Cripe RV, Henschied JJ, Domoff S. Content and accuracy of nutrition-related posts in bariatric surgery Facebook support groups. Surg Obes Relat Dis. 2018;14:1897–902. https://doi.org/10.1016/j.soard.2018.08.017.CrossRefPubMed

19.

Meleo-Erwin Z, Basch C, Fera J, Ethan D, Garcia P. Readability of online patient-based information on bariatric surgery. Health Promot Perspect. 2019;9(2):156–60. https://doi.org/10.15171/hpp.2019.22.CrossRefPubMedPubMedCentral

20.

Mahoney ST, Strassle PD, Farrell TM, Duke MC. Does lower level of education and health literacy affect successful outcomes in bariatric surgery? J Laparoendosc Adv Surg Tech A. 2019;29(8):1011–5. https://doi.org/10.1089/lap.2018.0806.CrossRefPubMed

21.

Erdogdu UE, Cayci HM, Tardu A, Demirci H, Kisakol G, Guclu M. Health literacy and weight loss after bariatric surgery. Obes Surg. 2019;29(12):3948–53. https://doi.org/10.1007/s11695-019-04060-7.CrossRefPubMed

22.

Miller-Matero LR, Hecht L, Patel S, Martens KM, Hamann A, Carlin AM. The influence of health literacy and health numeracy on weight loss outcomes following bariatric surgery. Surg Obes Relat Dis. 2021;17(2):384–9. https://doi.org/10.1016/j.soard.2020.09.021.CrossRefPubMed

23.

Hecht LM, Martens KM, Pester BD, Hamann A, Carlin AM, Miller-Matero LR. Adherence to medical appointments among patients undergoing bariatric surgery: do health literacy, health numeracy, and cognitive functioning play a role? Obes Surg. 2022;32(4):1391–3. https://doi.org/10.1007/s11695-022-05905-4.CrossRefPubMed

24.

Hecht L, Cain S, Clark-Sienkiewicz SM, Martens K, Hamann A, Carlin AM, Miller-Matero LR. Health literacy, health numeracy, and cognitive functioning among bariatric surgery candidates. Obes Surg. 2019;29(12):4138–41. https://doi.org/10.1007/s11695-019-04149-z.CrossRef

25.

Theiss LM, Wood T, McLeod MC, Shao C, Santos Marques ID, Bajpai S, Lopez E, Duong AM, Hollis R, Morris MS, Chu DI. The association of health literacy and postoperative complications after colorectal surgery: a cohort study. Am J Surg. 2022;223(6):1047–52. https://doi.org/10.1016/j.amjsurg.2021.10.024.CrossRefPubMed

26.

Baker S, Malone E, Graham L, Dasinger E, Wahl T, Titan A, Richman J, Copeland L, Burns E, Whittle J, Hawn M, Morris M. Patient-reported health literacy scores are associated with readmissions following surgery. Am J Surg. 2020;220(5):1138–44. https://doi.org/10.1016/j.amjsurg.2020.06.071.CrossRefPubMed

27.

Williamson JM, Rink JA, Hewin DH. The portrayal of bariatric surgery in the UK print media. Obes Surg. 2012;22(11):1690–4. https://doi.org/10.1007/s11695-012-0701-5.CrossRefPubMed

Title: Assessing the Accuracy of Responses by the Language Model ChatGPT to Questions Regarding Bariatric Surgery
Authors: Jamil S. Samaan
Yee Hui Yeo
Nithya Rajeev
Lauren Hawley
Stuart Abel
Wee Han Ng
Nitin Srinivasan
Justin Park
Miguel Burch
Rabindra Watson
Omer Liran
Kamran Samakar
Publication date: 27-04-2023
Publisher: Springer US
Keywords: Bariatric Surgery
Artificial Intelligence
Bariatric Surgery
Published in: Obesity Surgery / Issue 6/2023
Print ISSN: 0960-8923
Electronic ISSN: 1708-0428
DOI: https://doi.org/10.1007/s11695-023-06603-5

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Assessing the Accuracy of Responses by the Language Model ChatGPT to Questions Regarding Bariatric Surgery

Abstract

Purpose

Materials and methods

Results

Conclusion

Graphical Abstract

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Purpose

Materials and methods

Results

Conclusion

Graphical Abstract

Please log in to get access to this content

Other articles of this Issue 6/2023

Practical Issues with Using Direct Oral Anticoagulant After Bariatric Metabolic Surgery

One Anastomosis Gastric Bypass (OAGB) with a 150-cm Biliopancreatic Limb (BPL) Versus a 200-cm BPL, a Systematic Review and Meta-analysis

Remote Follow-up with a Mobile Application Is Equal to Traditional Outpatient Follow-up After Bariatric Surgery: the BELLA Pilot Trial

Bilateral Pulmonary Embolism Provoked by Endoscopic Sleeve Gastroplasty: Is There a Need for Venous Thromboembolism Prophylaxis After Endo-Bariatric Procedures?

Psychological Profiles and 12-Month Weight Outcomes of Women Evaluated for Reoperative Bariatric Surgery

A Systematic Review Investigating Maternal Nutrition During Pregnancy After Bariatric Surgery