Commonly Asked Questions in Urogynecology - Accuracy and Limitations of ChatGPT

Friday, October 25, 2024

12:57 PM – 1:01 PM EDT

Location: Liberty Ballroom (Meeting Level 4)

ChatGPT Brief Discern (BD) Average Scores

ChatGPT BD Score Table.docx

Objectives: The ease of use of Large language models (LLMs) such as Chat Generative Pre-training Transformer (ChatGPT) has facilitated their wide-scale use by patients and physicians. Nonetheless, this artificial intelligence (AI) technology is still in its infancy. There is a paucity of literature regarding its accuracy in healthcare, and to date, none in urogynecology. The aim of the present study is to assess ChatGPT’s ability to accurately answer commonly asked questions in urogynecology based on published guidelines using a validated scoring tool.

Methods: We developed 10 commonly asked questions by patients in a urogynecology office that could be answered by published guidelines. These questions were fed into ChatGPT. Each question was asked in a separate instance to avoid bias in response. Answers were evaluated by a panel of board certified urogynecologists for appropriateness and assigned a score using the Brief DISCERN (BD) scoring system. BD is a validated healthcare information assessment questionnaire that evaluates the quality of healthcare content. Each question is scored 0 to 5 with a maximum score of 30. A score of ≥ 16 is consistent with good quality content. Responses were graded based on their accuracy and consistency with expert opinion and published guidelines. Statistical analyses were performed using JMP v.16.2 software (SAS Institute Inc, Cary, NC, USA).

Results: The average BD score across all 10 questions was 18.9 ± 2.7. Twelve out of thirteen (92.3%) questions had a response that was determined to be of good quality (BD ≥ 16). The lowest scoring topic was “Pelvic Organ Prolapse” (mean BD=14.0 ± 2.0). The highest scoring topic was “Interstitial Cystitis” (mean BD=22.0 ± 0). ChatGPT provided no references for its responses.

Conclusions: ChatGPT provided appropriate responses to 92.3% of the questions based on an expert panel’s review with the BD scoring system. ChatGPT may be a promising option for patients looking for answers to their urogynecological problems on the internet. However, given the lack of references in its responses and evolving nature of LLM technology, continued analysis is essential before ChatGPT can be accepted as an accurate and reliable source for urogynecological information.

Disclosure(s):

Gregory Vurture, MD: No financial relationships to disclose

Nicole P. Jenkins, DO: No financial relationships to disclose

James H. Ross, MD: No financial relationships to disclose

Stephanie A. Sansone, M.D.: No financial relationships to disclose

Ellen Conner, MD: No financial relationships to disclose

Nina Jacobson, MD: No financial relationships to disclose

Scott W. Smilen, MD: No financial relationships to disclose

Jonathan D. Baum, Doctor of Medicine: No financial relationships to disclose

Presenting Author(s)

Gregory Vurture, MD

Fellow
Albert Einstein - Montefiore Medical Center
Port Monmouth, New Jersey, United States

Disclosure(s): No financial relationships to disclose

Co-Author(s)

Nicole P. Jenkins, DO

Resident
Hackensack Meridian Health - Jersey Shore University Medical Center
EATONTOWN, New Jersey, United States

Disclosure(s): No financial relationships to disclose
JR

James H. Ross, MD

Urogynecologist
Trinity Health IHA Medical Group
Bloomfield Hills, Michigan, United States

Disclosure(s): No financial relationships to disclose
SS

Stephanie A. Sansone, M.D.

Attending Urogynecologist
Hackensack Meridian Health Jersey Shore University Medical Center
Neptune, New Jersey, United States

Disclosure(s): No financial relationships to disclose
EC

Ellen Conner, MD (she/her/hers)

Staff physician
Jersey Shore University Medical Center
Neptune, New Jersey, United States

Disclosure(s): No financial relationships to disclose
NJ

Nina Jacobson, MD

Assistant Professor of Urogynecology
Hackensack Meridian Health
Neptune City, New Jersey, United States

Disclosure(s): No financial relationships to disclose
SS

Scott W. Smilen, MD (he/him/his)

Chair, Department of Obstetrics and Gynecology; Professor Hackensack Meridian School of Medicine
Jersey Shore University Medical Center/Hackensack Meridian School of Medicine
Mantoloking, New Jersey, United States

Disclosure(s): No financial relationships to disclose
JB

Jonathan D. Baum, Doctor of Medicine

Associate Professor
Hackensack Meridian School of Medicine
Neptune, New Jersey, United States

Disclosure(s): No financial relationships to disclose