Use of ChatGPT to Generate Informed Consent for Surgery in Urogynecology

Objectives: Use of the publicly available Large Language Model (LLM), Chat Pretrained Generative Transformer (ChatGPT 3.5, OpenAI 2022), is growing in healthcare despite varying accuracy. ChatGPT is a convenient, accessible tool for both patients and healthcare providers, and can give empathetic answers. To incorporate how patients and physicians may use this LLM, we asked ChatGPT to answer questions encompassing informed surgical consent and evaluated the quality, accuracy, and readability of responses.

Methods: Five fellowship-trained Urogynecology and Pelvic Reconstructive Surgery (URPS) attendings and one Reconstructive Female Urologist evaluated ChatGPT’s responses to questions about four surgeries: retropubic mid-urethral sling (rMUS), total vaginal hysterectomy (TVH), uterosacral ligament suspension (USLS), and sacrocolpopexy (SCP). Questions included: “What is a …”; “What are the risks/benefits/alternatives of a …”; “Where can I look for more information about …”; and “Where did you get your information about …”. Responses were rated using the DISCERN tool (a validated instrument for assessing quality of healthcare information using a 5-point scale), a 4-point accuracy scale, and the Flesch-Kinkaid Grade Level (FKG) score.

Results: Questions 1-8 of the DISCERN tool yielded a median score of 3 (IQR 2-5), indicating that reviewers felt answers were “partially” readable and trustworthy (Table 1). Questions 9-15 yielded a median score of 5 (IQR 4-5), indicating that reviewers thought specific details about treatment options were accurate. The median DISCERN tool overall rating was 3 (IQR 3-4), indicating a moderate rating (“potentially important but not serious shortcomings”). RMUS received the highest overall score (median 4, IQR 3-4), and USLS received the lowest (median 3, IQR 3-3).

Using the 4-point accuracy scale, 44.0% of responses received a score of 4 (“correct and adequate”), 22.6% received a score of 3 (“correct but insufficient”), 29.8% received a score of 2 (“accurate and misleading information together”) and 3.6% received a score of 1 (“wrong or irrelevant answer”). TVH scored the highest overall with 57.1% receiving a score of 4. ChatGPT performance was notably poor for discussion of benefits and alternatives for all surgeries, with some responses being inaccurate.

The mean FKG score for all responses was 17.5 (SD 2.1), corresponding to a post-graduate reading level, which is only readable by 24% of the general public.

Table 1 legend: rMUS=retropubic mid-urethral sling, TVH=total vaginal hysterectomy, USLS=uterosacral ligament suspension, SCP=sacrocolpopexy, IQR= interquartile range, SD=standard deviation.

Conclusions: Overall, ChatGPT generated accurate responses to questions about surgical informed consent. However, it produced clearly false portions of responses, highlighting the need for careful review of responses by qualified healthcare professionals. ChatGPT failed to provide concrete references to sources, resulting in an overall moderate rating of response quality. The advanced reading level of responses may limit comprehension by the general public, and further study of patient perception is needed.