Evaluating AI Chatbot Responses to Patient Queries About Postpartum Incontinence

Objectives: This study aims to evaluate the accuracy and completeness of three artificial intelligence (AI) chatbots in addressing patient questions regarding postpartum urinary incontinence. Secondary objectives are to determine the level of variability between three different chatbots, to assess the level of inter-rater reliability between generalists and urogynecologists, and to evaluate the readability of AI-generated answers.

Methods: Three AI chatbots, ChatGPT-3.5 (Open AI), Claude (Anthropic), and Gemini (Google), were asked to generate answers to six basic patient questions about postpartum incontinence. Responses were qualitatively compared to expert opinion on postpartum incontinence and rated on a grading scale by three experienced generalists and three urogynecologists. Accuracy was graded using a 6-point Likert scale from (1) completely incorrect to (6) completely correct. Completeness was graded using a 3-point Likert scale, described as (1) incomplete, (2) adequate, (3) comprehensive. Final scores were averaged between all six graders and separately between the three generalists and the three urogynecologists. Data analysis included descriptive statistics and a two-sample t-test. Answers from each of the three chatbots were additionally analyzed for readability using established scales with the Flesch Reading Ease Score and Flesch-Kincaid Grade Level.

Results: Across all six questions, the mean accuracy score was 5.06 (nearly all correct) with SD of 0.85 and the mean completeness score was 2.5 (between adequate and comprehensive) with SD of 0.59 for all chatbots. ChatGPT had the highest average accuracy score of 5.2, followed by Claude with 5.1, and Gemini with 4.9. In terms of completeness, Claude had the highest average score of 2.7, followed by ChatGPT and Gemini both with 2.4. There were no answers that were unanimously rated as completely correct. Only one chatbot answer regarding duration of postpartum incontinence was unanimously rated as comprehensive. The model most associated with correct and comprehensive responses on average was ChatGPT. Accuracy and completeness scores differed between generalists and urogynecologists, with mean accuracy score 5.35 vs 4.78, respectively (p < 0.05), and completeness mean score 2.31 vs 2.67, respectively (p < 0.05). The readability of answers from each chatbot was moderate, with Claude having the highest Flesch reading ease score of 49.8 and therefore lowest Flesch-Kincaid grade level of 8.9. ChatGPT had the lowest reading ease score of 30.1 and highest grade level of 12.

Conclusions: Findings of this study suggest that AI chatbots generally produce mostly accurate information to medical queries about postpartum urinary incontinence, with adequate to comprehensive completeness. Chatbot generated answers are predominantly written at a high school reading level, thus limiting the understandability of such information. Such limitations are important to recognize as generative AI continues to grow with increased accessibility to both providers and patients.