Development of a ChatGPT-style App for Ob-Gyn Resident Education

Objectives: Generative Artificial Intelligence (GenAI) technology is currently under investigation for its applications in healthcare, with limited exploration in the context of medical education, despite its significant potential. Commercial GenAI apps like ChatGPT are limited due to not having been trained on domain-specific medical knowledge, and not tuned for a medical education use case.
This study explored emerging machine learning techniques in medical education. We demonstrate the creation of a mobile, chat-based, GenAI application specifically trained on domain-specific content of Obstetrics and Gynecology speciality.

Methods: We leveraged Retrieval-Augmented Generation (RAG), an innovative technique to extend a foundational large language model (LLM).
Data collection process involved gathering high-quality Ob/Gyn literature, including practice bulletins, committee opinions, and topic summaries from resident physicians. Data was chunked and stored in a vector database with an embedding model. A low-code bot building platform facilitated orchestration of data and question/answering, and enabled the deployment of a user-friendly chat interface for testing. We iteratively refined the system prompts and incorporated few-shot examples for prompt optimization.
Thirteen Ob/Gyn residents participated in testing and subjective feedback.

Results: We successfully developed and deployed a GenAI application aligned to the Ob/Gyn domain. Residents used the app to seek answers from evidence-based literature, finding it convenient on mobile devices. The chat interface was intuitive, while the quality of literature summarization met acceptable standards. Some users expressed a preference for more specific actionable responses, especially in cases related to cervical screening management.

Conclusions: Retrieval augmented generation can be an effective method for aligning a foundational LLM model to be domain-specific to the ob/gyn speciality. Such a system has the potential to be a valuable tool for assessing the accuracy and safety of GenAI in answering specialty-specific medical questions. The development of domain-specific GenAI applications could significantly enhance medical education and improve the accessibility of relevant medical knowledge.