AI Translation of Patient Education Handouts; How Good Are They?

Objectives: Language barriers reduce patient and provider satisfaction, directly affecting patient care. Patient education materials allow patients to understand conditions, provide aid in making decisions, and bridge gaps in communication with health care providers. Materials are not always available in a patient’s language. Artificial intelligence (AI), provides an inexpensive way to translate patient handouts.

Methods: We chose two languages; Chinese, the most spoken language in the world, and Swahili, a less common language to translate. The study consisted of three stages: Stage I: Translation of the MUS IUGA leaflet to Chinese or Swahili using Google translate. Stage II: Back translating the MUS leaflet to English using Google translate. Stage III: Evaluating differences in PEMAT scores as well as the CL and SI as developed by Sperger et al between the original and back translated English versions. To reduce bias, participants were randomized to either the back translation of the Chinese or Swahili version of the leaflet and were then randomized to which version was scored first (original English versus back translated). After reading the first pamphlet, the participant completed the PEMAT. After reading the second version, they completed the PEMAT as well as questions about the CL and SI of the two pamphlets. PEMAT understandability and actionability scores range from 0 to 100 with higher scores indicating better understanding and actionability. CL scores range from 1= extremely comparable to 7 not comparable at all; SI scores range from 1- extremely similar to 7-not at all similar. To demonstrate no difference in PEMAT scores between the original and back translated English version, 36 PEMAT scores were required to be 80% sure that the lower limit of a one-sided 90% confidence interval will be above the non-inferiority limit of 5.

Results: Eighteen urogynecology providers evaluated 36 PEMAT scores. Providers consisted of attendings, residents, fellows, advanced practice providers, and nurses. Understandability of the original English and back translated Chinese version were found to be different but actionability was similar. Neither comparisons reached our non-inferiority margin of 5. Comparison of the original English version and the back translated Swahili version found no differences in understandability or actionability. Likewise, these comparisons did not meet our non-inferiority margin. For comparability of language, the mean score was a 4.4 +/-0.9 which was moderately comparable. Lastly, the similarity of interpretation was a 4 +/-1.2, ranked as moderately similar.

Conclusions: Neither the original or back translated version of MUS IUGA handouts scored perfectly on understandability and actionability and we were unable to determine that the back translated versions were non-inferior to the original English version with our assumed non-inferiority margin of 5. CL and SI support that Google translate is a moderately useful tool in providing IUGA handouts in languages not commonly available.