2026-10-10T21:42:38Z https://api.figshare.com/v2/oai

oai:figshare.com:article/33872113 2026-09-17T05:54:19Z category_396 portal_316 item_type_3 month_year_09_2026

Data Sheet 8_A comparative cross-sectional evaluation of generative AI chatbots for patient-oriented bacterial vaginosis health advice: safety, accuracy, guideline concordance, empathy, and readability.pdf Ling Miao (1533910) Lina Gu (237116) Zhaole Gong (24548625) Zhengfeng Gu (24704371) Xiaoli Qian (10205046) Reproduction bacterial vaginosis chatbot generative artificial intelligence large language model patient education patient safety Background<p>Bacterial vaginosis (BV) concerns involve intimate symptoms, stigma, diagnostic uncertainty, medication use, pregnancy, sexual health, and self-care. Publicly accessible generative artificial intelligence chatbots offer immediate and potentially non-judgmental information, but the quality and safety of specific responses remain uncertain.</p>Objective<p>To compare five publicly accessible generative AI chatbot interfaces using standardized, researcher-developed patient-oriented questions about BV. In this study, patient-oriented denotes consumer-facing wording and does not imply direct patient derivation or validation.</p>Methods<p>Sixty-one standardized English-language questions were submitted once to ChatGPT, Gemini, Microsoft Copilot, DeepSeek, and Doubao in separate single-turn sessions, yielding 305 responses. Five senior obstetrician-gynecologists independently evaluated safety, accuracy, study-specific guideline-anchored concordance, and empathy using a question-specific reference framework. Independent ratings were locked before adjudication and were used to estimate inter-rater agreement. Consensus scores were used for primary comparisons, and a sensitivity analysis used the median of the five locked ratings. Readability was assessed separately using six formula-based indices. Paired comparisons used Cochran's Q, Friedman, McNemar, and Wilcoxon signed-rank tests with Benjamini-Hochberg adjustment.</p>Results<p>Thirty-one responses (10.2%) were classified as unsafe or potentially unsafe. Observed interface-specific rates ranged from 6.6% (95% CI, 2.6%–15.7%) to 14.8% (95% CI, 8.0%–25.7%), while the matched binary comparison did not detect an overall difference (Cochran's Q = 2.596, df = 4, P = 0.627). Under the documented query conditions, differences were detected in accuracy (Kendall's W = 0.686), study-specific guideline-anchored concordance (W = 0.243), empathy (W = 0.322), and all six readability indices (W range, 0.482–0.679; all P < 0.001). The sensitivity analysis produced the same inferential conclusions. Unsafe content occurred in every interface and was concentrated in recurrent-BV, sexual-health, medication, and self-care scenarios.</p>Conclusions<p>This exploratory study characterizes a recorded sample of specific chatbot responses, not stable or repeatable performance characteristics. Clinically relevant risks were observed in every interface, and the non-significant safety comparison should not be interpreted as equivalence. Because each researcher-developed prompt was submitted once and the interfaces were queried on different dates in a fixed order, the findings do not establish a stable ranking of underlying model capability, clinical effectiveness, or suitability for unsupervised care.</p> 2026-09-17T05:54:19Z Dataset Dataset 10.3389/frph.2026.1967818.s004 https://figshare.com/articles/dataset/Data_Sheet_8_A_comparative_cross-sectional_evaluation_of_generative_AI_chatbots_for_patient-oriented_bacterial_vaginosis_health_advice_safety_accuracy_guideline_concordance_empathy_and_readability_pdf/33872113 CC BY 4.0