Research Article: Performance of four large language model chatbots in responding to patient-facing questions about osteoporotic vertebral compression fractures: a cross-sectional paired evaluation
Abstract:
Large language model (LLM) chatbots are increasingly used by the public to obtain health information, but the safety and quality of their responses to questions about osteoporotic vertebral compression fractures (OVCFs) remain unclear.
To compare the safety and patient-facing response quality of four publicly accessible LLM chatbots when responding to standardized questions about osteoporotic vertebral compression fractures.
In this cross-sectional paired evaluation, 36 standardized OVCF-related patient-facing questions were submitted to each of four publicly accessible LLM chatbot interfaces—ChatGPT, Gemini, DeepSeek, and Doubao—in an initial response-generation round on June 20, 2026, from Jiangsu, China. Queries were conducted in English in independent single-turn sessions under matched conditions, yielding 144 responses for the primary between-chatbot analysis. A second independent generation round using the same 36 questions and matched query procedures was conducted 7 days later (June 27, 2026), yielding an additional 144 responses for within-model stability analysis. The questions covered symptoms and warning signs, diagnosis and imaging, conservative management, medication safety, rehabilitation, vertebral augmentation, fracture prevention, and special clinical scenarios. Five clinical raters, blinded to chatbot identity and generation round, independently evaluated safety, factual accuracy, empathic communication, treatment-related information quality using DISCERN, patient-information quality using EQIP, transparency using the JAMA benchmark criteria, and overall quality using the Global Quality Score. Readability was assessed automatically using six established indices. Between-chatbot comparisons and test–retest stability analyses used paired methods appropriate for the outcome type.
Inter-rater agreement was high, with a Fleiss’ kappa of 0.818 for safety and ICC(2,1) values of 0.816 for accuracy, 0.820 for empathy, 0.820 for DISCERN, 0.820 for EQIP, 0.821 for JAMA, and 0.818 for GQS. Potentially unsafe responses were identified for every chatbot: 3 of 36 ChatGPT responses (8.3%), 5 of 36 DeepSeek responses (13.9%), 6 of 36 Gemini responses (16.7%), and 7 of 36 Doubao responses (19.4%). The overall paired difference in safety was not statistically significant (Cochran's Q?=?3.621, df?=?3, P =?0.305). Factual accuracy differed significantly but only modestly among chatbots ( P =?0.009; Kendall's W?=?0.108). Empathy showed substantially greater variation ( P <?0.001; W?=?0.817), with higher scores for ChatGPT and Gemini than for DeepSeek and Doubao. DISCERN, EQIP, JAMA benchmark, and GQS scores also differed significantly among chatbots (all P <?0.001). Doubao produced responses with the greatest estimated textual complexity. Across the two generation rounds, safety agreement ranged from 88.9% to 97.2% (Gwet's AC1, 0.838–0.966), and median ICC(2,1) values across the 12 non-safety outcomes ranged from 0.781 to 0.904.
In this time-stamped, single-location, two-round evaluation, all four chatbot interfaces produced some responses classified as potentially unsafe despite generally high clinician-rated factual accuracy. Differences were greater in empathy, information quality, transparency, overall quality, and readability than in factual accuracy. The observed within-model stability was generally good but did not eliminate variability across generations. These findings are specific to the evaluated access location, language, single-turn conditions, and generation dates and should not be assumed to represent stable performance across settings or future model updates. LLM-generated OVCF information should not be used as a standalone patient education resource; any potential use should involve clinician oversight, safety safeguards, and further validation.
Introduction:
Large language model (LLM) chatbots are increasingly used by the public to obtain health information, but the safety and quality of their responses to questions about osteoporotic vertebral compression fractures (OVCFs) remain unclear.
Read more