Text

Objectives and questions: Generative artificial intelligence (AI), particularly large language models (LLMs), are increasingly utilized in healthcare. While this may reduce the initial workload of healthcare professionals, unvalidated model outputs can pose a relevant risk to patient safety if they are inaccurate, incomplete, or inconsistent with recommendations. This study evaluates and compares the performance of local and on-line LLMs on patient questions to inform institutional strategies for patient communication and management. We hypothesize that there are significant differences in responses between locally deployed and commercially available LLMs.

Material and methods: Twenty postoperative patient questions were constructed and posed to GPT-5, Claude 4.5 (commercial models) and GPT-OSS, Apertus (locally run). Responses were assessed using the QUEST framework evaluating quality, understanding, expression, safety and trust. Four blinded reviewers, 2 board certified fellowship trained orthopedic surgeons and 2 orthopedic surgery residents assessed responses. Model performance based on reviewer response was compared using Kruskal Wallis/Friedman test. Metrics where lower values indicate better performance were inverted so that higher values uniformly represent better performance across all metrics. Inter rater agreement was tested with Cohen’s K and Fleiss’ K.

Results: The highest overall performance scores were observed for Claude 4.5 (mean 0.949, SD 0.124) and GPT-5 (0.937, 0.142), followed by GPT-OSS (0.873, 0.189), while Apertus performed worst (0.693, 0.317). On QUEST dimensions, GPT-5 and Claude 4.5 achieved consistently high ratings, whereas Apertus scored lower across information quality, reasoning, and expression. Safety-relevant issues were concentrated in Apertus, with harmful content and fabrication each occurring in 22.5% of outputs (18/80), compared with near-zero rates in the other models. Inter-rater agreement was moderate (overall Fleiss’ κ≈0.50).

Discussion and conclusions: Commercial LLMs currently provide the safest and most coherent tools for patient communication, but they remain externally controlled. Achieving privacy-compliant, real-time clinical integration will require advancing local LLMs through fine-tuning, rigorous validation, and robust safety guardrails.