26dkou623 10.3205/26dkou623 urn:nbn:de:0183-26dkou6233 Meeting Abstract Lanter Lanter Lea L

Universitätsklinikum Balgrist, Zürich, Schweiz

author Masel Masel Sophie S

Universitätsklinikum Balgrist, Zürich, Schweiz

author Hochreiter Hochreiter Bettina B

Universitätsklinikum Balgrist, Zürich, Schweiz

author Meisterhans Meisterhans Michel M

Universitätsklinikum Balgrist, Zürich, Schweiz

author Rebsamen Rebsamen Michael M

Universitätsklinikum Balgrist, Zürich, Schweiz

author Herzog Herzog Benedikt B

Universitätsklinikum Balgrist, Zürich, Schweiz

author Caprara Caprara Sebastiano S

Universitätsklinikum Balgrist, Zürich, Schweiz

author Oettl Oettl Felix F

Universitätsklinikum Balgrist, Zürich, Schweiz

author German Medical Science GMS Publishing House

Düsseldorf

610 20261020 engl This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 License. Dieser Artikel ist ein Open-Access-Artikel und steht unter den Lizenzbedingungen der Creative Commons Attribution 4.0 License (Namensnennung). M0659 623 Deutsche Gesellschaft für Orthopädie und Unfallchirurgie Deutsche Gesellschaft für Orthopädie und Orthopädische Chirurgie Deutsche Gesellschaft für Unfallchirurgie Berufsverband für Orthopädie und Unfallchirurgie Deutscher Kongress für Orthopädie und Unfallchirurgie (DKOU 2026) Posterpreise klinisch Berlin 20261020 20261023 DKOU26_Abs_2091 TextObjectives and questions: Generative artificial intelligence (AI), particularly large language models (LLMs), are increasingly utilized in healthcare. While this may reduce the initial workload of healthcare professionals, unvalidated model outputs can pose a relevant risk to patient safety if they are inaccurate, incomplete, or inconsistent with recommendations. This study evaluates and compares the performance of local and on-line LLMs on patient questions to inform institutional strategies for patient communication and management. We hypothesize that there are significant differences in responses between locally deployed and commercially available LLMs.Material and methods: Twenty postoperative patient questions were constructed and posed to GPT-5, Claude 4.5 (commercial models) and GPT-OSS, Apertus (locally run). Responses were assessed using the QUEST framework evaluating quality, understanding, expression, safety and trust. Four blinded reviewers, 2 board certified fellowship trained orthopedic surgeons and 2 orthopedic surgery residents assessed responses. Model performance based on reviewer response was compared using Kruskal Wallis/Friedman test. Metrics where lower values indicate better performance were inverted so that higher values uniformly represent better performance across all metrics. Inter rater agreement was tested with Cohen’s K and Fleiss’ K.Results: The highest overall performance scores were observed for Claude 4.5 (mean 0.949, SD 0.124) and GPT-5 (0.937, 0.142), followed by GPT-OSS (0.873, 0.189), while Apertus performed worst (0.693, 0.317). On QUEST dimensions, GPT-5 and Claude 4.5 achieved consistently high ratings, whereas Apertus scored lower across information quality, reasoning, and expression. Safety-relevant issues were concentrated in Apertus, with harmful content and fabrication each occurring in 22.5% of outputs (18/80), compared with near-zero rates in the other models. Inter-rater agreement was moderate (overall Fleiss’ κ≈0.50).Discussion and conclusions: Commercial LLMs currently provide the safest and most coherent tools for patient communication, but they remain externally controlled. Achieving privacy-compliant, real-time clinical integration will require advancing local LLMs through fine-tuning, rigorous validation, and robust safety guardrails. 0 0 0 0