25gmds122 10.3205/25gmds122 urn:nbn:de:0183-25gmds1224 Meeting Abstract Credidio Credidio Graziella G

University Medical Center Hamburg-Eppendorf (UKE), Institute for Applied Medical Informatics, Hamburg, Germany

author Größler Größler Michael M

University Medical Center Hamburg-Eppendorf (UKE), Institute for Applied Medical Informatics, Hamburg, Germany

author Düsterbeck Düsterbeck Lilly Marie LM

University Medical Center Hamburg-Eppendorf (UKE), Institute for Applied Medical Informatics, Hamburg, Germany

author Löhndorf Löhndorf Anke A

University Medical Center Hamburg-Eppendorf (UKE), Department of Biochemistry and Molecular Cell Biology, Hamburg, Germany

author Eisenbarth Eisenbarth Sophie S

University Medical Center Hamburg-Eppendorf (UKE), Department of Biochemistry and Molecular Cell Biology, Hamburg, Germany

author Guse Guse Andreas H. AH

University Medical Center Hamburg-Eppendorf (UKE), Department of Biochemistry and Molecular Cell Biology, Hamburg, Germany University Medical Center Hamburg-Eppendorf (UKE), Dean’s Office for Education and Students’ Affairs, Hamburg, Germany

author Riemann Riemann Layla Tabea LT

University Medical Center Hamburg-Eppendorf (UKE), Institute for Applied Medical Informatics, Hamburg, Germany

author German Medical Science GMS Publishing House

Düsseldorf

610 large language model (LLM) personalized learning platform medical education retrieval-augmented generation 20251103 engl This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 License. Dieser Artikel ist ein Open-Access-Artikel und steht unter den Lizenzbedingungen der Creative Commons Attribution 4.0 License (Namensnennung). M0631 122 Deutsche Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie 70. Jahrestagung der Deutschen Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie e. V. (GMDS) PS 4: Fachbereichsübergreifende Themen Jena 20250907 20250911 Abstr. 228 Claussen-Simon-Stiftung TextIntroduction: Large Language Models (LLMs) are transforming medical education, influencing teaching methodologies , assisting medical writing , and personalizing learning material , , among others. Techniques like Retrieval-Augmented Generation (RAG) have gained attention, enabling not only explainability but also grounding responses. While LLM-based platforms for education exist, there are no open-source, flexible, and scalable tools tailored to a German university curriculum. Our RAG-powered tool, KiMED, will soon provide individualized learning material and progress reports, bridging knowledge gaps of prerequisite subjects relevant to medical studies. Initially, students assess their knowledge with a test on a selected subject. Topics where they showed weaker performance are prioritized, guiding generation of learning plans. KiMED also supports exam preparation with a curated pool of multiple-choice questions (MCQs) with explanations on alternatives. First evaluations of the prototype regarding KiMED’s relevance and usability are depicted. Methods: Internally crafted slides, texts, and license-compliant textbooks sections were incorporated in the pipeline. These were chunked by title, metadata mapped, and a hybrid retrieval system (semantic matching and keyword search) was implemented. For data privacy reasons and German fluency, a compact multilingual LLM running locally was selected. RAGAS framework , prompt engineering, in-line source citation, metadata filtering, human evaluation, were used to assess and improve retrieval and generation quality.The system is scalable for multiple simultaneous accesses, being hosted by servers of the University of Hamburg connected to local servers running the database and LLM serving engine.A prototype tool for biochemistry is currently under evaluation. Final improvements are being guided by a System Usability Scale (SUS) along with qualitative feedback within the tool. Preliminary evaluation was performed by two biochemistry experts, three medical students, one natural sciences master’s student, and four AI specialists.Results: The model Qwen2.5-7B-Instruct was used for content generation. A multi-prompting approach coupled with agents and structured outputs ensured pre-established quality criteria were followed. Final SUS scores ranged from 72.5 to 87.5. All ten evaluators strongly agreed they would use the tool more frequently and eight expressed strong confidence using it. Three were neutral regarding success of features integration. Qualitative feedback suggested color-blind accessibility, changes in navigability and button placement.Discussion: Although small evaluators sample size, our first results indicate that KiMED is intuitive and user-friendly, needing minor adjustments. Assessment of the tool’s usability by a broader cohort of medical students and experts will be carried out. Content correctness is challenging in LLM-based systems. So far, human-in-the-loop as final quality assurance ensures educational and legal compliance. To improve quality of generated content before human review, we plan to apply more robust hallucination detection frameworks and a self-optimizing multi-agentic approach for MCQs generation. To enhance learning outcomes, next steps include incorporation of chatbot and spaced repetition. Conclusion: KiMED will further extend to additional medical topics at our University Medical Center and, as an open-source platform, can be adapted by other faculties, thus enhancing personalized learning on bigger scopes and help reshaping medical education.Acknowledgements: AHG and LTR gratefully acknowledge funding by Claussen-Simon-Stiftung, Hamburg.The authors declare that they have no competing interests.The authors declare that an ethics committee vote is not required. Grunhut J Marques O Wyatt A Needs, Challenges, and Applications of Artificial Intelligence in Medical Education Curriculum 2022 JMIR Med Educ e35587 Grunhut J, Marques O, Wyatt A. Needs, Challenges, and Applications of Artificial Intelligence in Medical Education Curriculum. JMIR Med Educ. 2022;8(2):e35587. DOI: 10.2196/35587 http://dx.doi.org/10.2196/35587 Ali SR Dobbs TD Hutchings HA Whitaker IS Using ChatGPT to write patient clinic letters 2023 Lancet Digit Health e179–81 Ali SR, Dobbs TD, Hutchings HA, Whitaker IS. Using ChatGPT to write patient clinic letters. Lancet Digit Health. 2023;5(4):e179–81. DOI: 10.1016/S2589-7500(23)00048-1 http://dx.doi.org/10.1016/S2589-7500(23)00048-1 Abd-alrazaq A AlSaad R Alhuwail D Ahmed A Healy PM Latifi S Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions 2023 JMIR Med Educ e48291 Abd-alrazaq A, AlSaad R, Alhuwail D, Ahmed A, Healy PM, Latifi S, et al. Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions. JMIR Med Educ. 2023;9:e48291. DOI: 10.2196/48291 http://dx.doi.org/10.2196/48291 Anonym Anki - powerful, intelligent flashcards Anki - powerful, intelligent flashcards. [cited 2025 Apr 18]. Available from: https://apps.ankiweb.net/ https://apps.ankiweb.net/ Es S James J Espinosa-Anke L Schockaert S RAGAS: Automated Evaluation of Retrieval Augmented Generation [Preprint] 2023 arXiv Es S, James J, Espinosa-Anke L, Schockaert S. RAGAS: Automated Evaluation of Retrieval Augmented Generation [Preprint]. arXiv. 2023 Jan 1. DOI: 10.48550/arXiv.2309.15217 http://dx.doi.org/10.48550/arXiv.2309.15217 Brooke J SUS: A “Quick and Dirty” Usability Scale 1996 Usability Evaluation in Industry Brooke J. SUS: A “Quick and Dirty” Usability Scale. In: Usability Evaluation in Industry. London: CRC Press; 1996. Qwen Yang A Yang B Zhang B Hui B Zheng B Qwen2.5 Technical Report [Preprint] 2025 arXiv Qwen, Yang A, Yang B, Zhang B, Hui B, Zheng B, et al. Qwen2.5 Technical Report [Preprint]. arXiv. 2025. DOI: 10.48550/arXiv.2412.15115 https://doi.org/10.48550/arXiv.2412.15115 0 0 0 0