Simulation-based evaluation of large language models for comorbidity detection in sleep medicine : a pilot study on ChatGPT o1 preview
Loading...
Date issued
Editors
Journal Title
Journal ISSN
Volume Title
Publisher
Reuse License
Abstract
Purpose: Timely identification of comorbidities is critical in sleep medicine, where large language models (LLMs) like ChatGPT are
currently emerging as transformative tools. Here, we investigate whether the novel LLM ChatGPT o1 preview can identify individual
health risks or potentially existing comorbidities from the medical data of fictitious sleep medicine patients.
Methods: We conducted a simulation-based study using 30 fictitious patients, designed to represent realistic variations in demo-
graphic and clinical parameters commonly seen in sleep medicine. Each profile included personal data (eg, body mass index, smoking
status, drinking habits), blood pressure, and routine blood test results, along with a predefined sleep medicine diagnosis. Each patient
profile was evaluated independently by the LLM and a sleep medicine specialist (SMS) for identification of potential comorbidities or
individual health risks. Their recommendations were compared for concordance across lifestyle changes and further medical measures.
Results: The LLM achieved high concordance with the SMS for lifestyle modification recommendations, including 100% concor-
dance on smoking cessation (κ = 1; p < 0.001), 97% on alcohol reduction (κ = 0.92; p < 0.001) and endocrinological examination (κ =
0.92; p < 0.001) or 93% on weight loss (κ = 0.86; p < 0.001). However, it exhibited a tendency to over-recommend further medical
measures (particularly 57% concordance for cardiological examination (κ = 0.08; p = 0.28) and 33% for gastrointestinal examination
(κ = 0.1; p = 0.22)) compared to the SMS.
Conclusion: Despite the obvious limitation of using fictitious data, the findings suggest that LLMs like ChatGPT have the potential to
complement clinical workflows in sleep medicine by identifying individual health risks and comorbidities. As LLMs continue to
evolve, their integration into healthcare could redefine the approach to patient evaluation and risk stratification. Future research should
contextualize the findings within broader clinical applications ideally testing locally run LLMs meeting data protection requirements.
Description
Keywords
Citation
Published in
Nature and science of sleep, 17, Dove Medical Press, Macclesfield u.a., 2025, https://doi.org/10.2147/NSS.S510254