Evaluating GPT’s Capability in Identifying Stages of Cognitive Impairment from Electronic Health Data These authors contributed equally

Yu Leng, Yingnan He, Colin Magdamo, Ana-Maria Vranceanu, Christine S. Ritchie, Shibani S. Mukerji, Lidia M. V. R. Moura, John R. Dickson, Deborah Blacker, Sudeshna Das · 2025

Read Evaluating GPT’s Capability in Identifying Stages of Cognitive Impairment from Electronic Health Data These authors contributed equally by Yu Leng, Yingnan He, Colin Magdamo, Ana-Maria Vranceanu, Christine S. Ritchie, Shibani S. Mukerji, Lidia M. V. R. Moura, John R. Dickson, Deborah Blacker, Sudeshna Das (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09715.txt.

Introduction

Yu Leng and nine co-authors present a 2025 paper testing whether a large language model can stage cognitive impairment from health records. Clinicians bury dementia clues in free text notes, and manual chart review is slow and error prone. An automated reader could speed both diagnosis and research.

The team asked a chat model, with no task specific training, to grade memory clinic notes from 769 patients on a standard dementia scale. Its scores agreed strongly with expert ratings. A second task separated normal cognition from impairment in broader records. The paper reports where the model matched specialists and where it strayed, weighing its use as a screening aid.

Catalog record

CREATORYu Leng, Yingnan He, Colin Magdamo, Ana-Maria Vranceanu, Christine S. Ritchie, Shibani S. Mukerji, Lidia M. V. R. Moura, John R. Dickson, Deborah Blacker, Sudeshna Das
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-09715

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating GPT’s Capability in Identifying Stages of Cognitive Impairment from Electronic Health Data These authors contributed equally Yu Leng, Yingnan He, Colin Magdamo, Ana-Maria Vranceanu, Christine S. Ritchie, Shibani S. Mukerji, Lidia M. V. R. Moura, John R. Dickson, Deborah Blacker, Sudeshna Das Abstract Identifying cognitive impairment within electronic health records (EHRs) is crucial not only for timely diagnoses but also for facilitating research. Information about cognitive impairment often exists within unstructured clinician notes in EHRs, but manual chart reviews are both time-consuming and error-prone. To address this issue, our study evaluates an automated approach using zero-shot GPT-4o to determine stage of cognitive impairment in two different tasks. First, we evaluated the ability of GPT-4o to determine the global Clinical Dementia Rating (CDR) on specialist notes from 769 patients who visited the memory clinic at Massachusetts General Hospital (MGH), and achieved a weighted kappa score of 0.83. Second, we assessed GPT-4o’s ability to differentiate between normal cognition, mild cognitive impairment (MCI), and dementia on all notes in a 3-year window from 860 Medicare patients. GPT-4o attained a weighted kappa score of 0.91 in comparison to specialist chart reviews and 0.96 on cases that the clinical adjudicators rated with high confidence. Our findings demonstrate GPT-4o’s potential as a scalable chart review tool for creating research datasets and assisting diagnosis in clinical settings in the future. \jmlrpages 1 1 footnotemark: 1 † † workshop: Machine Learning for Health (ML4H) 2024 keywords GPT, EHR, MCI, CDR Data and Code Availability We will share our GitHub repository for the camera-ready version of the paper. Due to the presence of PHI, the data cannot be shared publicly. Institutional Review Board (IRB) Relevant ethics approval information will be provided if the paper is accepted. 1 Introduction Alzheimer’s Disease and Related Dementias (ADRD, referred to hereafter as dementia) describes a group of related neurodegenerative disorders that affect over 6 million people over age 65 in the United States and represent a large and growing problem in the 21st century ( Association 2024 ) . Timely diagnosis of dementia is crucial for interventions and treatment plans that can help manage symptoms and improve the quality of life for persons living with dementia and their families ( Robinson et al. 2015 ) . Yet, dementia remains under-recognized, under-diagnosed, and under-reported in healthcare records ( Amjad et al. 2018 ) . Automated mining of clinical notes have the potential to facilitate clinical diagnosis as well as research studies of dementia. The Electronic Health Record (EHR)—which includes detailed health history, clinical notes, and other health-system interaction information—offers readily available data and great potential for identifying cognitive impairment in patients without a formal diagnosis in EHR. Despite the prevalence of cognitive impairment data within EHR, these critical insights are often buried in unstructured clinician notes and not readily accessible for clinical decision-making or research. Traditional methods for extracting this information involve labor-intensive manual reviews, which are not only time-consuming but also prone to inconsistencies and errors. To address this gap, several prior studies have applied natural language processing (NLP) and/or large-language models (LLMs) to detect cognitive impairment in clinical notes within EHR, for example Yan et al. 2024 However, to our knowledge none of the prior efforts have applied the latest GPT models to this problem. This study introduces and evaluates the use GPT-4o to automate the extraction and interpretation of cognition data from EHRs. We evaluated GPT-4o in two different studies. First, we use GPT-4o to assign a global CDR score on specialist notes (with detailed cognitive evaluation) on patients who visited the memory clinic at Massachusetts General Hospital (MGH). These memory specialist notes have detailed information on cognitive evaluation; our goal was to evaluate whether we could automatically create structured datasets of patient global CDR scores. Second, we evaluated GPT-4o’s performance in assessing stage of cognitive impairment (normal cognition (NC), mild cognitive impairment (MCI), and dementia) in a Medicare patient group using all notes spanning a 3-year period. Here, the motivation was to compare automated to manual chart reviews for either research or clinical diagnosis. 2 Datasets and Processing For the first study, the dataset comprised of 769 latest visit notes of 769 unique patients from the memory clinic at MGH from February 2016 to July 2019. (tab:memdemo). These patients consented to be part of a registry which recorded the global CDR score and diagnoses at their visit, along with other data. Any sentence with mention of CDR was redacted from the notes using regex before evaluation by GPT-4o. GPT-4o was prompted to assign a global CDR score. For the second study, the dataset consisted of a sample of 860 Medicare fee-for-service patients from a previous study by Moura et al. 2021 . Each patient’s EHR data between 01/01/2016 – 12/31/2018 was reviewed by an expert physician to label patients with the stage of cognitive impairment (Normal, Normal-to-MCI, MCI, MCI-to-dementia, and dementia) and to assign a confidence level of 1 (lowest) to 4 (highest). The MCI-to-dementia patients were included in the MCI category and Normal-to-MCI were excluded to get three final categories: NC, MCI, and dementia. For this dataset (tab:mourademo), we prepared a summary of summaries with GPT-4o. For each patient, we aggregated outpatient visit notes in chronological order. The context of the note (i.e., the date, department, specialty) was added to each note and summarized by GPT-4o. The notes summaries were then combined chronologically into one document. For each patient, GPT-4o was prompted to generate a ”summary of summaries” and make a final diagnosis based on this summary of summaries, classifying the patient’s cognitive status as: NC, MCI, or Dementia. Global CDR Score Total N 0 0.5 1 2 3 Characteristics N=769 N=38 (5%) N=267 (35%) N=218 (28%) N=179 (23%) N=67 (9%) Age (Mean, SD) 77.9 (8.0) 72.3 (9.5) 76.6 (7.5) 78.3 (7.7) 80.2 (7.4) 79.1 (8.6) Female 397 (52%) 21 (55%) 111 (42%) 120 (55%) 109 (61%) 36 (54%) Male 372 (48%) 17 (45%) 156 (58%) 98 (45%) 70 (39%) 31 (46%) Cognitive Impairment Stage Total N Normal MCI Dementia N=860 N=530 N=106 N=224 Characteristics (62%) (12%) (26%) Age (Mean, SD) 75.8 (6.5) 78.2 (6.6) 83.1 (7.5) Female 304 (57%) 53 (50%) 150 (67%) Male 226 (43%) 53 (50%) 74 (33%) 3 Methodology In this study, we evaluated the capability of GPT-4o to identify stage of cognitive impairment by implementing a series of prompt engineering techniques and a Retrieval-Augmented Generation (RAG) approach in the two datasets described above. 3.1 GPT, Prompt Engineering and Retrieval-Augmented Generation For Study I, we tried three approaches. The first approach utilized a structured answer template to guide GPT-4o’s analysis of patient visit notes. The response format asks GPT-4o to provide observations and summaries across six key domains in the CDR scoring system ( Hughes et al. 1982 ) : i) Memory, ii) Orientation, iii) Judgment and Problem Solving, iv) Community Affairs, v) Home and Hobbies, and vi) Personal Care. The model was required to conclude with an explicit CDR score. This structured format helped standardize the output and ensured that each relevant domain was systematically considered before making the final decision. Second, to further enhance GPT-4o’s ability to determine stages of cognitive impairment, we implemented a Retrieval-Augmented Generation (RAG) approach. We extracted information

Read the complete text