A multi-site benchmarking framework for scalable extraction of geriatric care constructs from electronic health records Authors: Sunyang Fu, Min Ji Kwak, Jaerong Ahn, Qiuhao Lu, Zhiyi Yue, Nan Wang, Huipeng Liu, Grace Giles, Alexa Cumming, Keziah M Thomas, Shreyans Sanghvi, Jiang Jun, Ming Huang, Xiaoyang Ruan, Andrew Wen, Liwei Wang, Erin Hommel, Yanshan Wang, Lichao Sun, Huiwen Xu, Chan Mi Park, Jennifer St Sauver, Jeffrey S Wefel, Nahid Rianon, Dae Hyun Kim, Hongfang Liu Published: 2026 ABSTRACT AbstractCurrent Natural Language Processing (NLP) algorithms for detecting geriatric conditions are largely limited to domain-specific models that fail to capture the interdependent, multidimensional nature of comprehensive geriatric assessment. This study aimed to develop and evaluate a comprehensive, scalable, and robust information extraction framework to identify Comprehensive Geriatric Assessment (CGA) and Age-Friendly Health Systems (AFHS) 4Ms-related data elements from unstructured electronic health record (EHR) text across multiple health systems. Using a team science approach grounded in the TRUST framework, we annotated pooled clinical notes from four health systems to produce a gold-standard dataset of 41 CGA- and 4Ms-related geriatric care data elements. Three information extraction approaches were implemented and evaluated: an in-context learning generative large language model (GPT-4o), a hybrid heuristic-LLM model (MedAgingIE), and an instruction-tuned open-source lightweight model (Qwen2-7B-Instruct). Performance was assessed on a blinded test set using macro- and micro-averaged metrics. GPT-4o achieved a macro F1-score of 0.56 and micro F1-score of 0.87; MedAgingIE achieved 0.55 and 0.92; and Qwen2-7B-Instruct achieved 0.30 and 0.81, respectively. MedAgingIE demonstrated the strongest consistency between precision and recall, while GPT-4o showed superior sensitivity for diverse, context-rich geriatric concepts. These findings highlight key trade-offs among symbolic, generative, and instruction-tuned approaches for CGA and 4Ms phenotyping, suggesting that hybrid heuristic-LLM methods offer interpretability and stability, whereas large language models provide greater adaptability for complex clinical narratives.Subject terms: Computational biology and bioinformatics, Health care, Medical research Introduction The world’s population is aging rapidly, the proportion of people aged over 65 years is expected to nearly double between 2000 and 20501. Older adults admitted to the hospital, particularly those with acute illness, multiple chronic conditions, or frailty, are at an increased risk of adverse outcomes, including prolonged hospitalization, unplanned readmission, and subsequent institutionalization to a long-term care facility2. Delivering high-quality, patient-centered care to this growing population requires a comprehensive understanding of older adults’ medical, functional, cognitive, psychological, and social needs. The Comprehensive Geriatric Assessment (CGA) is a well-established, holistic, and multidimensional approach for evaluating older adults’ medical, functional, mental, and social capabilities and managing limitations and informing individualized care planning3,4. Unlike a routine medical assessment, CGA emphasizes function and quality of life in addition to disease management, and has been shown to improve care planning and outcomes in older populations, while reducing healthcare utilization3. In parallel, the Age-Friendly Health Systems (AFHS) initiative operationalizes core geriatric care principles through the “4Ms” framework-What Matters, Medication, Mentation, and Mobility-providing an implementation-focused structure for health systems to monitor and improve care quality for older adults5–7. Both CGA and the AFHS 4Ms reflect overlapping underlying constructs of aging-related health and rely on similar clinical data elements. Routinely collected real-world data (RWD), such as Electronic Health Records (EHRs), contain abundant longitudinal information relevant to both CGA and AFHS 4Ms. However, many clinically meaningful geriatric data elements, such as delirium, falls, functional status, cognitive impairment, and social support, are inconsistently captured in structured EHR fields and predominantly documented in unstructured clinical notes or semi-structured flowsheets8–11. This reliance on narrative documentation presents a major barrier to scalable assessment, quality measurement, and clinical research for aging populations. Systematically extracting these data elements at scale is therefore essential to enable population-level geriatric phenotyping and data-driven evaluation of age-friendly, patient-centered care. Clinical concept extraction, a sub-domain of clinical natural language processing (NLP), offers programmatic solutions to extract relevant information and standardize the CGA12. Despite significant progress in clinical concept extraction, most existing NLP efforts target a narrow subset of geriatric concepts, such as activities of daily living8,13–15, delirium9,16,17, Alzheimer’s Disease and related dementias (AD/ADRD)18, and social support19,20. While valuable, these siloed approaches do not reflect the multidimensional and interdependent nature of geriatric care and limit the ability to support broader clinical, quality, and research use cases aligned with CGA or AFHS 4Ms. From a geriatric care perspective, effective care depends on a holistic understanding of patient needs that spans comprehensive CGA and 4Ms data elements across physical, cognitive, psychological, and social domains21,22. Fragmented NLP models trained on isolated domains cannot capture the interdependencies between these elements. For example, the current model identifying delirium does not answer how it is influenced by mobility or social support needs9,23. Moreover, using multiple specialized models may introduce inconsistencies in concept definitions, annotation, and data standards, limiting scalability and reproducibility across health systems. A comprehensive NLP framework capable of extracting a broad set of aging-relevant data elements from unstructured clinical text is, therefore, essential to enable larger-scale phenotyping and further facilitate data-driven evaluation of age-friendly and patient-centered care. From an informatics perspective, scalable extraction of geriatric care data elements provides the foundational infrastructure for Learning Health System (LHS), where extracted data can inform iterative feedback loops between clinical practice and quality improvement and transform routinely collected EHR data into continuously learning resources. This will further enable a range of downstream digital applications (e.g., risk prediction modeling and patient trajectory modeling for digital twins) and translational research use cases (e.g., geriatrics outcome detection, cohort identification, and population-level geriatric quality surveillance). The objective of this study was to develop and evaluate a comprehensive, real-world NLP framework for multi-site and multi-domain extraction of key geriatrics and aging-relevant clinical data elements that span both Comprehensive Geriatric Assessment and Age-Friendly Health Systems 4Ms constructs from multi-site unstructured EHR data. The framework aims to address the long-standing gap in geriatric informatics by unifying the extraction of functional, cognitive, behavioral, and social domains into a single interoperable system. To ensure both clinical trustworthiness and methodological rigor, we adopted a TRUST-based24 gold-standard development process and implemented three widely used clinical concept extraction approaches. We systematically benchmarked each component across multiple EHR systems to evaluate performance, robustness, and portability, advancing a generalizable NLP methodology to support multi-site and multi-domain geriatric informatics and age-friendly care evaluation. Results Data characteristics We developed an expert-annotated evaluation dataset comprising 49,195 fully annotated sentences, derived from 400 clinical notes stratified by concept density and drawn from aging cohorts across four sites. Figure 1 shows the geriatric concept distribution across four EHR systems. Each institution contributed distinct documentation patterns shaped by its clinical focus and care settings. Specifically, UTPhysicians (UTP, outpatient setting, n = 100) provided rich ambulatory and functional data, Memorial Hermann Health System (MHHS, inpatient setting, n = 100) contributed dense inpatient documentation, Harris County Psychiatric Center (HCPC, inpatient setting, n = 100) emphasized behavioral and cognitive aspects, and Beth Israel Deaconess Medical Center (BIDMC, ICU setting, n = 100)’s Medical Information Mart for Intensive Care (MIMIC) data offered granular critical care information. Distribution of clinical concept annotations across four healthcare sites (BIDMC, HCPC, MHHS, and UTP). Heatmaps display annotation counts for four CGA domains: (a) Goals of Care and Advanced Care Planning, (b) Socioeconomic Status, (c) Cognitive and Behavioral Status, and (d) Mobility and Functional Status. Color intensity reflects annotation frequency, with darker blue indicating higher counts. The geriatric concepts are grouped into 4 domains: Goals of Care, Socioeconomic Status, Cognitive and Behavior Status, and Mobility and Functional Status. Goals of Care is the most prevalent concept within the domain (Fig. 1a) and is frequently documented across all institutions (BIDMC = 107, UTP = 117, and MHHS = 113). This reflects frequent documentation of advance care planning discussions in the ICU, inpatient, and outpatient settings. Limitations of Life-Sustaining Treatment, Hospice, and Palliative Care concepts appear less frequently documented and are mostly found in the ICU and inpatient settings. The Socioeconomic Status (SEC) (Fig. 1b) category shows moderate representation, with HCPC contributing most SEC concepts, including Financial Strain, Food and Housing Insecurity, and Social Family Support (n = 74, 10, 77, and 50 concepts, respectively). This finding aligns with the clinical and operational context of HCPC, where psychiatric admissions often stem from or are exacerbated by Social Determinants of Health (SDOH)-related stressors25. Cognitive and Behavioral Status (Fig. 1c) shows dense and heterogeneous variable representation across four sites. BIDMC and MHHS collectively contribute high counts for Delirium, and Delirium-related acute mental status change, including Altered Mental Status (AMS), Confusion, and Disorientation. These findings highlight a consistent clinical emphasis on detecting and documenting acute cognitive fluctuations in inpatient and ICU settings. Delirium is a common condition among hospitalized patients, affecting approximately one in five general inpatients and between 20% and 80% of the most critically ill patients in ICUs26–29. On the other hand, MHHS and UTP capture more Cognitive Impairment (n = 202 and 445) and Depression (n = 469 and 191), reflecting their psychiatric and outpatient emphasis. The Mobility and Functional Status (Panel d) domain contains the most sub-concepts, with a mixture of richly and sparsely documented variables. UTP shows the highest representation within this domain, particularly for Physical Activity, Falls, and Activities of Daily Living (ADLs). This directly reflects routine objective functional assessments, including the FRAIL scale, grip strength, walking speed, and Zibro balance scores from the Geriatric Osteoporosis Clinic at UT Physicians. MHHS demonstrates the highest frequency of the Transferring concept (n = 444), indicating that mobility and transfer ability are routinely assessed by the care team in inpatient settings. Lastly, BIDMC contributes higher frequencies for ADL concepts such as feeding (n = 52) and continence (n = 44), aligning with the detailed nursing and ICU documentation typical of MIMIC data. We further quantified the overlap of geriatric care concepts across sites. Among 41 unique concepts, 20 (44.4%) were consistently observed across all four sites, representing a shared core set of geriatric constructs. A larger proportion, 39 (86.7%), appeared in at least two sites, indicating substantial cross-institutional consistency. In contrast, 6 concepts (13.3%) were site-specific, reflecting heterogeneity in clinical documentation practices. Pairwise Jaccard similarity between sites ranged from 0.571 to 0.795, suggesting moderate-to-high overlap in concept distributions across institutions. Model performance The overall average performance of the three models: GPT-4o (in-context learning using GPT4o), Qwen2-7B-Instruct (instruction-tuned Qwen model), and MedAgingIE (hybrid heuristic-LLM model) was evaluated using micro-average and macro-average F1 score, precision, and recall, as well as with per-concept metrics across all four major geriatrics domains. Micro average measures the model’s overall performance across all geriatrics-related concepts, whereas macro average represents the model’s average performance across individual concepts, giving equal weight to each CGA construct regardless of sample frequency. Among these metrics and across 41 geriatrics data elements, GPT-4o achieved scores of 0.47 in macro precision, 0.83 in macro recall, 0.56 in macro F1-score, and 0.87 in micro F1-score. Qwen2-7B-Instruct yielded 0.26 in macro precision, 0.50 in macro recall, 0.30 in macro F1-score, and 0.81 in micro F1-score. MedAgingIE yielded 0.75 in macro precision, 0.49 in macro recall, 0.55 in macro F1-score, and 0.92 in micro F1-score. From the micro average perspective (Fig. 2a), across all concepts, MedAgingIE achieved the highest overall micro-averaged F1-scores (0.92), followed by GPT-4o (0.87) and Qwen2-7B-Instruct (0.81). These results indicates that, when aggregating predictions over the entire dataset, the LLM derived, and ontology guided symbolic approach demonstrated the strongest consistency and precision-recall balance. GPT-4o outperformed the instruction-tuned 7B Qwen model. This finding suggests the advantage of large model for handling extreme diverse task. Variant performance within and across models was observed across different geriatrics domains. GPT-4o consistently demonstrated the highest performance in handling positive instance across all domains, especially in these semantically nuanced categories including cognitive and socioeconomic domains. MedAgingIE achieved the highest F1-score in handling negative cases (i.e., no mentions of geriatrics-related data elements), which is consistent with prior findings that rule based models tend to follow a conservative pattern, showing high precision but limited sensitivity. The Qwen2-7B-Instruct model, although it required the largest computational resources to develop, demonstrated overall moderate performance across all geriatric domains. Overview of three models’ performance across micro-, macro-, and per-concept averages for automated extraction of geriatric assessment and care concepts from clinical narratives: GPT-4o (in-context learning using GPT-4o), Qwen2-7B-Instruct (instruction-tuned Qwen model), and MedAgingIE (hybrid heuristic-LLM model). The models were evaluated across six aggregated geriatrics concept categories, including Cognitive and Behavioral Status (CBS), Goals of Care (GOC), Mobility and Functional Status (MFS), concepts not mapped to a specific domain (None), and Socioeconomic Status (SES). Bars represent the micro-averaged F1 score, which measures the harmonic mean of precision and recall across all instances within each category. a Micro F1-scores for GPT-4o, Qwen2-7B-Instruct, and MedAgingIE across overall and per-category performance. b Macro precision, recall, and F1-scores broken down by CGA category. c Per-concept F1-scores across all individual clinical concepts. From the macro average perspective (Fig. 2b), the overall model comparison revealed complementary strengths across models. GPT-4o demonstrated broader coverage across diverse inputs with the highest macro recall (0.83), while MedAgingIE showed stronger discriminative capability with the highest macro precision (0.75). This result indicates that LLM has stronger generalizability even when handling variant linguistics and contextual patterns in geriatrics care documentation. On the other hand, the hybrid heuristic-LLM approach can effectively capture explicit clinical expressions while minimizing false positives. The instruction-tuned small language model shows comparatively lower performance across most evaluation metrics, which potentially suggests some challenges for smaller models in handling the large-scale, multi-task (label>40), and domain-specific nuances inherent to comprehensive geriatric assessment. Among all geriatrics domains, the GOC domain remained the most challenging for all models and suggest the nuanced and implicit nature of care planning documentation (See error analysis section for more details). The per concept evaluation (Fig. 2c) showed substantial variability in model performance across the range of the geriatrics domain, indicating that automated geriatrics concept extraction remains highly complex. Certain concepts such as Hospice, Financial strain, and Continence were recognized with relatively high F1-scores across models, whereas other cognitive, behavioral, or functional descriptions, including Delirium, Disorganized thinking, Agitation, and Managing medication, were captured inconsistently or with poor F1-scores. MedAgingIE performed better on explicitly defined and ontology-guided concepts, whereas GPT-4o showed broader though uneven coverage for context-dependent expressions. Qwen2-7B-Instruct was generally performed lower across most concepts. The wide dispersion of F1-scores across concepts demonstrates that no existing approach fully addresses the diverse semantic and contextual demands of geriatrics representation in real-world multisite EHR narratives. Figure 3 compares the performance of the three models across four clinical sites. Models evaluated on the publicly available MIMIC dataset (derived from the BIDMC site) achieved the highest aggregated performance (Fig. 3a). Among the model families, generative approaches demonstrated greater performance variability across institutions than the symbolic model. Pairwise bootstrap comparisons revealed that most pairwise differences in both macro- and micro-F1 scores were statistically significant. The only non-significant comparison was observed for macro-F1 between GPT-4o and MedAgingIE at UTP, where the confidence interval included zero. These findings indicate that while performance differences are generally consistent, they are not uniformly significant across all settings (Supplementary Information 5–Table 3). Comparative performance of models across clinical sites. a Mean micro F1-scores with 95% confidence intervals for each clinical site (BIDMC, HCPC, MHHS, UTP). b Boxplots showing micro F1-score variability across sites for each model, with individual site scores overlaid as colored dots. Error analysis Quantitative systematic error analysis revealed model-specific, complex, and heterogeneous error patterns on all extraction tasks (see Supplementary Information 2 for comprehensive error definition). We first observed a distinct difference between generative (Qwen2-7B-Instruct and GPT-4o) versus non-generative (MedAgingIE) models. Common error classes for generative models include Conflation (the combination of two or more real entities into one), Omission (failure to extract a clearly mentioned concept), and Complete Hallucination (a fully fabricated or nonsensical output). These errors arise from the generative nature of next-token prediction. In contrast, common error classes for non-generative models include Implied Inference, Guideline Error, Negation, and Exclusion. Some of these errors are linked to human factors and are generally more interpretable and actionable for model refinement. Share error classes include Negation, Possible and probable language, and Exclusion. We found that false positive errors and false negative errors share distinct error characteristics, which is consistent with previous studies30,31. False positive errors were dominated by contextual and statistical inference error including Complete Hallucination, Negation, Exclusion, and Guideline Error. Among these false positive errors, the generative models were often able to extract information that was contextually relevant. For example, when describing neuropsychological assessments for bipolar disorder and other mental health conditions, the models incorrectly predicted the concept as depression. Such errors were “less wrong” compared with complete hallucination errors, where the models generated entirely irrelevant information, sometimes producing incomplete or nonsensical text. These types of errors not only affect model interpretability but are also difficult to correct. We included our perspectives and recommendations regarding these error types in the Discussion section. In contrast, false positive errors from the symbolic approach can be categorized into two main groups: contextual errors, where information is present in the surrounding context but modifies rather than defines the clinical condition, and semantic errors, where the meaning or interpretation of the clinical text is misunderstood by the model or annotator. Qualitative error analysis revealed that model misclassifications often arose from subtle linguistic and contextual ambiguities (Fig. 4). One typical example is syntactic discontinuity, such as line breaks or negations that alter meaning across sentences (e.g., “falling hematocrit, question change”), which leads models to misinterpret incomplete or negated statements as active findings. Another common error pattern is Hypothetical and Instructional expressions, including “Please contact your physician or return to the emergency room if…” or “To avoid delirium, infection… would recommend straight catheterizing.” These expressions were frequently over-interpreted as positive outcomes. List-like structures in mental status examinations (e.g., “[] irritability, [] delusions, [] hallucinations”) also caused partial extraction or false positives due to the lack of positive indicators. Similarly, quantitative test results such as “Mini-mental state examination (MMSE) score 15/16 (-1 recall)” were sometimes misread as abnormal findings despite contextual cues indicating normal performance. Lastly, we found that section headers such as “Family Medical/Psychiatric History” occasionally triggered extractions when the model failed to recognize structural rather than narrative text. Sankey diagram of systematic error analysis across three models and tasks. Nan: sentences are not associated with any of the predefined concepts. Discussion Despite growing interest in applying NLP algorithms to detect geriatric conditions, current approaches remain fragmented and limited in their ability to capture the complex nature of CGA and 4Ms. To address this critical gap, we developed a comprehensive clinical information extraction framework that identifies and standardizes CGA and 4Ms-related data elements from unstructured clinical narratives. We adopted a team science approach grounded in the TRUST24 framework to ensure transparency, rigor, and reproducibility. Through this process, a multidisciplinary team of geriatricians and medical trainees systematically annotated a pooled corpus of clinical notes from four health systems to establish a high-quality and comprehensive gold standard evaluation dataset. We implemented and evaluated three widely adopted information extraction approaches, including symbolic, instruction-tuned, and in-context generative models. To the best of our knowledge, this work represents one of the first multi-site, multi-domain evaluations of geriatric care concept extraction using both public and proprietary EHR data. This comprehensive framework based on the multi-component nature of CGA and 4Ms can be generalized to various aspects of geriatric care and enhance further development of broader use of real-world data. Here we discuss challenges, lessons learned, and our future perspective on each key step of the clinical NLP research life cycle. EHR data cascades: how data quality variability shapes model behavior and generalizability Routinely collected EHR data hold great promise for advancing use of AI in geriatrics and aging research, yet imperfections in data quality can propagate through every stage of model development, creating a cascade effect, where small, early data-related decisions or errors propagate, amplify, and compound across the AI system lifecycle and distort its performance and generalizability32. We derived four key data perspectives that we hope can provide a conceptual foundation for improving model translational readiness. Data heterogeneity as the foundation of cascades Textual EHR data are a reflection of the entire clinical and informatics ecosystem, including patient characteristics, care practices, documentation workflows, screening instruments, EHR systems, data Extract, Transform, Load (ETL) processes, and data modeling24,33. These macro and micro contextual factors can all contribute to data variability across institutions. As shown in Fig. 1, each institutional cohort displayed a very distinct distribution of geriatrics care concepts. At the population level, these differences often reflect underlying variations in clinical practices, patient characteristics, geographic distribution, and regional population characteristics. From the processes of data generation, documentation, and management, we learned that differences between EHR platforms (e.g., Epic vs. Cerner) and variations in geriatric screening instruments (e.g., CAM, Frail Scale) contribute to significant variability in textual representation. These findings highlight the importance of understanding site-specific data characteristics before model development to ensure reliable and generalizable performance. The following two sections will further discuss the implications of data quality variability for different clinical concept extraction models and how these sources of heterogeneity may introduce bias, noise, and variability that compound as models are trained and evaluated. Public versus private data and the illusion of stability Our evaluation included one public dataset (MIMIC) and three private institution EHR datasets. All approaches achieved significantly higher performance on the public dataset compared with private EHR data (Fig. 3). This advantage likely reflects the relative homogeneity of public data due to information loss caused by de-identification, segmentation and data sampling, as well as the potential risk of inadvertent data leakage that may occur in well-studied benchmark corpora - an issue that is difficult to quantify. In contrast, performance across institutional datasets was more variable, which reveals how real-world diversity exposes model instability. These findings strongly suggest that 1) performance estimates derived solely from public data may overstate model generalizability and 2) there is a strong need to incorporate private EHR data, especially those for which the model will be applied to. Model sensitivity and interaction with data quality Figure 3b further demonstrates that the performance gap between the best (public dataset) and second-best sites was significantly greater among generative models compared with those using a non-generative approach. This difference indicates that generative models can be more sensitive to contextual variability in documentation, whereas symbolic models maintain more consistent behavior across heterogeneous datasets. Such results illustrate that data cascades do not stop at data quality alone but extend to how model architecture interacts with data variability. Federated infrastructure for multi-site EHR research In addition to data harmonization, another critical path to increase the availability of multisite EHR data is through a federated infrastructure that supports distributed analysis, governance, and privacy preservation. Federated networks such as the ENACT Network provide a model of how institutional EHR data can be leveraged at scale while respecting regulatory, privacy, and institutional constraints17,34–36. ENACT builds upon the Accrual to Clinical Trials (ACT) Network to link dozens of Clinical and Translational Science Awards (CTSA) hub sites, supporting cohort discovery and eventually analytical enclaves in which site-level data remain under local control while algorithms or queries run in a federated manner. Such federated architectures can help mitigate the data cascade by 1) enabling site-aware evaluation by preserving local distributions and exposing inter-site variability, 2) supporting privacy-preserving analytics to reduce the risk of data leakage or overfitting, and 3) facilitating standard-governance mechanisms, enabling consistent data quality checks, ontology alignment, and distributed model validation across sites. Toward adaptive AI: integrating continuous human feedback Across our benchmarking experiments, the GPT-4o framework using the OpenAI GPT-4o model yielded competitive performance across most geriatric care categories. In contrast, the MedAgingIE system, which employs a hybrid heuristic-LLM approach, showed complementary strength in precision but still required substantial manual effort during development, despite being empowered by LLM-assisted corpus analysis. Instruction-tuned small language models (SLM) struggled to generalize across multitask settings involving more than forty concept categories. SLM also required substantially greater computational resources and tuning time. These findings suggest that for complex, multitask, and context-specific information extraction, leveraging in-context learning with LLMs or developing hybrid frameworks that combine symbolic and data-driven reasoning may offer a more balanced trade-off between performance and computational efficiency. Rethinking rule-based architectures for adaptivity In linear pipeline frameworks such as cTAKES37 and MedTagger38, information extraction is performed through sequential stages (e.g., section detection, sentence segmentation, context classification, concept extraction, concept normalization), where each stage depends on the outputs of the previous one. When modeling a large number of interdependent geriatric concepts, this design introduces several coordination challenges. First, error propagation can occur, where inaccuracies in early stages (e.g., concept recognition or negation detection) cascade and adversely affect downstream modules, particularly for context-sensitive geriatric concepts. Second, cross-concept dependency management becomes difficult, as many geriatric constructs (e.g., identify impaired mentions [not able to perform ADLs] would treat negation as positive) require integration of information across multiple mentions and contextual cues, which is not easily coordinated across isolated pipeline components. Third, the scalability of rule management presents a challenge; as the number of concepts increases (e.g., >40 in our study), maintaining and synchronizing concept-specific rules across multiple pipeline stages becomes increasingly complex and brittle. Finally, linear pipelines are typically limited to local decision-making at each stage, making it difficult to incorporate document-level or cross-sentence context that is often essential for comprehensive geriatric assessment. Future architectures may benefit from adaptive modularization that retains the interpretability of rule components while allowing dynamic communication between modules. For example, a domain focused context classifier, such as one specialized in ADL concepts, could adjust negation as an inclusion concept rather than being removed. This adaptive modular design enhances knowledge management by allowing individual rule sets and contextual models to be customized, versioned, and refined independently. It supports reproducibility across sites and studies, since updates can be systematically documented and shared without disrupting the broader processing workflow. Future designs should also leverage LLMs as an intelligence layer or dynamic knowledge base. Integrating LLMs with rule-based systems can provide contextual reasoning support and assist the knowledge engineering process by suggesting candidate rules, detecting inconsistencies, and refining ontological definitions. A hybrid and interoperable architecture that combines structured rule representations with the generative reasoning capacity of LLMs would ensure a balanced system that maintains reproducibility, transparency, and adaptability across both generic NLP processing and domain-specific customization. From static models to feedback-driven learning LLMs have demonstrated remarkable capacity for reasoning, contextual understanding, and generalization across diverse biomedical tasks39. However, their generative nature also introduces persistent error modes that are difficult to correct once deployed. Our error analysis showed that the predominant errors made by generative models include Conflation, Omission, and Complete Hallucination. These errors all arise from next token prediction, a mechanism that lacks explicit grounding in factual or contextual constraints40. Once generated, such errors are difficult to detect and nearly impossible to fix without retraining or extensive manual correction on top of the model. Adjusting prompts or employing reasoning strategies such as a chain of thought provides only partial mitigation. In the context of EHR, clinical care, documentation patterns, and patient context evolve continuously, and a model that cannot learn from new information or human feedback risks becoming obsolete or misleading. The next generation of clinical AI must therefore move beyond static deployment toward adaptive systems that can incorporate user (e.g., clinician or caregiver) input and real time feedback. When models learn from validation, clarification of uncertain outputs, or identification of context specific nuances, they transition from passive information extractors to active learning partners. Building such feedback driven intelligence requires both methodological and infrastructural innovation. Systems must be capable of securely capturing user feedback, updating model behavior under appropriate governance and privacy constraints, and prospectively validating the resulting changes within operational workflows. Embedding this adaptive capability within clinical NLP can also enhances transparency and trust. Achieving translational clinical NLP paradigm: the need for team-science collaboration, research reproducibility, and implementation science The translation of clinical NLP models from research to real-world healthcare applications requires more than algorithmic innovation. It depends on a sustained, team-based approach that aligns clinical relevance, methodological rigor, and implementation science. This translational paradigm must integrate diverse expertise across medicine, clinical informatics, computer science, biostatistics, human-computer interaction and implementation science to ensure that model outputs are meaningful, reproducible, and actionable within routine care: Collaborative task formulation A key step toward translational success lies in aligning clinical questions with AI task formulation. Many current NLP efforts solely focus on model performance without ensuring that the extracted information maps to clinically meaningful outcomes. Collaborative task design should therefore begin with a clear definition of clinical intent, such as identifying functional decline, delirium, or frailty status, and then translate these goals into specific computational targets and evaluation metrics17. This process requires iterative and continuous discussion and consensus development between domain experts and model developers. Infrastructure to standardize and empower collaboration Establishing a shared infrastructure for task formulation is essential to support scalability and reproducibility across institutions. Leveraging existing guidelines and definitions, such as common data elements, the 4Ms framework for Age-Friendly Health Systems, or domain-specific ontologies, can ensure that each NLP task is anchored in recognized clinical standards. In addition to the data infrastructure such as ENACT or OHDSI, there is also a need for community-driven platforms that support shared annotation resources, ontology versioning, and provenance tracking and that can further enable transparent knowledge management. These platforms can create a foundation for collective progress that extends beyond isolated research silos. Bridging team and implementation science Translational progress in clinical NLP for geriatrics care and aging research requires sustained team science collaboration that brings together diverse expertise across institutions and organizational boundaries. Multi-institutional partnerships enable model evaluation using heterogeneous datasets and provide a more accurate representation of real-world clinical documentation. A key next step of our study is to implement the MedCGA framework at collaborating sites, allowing local customization and optimization based on specific use cases and operational contexts. Effective deployment of clinical NLP systems also requires a deep understanding of the human and organizational factors that shape implementation, including workflow integration, stakeholder trust, and feedback mechanisms41. Implementation science frameworks can guide systematic assessment of readiness, barriers, and facilitators to adoption, ensuring that innovations transition efficiently from research settings into clinical environments41. Additionally, we need to ensure process reproducibility, such as the RITE-FAIR principle42, which emphasizes not only model parameters but also data provenance, ETL pipelines, annotation process, and evaluation. We therefore hope to leverage the convergence of team science, implementation science, and research reproducibility to accelerate the development of scalable, reliable, and human-centered NLP systems capable of transforming geriatrics care delivery and aging research across health systems. Our study has several limitations. First, our sampling strategy enriched for longer, concept-dense notes, improving coverage of rare geriatric concepts but potentially limiting external validity. As a result, the dataset may not fully reflect the distribution of concepts in routine clinical notes, and prevalence estimates derived from this dataset should be interpreted with caution. Second, despite multi-site refinement, the annotation schema may still reflect institutional biases and may not fully generalize to unseen healthcare settings or documentation styles. The annotation process is also subject to human error and residual inter-annotator variability, which may affect consistency despite adjudication efforts. Third, our sentence-level, single-label formulation introduces a structural limitation, as clinical text often contains multiple overlapping concepts and cross-sentence dependencies. This design choice may impose a performance ceiling by restricting the model’s ability to capture multi-label and contextual relationships. Future work should evaluate performance on randomly sampled clinical notes to better reflect real-world documentation distributions as well as explore multi-label and document-level modeling frameworks to better reflect the complexity of clinical narratives. Despite these limitations, our study presents a comprehensive effort to advance geriatric informatics through the development of a scalable and trustworthy NLP framework for extracting 41 comprehensive geriatric assessment data elements from unstructured electronic health records. We believe our work represents a step toward realizing an age-friendly learning health ecosystem and contributes to the broader mission of geriatric and aging research. Method Our study was approved by the Institutional Review Boards of The University of Texas Health Science Center at Houston (UTHealth), Memorial Hermann Health System, and Harris County Psychiatric Center. The overview of the gold standard corpus and model development process is summarized in Fig. 5. Overview of MedCGA models’ development and validation process. Data source The data set was developed based on four distinct data sources from four EHR systems including: UTPhysicians (UTP, training and test), the clinical practice group of UTHealth Houston, providing outpatient and specialty care across multiple clinics and serving as a major source of ambulatory EHR data; Memorial Hermann Health System (MHHS, training and test), a large, non-profit healthcare system based in Southeast Texas and a key provider of inpatient EHR data; Harris County Psychiatric Center (HCPC, test only), The UTHealth Houston Harris County Psychiatric Center, the largest academic psychiatric hospital in the U.S., specializing in acute inpatient mental health services and psychiatric EHR data; and Beth Israel Deaconess Medical Center’s EHR contributing to the MIMIC database (test only), focused on ICU and hospital data (Table 1). Table 1Overview of study setting, EHR source, and study populationInstitutionDescriptionPopulationUTPOutpatient clinic EHR used within UT Physicians/UTHealth McGovern outpatient clinics.Older adults seen in the osteoporosis clinicMHHSMemorial Hermann Health System EHR used across a large regional network of hospitals and clinics in Greater Houston.Inpatient older adult cohortHCPCSafety-net hospital with an integrated EHR supporting inpatient psychiatric and psychological services.Inpatient psychiatric older adult patientsBIDMCBeth Israel Deaconess Medical Center’s EHR contributing to the MIMIC database, focused on ICU and hospital data.MIMIC ICU cohort, age ≥ 65UTP UTPhysicians, MHHS Memorial Hermann Health System, HCPC Harris County Psychiatric Center, and BIDMC Beth Israel Deaconess Medical Center. Overview of study setting, EHR source, and study population UTP UTPhysicians, MHHS Memorial Hermann Health System, HCPC Harris County Psychiatric Center, and BIDMC Beth Israel Deaconess Medical Center. The Qwen model’s training set was further enhanced based on a large-scale silver-standard corpus developed using distant supervision and human review. We applied distant supervision to EHR data from 2012 to 2022 using the OHNLP Backbone rule-based dictionary lookup approach to identify positive mentions of cognitive, functional, and social descriptors. The dictionary set was derived from previously validated rule-based models9,11,12,15,16. This process yielded 52,253 clinical text snippets. Annotators labeled positive examples corresponding to CGA-related concepts, while negative examples were generated by randomly sampling unannotated sentences at a 3:1 ratio and labeling them as “None” for instruction tuning. This approach may introduce noise, as some sampled negatives may contain unlabeled or ambiguous concepts. We ensured no patient overlap between training and test datasets by restricting distant supervision to data prior to 2022, while the test set consisted of prospectively collected patients from the UT Physicians osteoporosis clinic. The test dataset was constructed from four distinct EHR sources: UT Physicians (UTP), Memorial Hermann Health System (MHHS), Harris County Psychiatric Center (HCPC), and Beth Israel Deaconess Medical Center (via the MIMIC database) (Table 1). For each site, we randomly sampled 50 patients aged >= 65 from the corresponding cohorts. To improve annotation efficiency and ensure sufficient coverage of geriatric concepts, we selected the 100 largest notes per site based on file size, resulting in a total of 400 notes. This strategy enriches for concept-dense documents but may distort prevalence estimates and limit generalizability to routine clinical note distributions. Therefore, results should be interpreted as reflecting performance in concept-rich evaluation settings rather than naturalistic clinical distributions. Task formulation The task formulation was informed through consultation with geriatricians, the Comprehensive Geriatric Assessment-based Frailty Index (CGA-FI), which is a deficit accumulation measure derived from routinely collected elements of a comprehensive geriatric assessment, the Guide to Using the 4Ms in the Care of Older Adults developed by the Institute for Healthcare Improvement (IHI) in collaboration with The John A. Hartford Foundation, the American Hospital Association (AHA), and the Catholic Health Association (CHA), which provides a structured framework for delivering age-friendly care to older adults, as well as other literature related to geriatric care. As a result, we focus on four major task categories with a total of 41 sub-tasks to comprehensively capture multidimensional aspects of geriatric health. These categories include: 1) Goals of Care and Advanced Care Planning (4 sub-tasks), 2) Socioeconomic Status (5 sub-tasks), 3) Cognitive and Behavioral Status 14 sub-tasks), and 4) Mobility and Functional Status (18 sub-tasks) (Table 2). Detailed definition and annotation examples of each task can be found in Supplementary Information 1. Table 2Task formulation for comprehensive geriatric assessment domainsTaskSub-TasksGoals of Care and Advanced Care PlanningGoals of Care; Limitations of Life-Sustaining Treatment; Hospice; Palliative CareSocioeconomic StatusFinancial Strain; Food Insecurity; Housing Insecurity; Social and Family Support; Transportation BarriersCognitive and Behavioral StatusDelirium; Agitation; Altered Mental Status; Confusion; Disconnected; Disorganized Thinking; Disorientation; Encephalopathy; Fluctuation in Mental Status; Hallucinations; Inattention; Reorientation; Cognitive Impairment; DepressionMobility and Functional StatusPhysical Activity; Falls; Basic Activities of Daily Living; Bathing; Dressing; Transferring; Continence; Toileting; Feeding; Telephone Use; Transportation Use; Shopping; Meal Preparation; Housekeeping; Laundry; Handyman Work; Medication Management; Financial Management Task formulation for comprehensive geriatric assessment domains Gold standard development We employed an iterative, multi-stage annotation schema development process grounded in the TRUST framework24,43. Specifically, initial schema development was informed by established CGA and AFHS 4Ms definitions, along with domain expert input to define core concepts and domains. We also adopted existing well-established guideline from previous community efforts. For example, the NLP-CAM guideline was developed based on the Confusion Assessment Method44 and validated through formal corpus annotation of 300 patients ( > 3,000 clinical notes) from Mayo Clinic9,16. The functional status guideline was developed based on the International Classification of Functioning, Disability and Health (ICF) standard and validated by the Mayo Clinic Study of Aging (MCSA) cohort8. 112. During annotation, we conducted iterative refinement cycles in which annotators reviewed batches of clinical notes across multiple sites, identifying previously unrecognized or institution-specific expressions of target concepts (e.g., variations in documenting activities of daily living)3. These newly identified expressions were incorporated into the annotation guidelines through consensus discussions and adjudication, enabling progressive expansion of concept coverage. This combination of expert-driven curation, cross-site variability exposure, and iterative schema refinement allowed us to better capture the breadth of real-world clinical language. The annotation team consisted of a geriatrician (KZ), a trained senior clinical data abstractor (HL), and two medical students (GG, AC). The process was overseen by a senior geriatrician (MK) and a clinical informatician (SF). All annotators had prior experience in at least one annotation study in related research. The initial training activities included walk through annotation guideline, perform test annotation and demonstrating the annotation tool, and data sharing practice. Two rounds of consensus development were conducted. Phase 1 focused on the systematic annotation of 25 randomly sampled clinical documents, which focused on measuring both false positive and false negative cases (a positive case means a patient has impairment in that geriatric care concept), whereas Phase 2 focused on a stratified sample that included at least three mentions of each sub-task (120 total) to ensure rare mentions were reviewed and measured. At the end of each phase, inter-annotator agreement (IAA) was measured using the F1-score, followed by post-annotation discussion and adjudication. Since more than two annotators were involved and each mention was annotated by four reviewers, the F1-score was calculated for each rater against the majority vote. The average IAA for Phase 1 and Phase 2 were 0.845 and 0.919, respectively. Disagreements were resolved through consensus adjudication led by a senior geriatrician (MK) and a clinical informatician (SF), who reviewed discrepant annotations and made final determinations. Annotators were not explicitly blinded to site due to the inherent context within clinical notes. Detailed per-domain and per-concept agreement statistics are provided in the Supplementary Materials Supplementary Information 5 – Table 1. Model development GPT-4o leverages in-context learning to classify clinical sentences into categories derived from the task. The development of GPT-4o involves an iterative process and in consultation with domain experts in geriatrics and clinical NLP. Based on the annotation guidelines developed from the prior step, we developed a structured in-context prompt that incorporated best practices in few-shot learning from prior studies39,45. The final prompt included the following major components: task instruction, output constraint, category definition, 3-shot examples, fallback logic, and formatting rules (Supplementary Information 2). We evaluated multiple large language models, including GPT-4o, GPT-4.1, O1, and O4-mini, using a pilot corpus of 2,000 sentences sampled across all geriatric care domains. Pilot evaluation was conducted by domain experts to examine extraction accuracy. We also conducted cost estimation analysis based on Azure pricing. The model (GPT-4o) demonstrating the best balance of performance and computational efficiency was selected for production. The finalized model was deployed on an independent held-out test dataset. Qwen2-7B-Instruct is a supervised instruction-tuning on a lightweight open-source LLM (Qwen2-7B-Instruct) to classify clinical sentences into 41 pre-defined geriatric care categories. The training data consisted of 52,253 clinical text snippets extracted from EHR notes. Human annotators labeled positive examples corresponding to CGA and 4Ms-related concepts. Negative examples were generated by randomly sampling unannotated sentences from the same dataset at a 3:1 ratio and labeling them as ‘None’ for inclusion in the instruction-tuning process. This strategy may introduce noise, as some sampled negative instances may contain unlabeled or ambiguous concepts. A stratified 80/20 train-validation split was implemented to preserve the original proportion of categories across training and validation sets. Low-Rank Adaptation (LoRA) was used for parameter-efficient fine-tuning of the model, enabling targeted weight updates with reduced computational overhead46. Model training was conducted using the Hugging Face Supervised Fine-Tuning (SFT) Trainer with a learning rate of 1e-4, 17 gradient accumulation steps, and 20 epochs. MedAgingIE is a hybrid heuristic–LLM model that integrates a rule-based expert system with ontology-guided, LLM-assisted knowledge engineering. The core extraction47 process38 is governed by explicit, human-interpretable rules and structured knowledge representations, with the LLM serving as an intelligence layer1 for rule prototype, error analysis and refinement. MedAgingIE was developed using our previously developed LLM4Rules framework (Supplementary Information 4). MedAgingIE was initially prototyped based on prior aging NLP models and the MedTaggerIE framework2,3. LLM-assisted synonym expansion was performed to enrich the syntactic and semantic coverage of the initial model. During training, the prototype model was applied to the training data, and errors were systematically identified. GPT-4o then analyzes these error sets by assigning error classes and generating explanations for likely causes. Based on this analysis, the GPT-4o proposes candidate regular expression rules to address the observed failures. These rules are subsequently incorporated into MedAgingIE and evaluated on the previous training set to assess improvements in concept extraction performance. The process iteration continues until optimal performance is reached. LLM use is limited to two roles1: rule generation during development and2 contextual validation of candidate extractions. Final decisions are made by the deterministic rule engine rather than generative inference. All experiments were conducted within a HIPAA-compliant Microsoft Azure OpenAI environment to ensure secure processing of clinical text. The source code for the framework is publicly available at: https://github.com/OHNLP/MedCGA. Model evaluation We computed micro-averaged and macro-averaged precision, recall, and F1 scores to capture both global and concept-level performance. The micro-average aggregates true positives, false positives, and false negatives across all instances, reflecting overall performance weighted proportionally by concept frequency. The macro-average, by contrast, computes precision, recall, and F1 independently for each clinical concept and then averages across concepts, treating each concept with equal weight regardless of its prevalence in the data. To enable interpretable comparison, results were further summarized by higher-order geriatric categories representing goals of care, cognitive and behavioral status, mobility and functional status, and social and environmental status. Per-concept F1 scores were also reported to characterize fine-grained variability and identify specific concepts where the model demonstrated relative strength or encountered persistent difficulty. Formally, let C={1,…,K} denote the set of clinical concepts. For each concept k∈C, let TPk, FPk and FNk denote the number of true positives, false positives, and false negatives, respectively, aggregated across all evaluation instances. The concept-level precision, recall, and F1 score are defined as: All three model variants were evaluated using identical datasets and annotation guidelines to ensure fair comparison. To enable inferential comparison between models, we conducted paired bootstrap resampling at the sentence level (1,000 iterations) within each site. For each bootstrap sample, we computed differences in macro- and micro-F1 scores between model pairs. Ninety-five percent confidence intervals (CIs) were obtained from the empirical bootstrap distribution. We additionally computed two-sided p-values based on the proportion of bootstrap samples with differences crossing zero, and applied Bonferroni correction to account for multiple pairwise comparisons across models and sites. Statistical significance was determined based on whether the 95% CI excluded zero. We also conducted a systematic error analysis using a standard taxonomy and process30,31 (Supplementary Information 3). This process was conducted by two informaticians who iteratively reviewed and categorized false positive and false negative cases into a standardized set of error classes, comprising four major categories. Descriptive statistics and a Sankey diagram were created to visualize the relationships between error classes and geriatric care concepts, helping to identify model limitations and inform future improvement strategies. Ethics Statement Our study is approved by UT Physicians, Memorial Hermann Health System and Harris County Psychiatric Center IRB and ethics committees (IRB#: HSC-MS-25-1090, HSC-SBMI-23-0765, HSC-SBMI-24-0403, HSC-SBMI-17-0354). Supplementary information Supplementary Information 1. Supplementary Information 2. Supplementary Information 3. Supplementary Information 4. Supplementary Information 5. Acknowledgements This study was supported by the National Institute on Aging of the National Institutes of Health (grant numbers K01AG088452, K24AG073527, P30AG024832, R61AG068926, R01AG072799, U24AG098157 and R33AG071744) and the Cancer Prevention and Research Institute of Texas (RR230020). The funders had no role in study design, data collection, analysis, interpretation,or manuscript preparation. The authors also acknowledge the Center for TEAM-AI for computing infrastructure and data service support. Author contributions S.F., M.J.K., HF.L. conceptualized the study. S.F. led the overall methodological design, coordinated the team science workflow, and wrote the main manuscript. M.J.K., D.H.K., N.R., C.M.P, S.S. and E.H. provided geriatric and clinical domain expertise and guided the definition of CGA and 4Ms data elements. A.W. and J.A. contributed data and computing infrastructure support. J.A., Q.L., Z.Y., N.W., and S.F. contributed to data processing, statistical analysis, model development, implementation, and experimental evaluation. Huipeng.L., G.G., A.C., K.M.T., and M.J.K. participated in clinical note annotation, adjudication, and gold-standard dataset development. J.J., X.R., M.H., L.W., Y.W., L.S., and H.X. contributed to study design refinement, methodological consultation, and cross-site coordination. J.S.S. and J.S.W. provided guidance on research design and interpretation of cognitive and functional outcomes. HF.L. and D.H.K provided senior mentorship and oversight of the research methodology. All authors reviewed, edited, and approved the final manuscript. Data availability No datasets were generated or analysed during the current study. Competing interests HF.L. is an Editorial Board Member of npj Digital Medicine and was not involved in the review of, or decisions related to, this manuscript. D.H.K. received personal fees from Pfizer, Alosa Health (ended 12/31/2022), and VillageMD (ended 12/13/2022) for work unrelated to this study. All other authors declare no competing financial or non-financial interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Supplementary information The online version contains supplementary material available at https://doi.org/10.1038/s44401-026-00114-y. References Associated Data Supplementary Materials Supplementary Information 1. Supplementary Information 2. Supplementary Information 3. Supplementary Information 4. Supplementary Information 5. Data Availability Statement No datasets were generated or analysed during the current study.