Structured Semantics from Unstructured Notes: Language Model Approaches to EHR-Based Decision Support
Wu Hao Ran, Xi Xi, Furong Li, Jingyi Lu, Jian Jiang, Hui Huang, Yuzhuan Zhang, Shi Li · 2025
Read Structured Semantics from Unstructured Notes: Language Model Approaches to EHR-Based Decision Support by Wu Hao Ran, Xi Xi, Furong Li, Jingyi Lu, Jian Jiang, Hui Huang, Yuzhuan Zhang, Shi Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06340.txt.
Introduction
Wu Hao Ran and co-authors survey in this 2025 paper how large language models can mine hospital records for decision support. Electronic health records mix free-text notes with lab values and codes, and the richest clues often hide in the notes that classic analysis skips. Language models can turn that prose into structured meaning.
The authors discuss how text features enrich patient pictures and help align data across hospitals with differing habits. They also weigh the harder parts: folding in medical codes cleanly, and keeping models fair and general when moved between sites. The paper maps both the promise and the homework for clinical AI. Hospitals could pilot the tools on old charts first.
Catalog record
| CREATOR | Wu Hao Ran, Xi Xi, Furong Li, Jingyi Lu, Jian Jiang, Hui Huang, Yuzhuan Zhang, Shi Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-06340
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Structured Semantics from Unstructured Notes: Language Model Approaches to EHR-Based Decision Support Wu Hao Ran, Xi Xi, Furong Li, Jingyi Lu, Jian Jiang, Hui Huang, Yuzhuan Zhang, Shi Li Abstract The advent of large language models (LLMs) has opened new avenues for analyzing complex, unstructured data, particularly within the medical domain. Electronic Health Records (EHRs) contain a wealth of heterogeneous information, including free-text clinical notes, structured lab results, time-series vitals, and diagnostic codes—each capturing different facets of patient care. Traditional approaches to EHR analysis have largely relied on structured features and tabular representations, often neglecting the nuanced and semantically rich content embedded in clinical narratives. In this work, we investigate the integration of advanced language models, particularly domain-specific variants like ClinicalBERT and Clinical ModernBERT, to unlock the latent structure of clinical text and enhance downstream clinical decision support. We demonstrate that language models can extract meaningful representations from unstructured notes that not only improve predictive performance, but also generalize more effectively across institutional boundaries. We further examine the role of medical codes—such as ICD and CPT—in complementing textual features and enabling models to reason over structured ontologies. Our approach combines both free-text and codified knowledge to create hybrid representations that are robust to variation in documentation practices. In addition, we explore recent advancements in parameter-efficient fine-tuning (PEFT) and domain-adaptive pretraining to scale these models within resource-constrained clinical environments. Finally, we address the critical challenges of model evaluation, generalizability, and fairness, advocating for robust benchmarking practices that include confidence intervals, stratified subgroup analysis, and cross-institutional testing. This work highlights the importance of semantically informed modeling in healthcare and outlines future directions toward multimodal, interpretable, and equitable AI systems for clinical decision support. † † affiliationnotice: Equal contribution 1 Introduction Healthcare is a data-rich environment, with Electronic Health Records (EHRs) serving as central repositories for patient information. These records encompass a wide array of data types, from numerical measurements like vital signs and laboratory results to more qualitative data such as physician’s notes, discharge summaries, and radiology reports ( Evans 2016 ) . The sheer volume and heterogeneity of EHR data present both immense opportunities and significant challenges for computational analysis ( Goldstein et al. 2017 ) . Traditionally, machine learning applications in healthcare have often focused on structured, high-dimensional EHR data, such as billing codes or demographic information. However, the “curse of dimensionality” can limit the effectiveness of these approaches ( Lee et al. 2024a ) . Recent advancements in natural language processing (NLP), particularly with the rise of Large Language Models (LLMs), have revolutionized the ability to process and understand human language. This development holds profound implications for healthcare, as a substantial portion of critical clinical information resides within unstructured text ( Rajpurkar et al. 2022 ) . For instance, clinical notes contain nuanced details about a patient’s condition, treatment plans, and responses that are often not captured in structured fields. By effectively extracting and interpreting these textual features, LLMs can unlock new insights for various applications, including disease diagnosis, risk prediction, and treatment optimization ( Zhou et al. 2024 ) . This paper aims to provide a pedagogical overview of how language models can be effectively integrated into clinical decision support systems. We will focus on the theoretical underpinnings of these models, their practical applications in processing EHR data, and the critical considerations for their responsible deployment in healthcare. The goal is to illustrate how text-based features can complement or even surpass traditional EHR features in certain contexts, offering a more holistic understanding of patient health. 2 Related Works The field of AI in healthcare has witnessed transformative growth over the past decade, driven by an increasing availability of digitized Electronic Health Records (EHRs) and advances in machine learning. A wide range of methodologies have been proposed to extract value from EHR data ( Rajpurkar et al. 2022 ) . Early approaches focused on traditional machine learning algorithms, such as logistic regression, random forests, and gradient-boosted trees ( Shwartz-Ziv & Armon 2022 ) , which proved effective for structured, tabular data. These models, especially XGBoost ( Chen & Guestrin 2016 ) , offered strong baseline performance with relatively low computational overhead. However, their capacity to capture longitudinal and unstructured information—such as temporal trends or free-text narratives—remains limited. The introduction of deep learning architectures, particularly Recurrent Neural Networks (RNNs) and their gated variants, marked a significant inflection point in EHR modeling ( Choi et al. 2016 ) . By explicitly modeling the temporal evolution of patient data, RNNs enabled more accurate predictions of adverse events and disease progression. Models like RETAIN introduced attention mechanisms into recurrent structures, offering partial interpretability while preserving sequential dependencies ( Choi et al. 2016 ) . Beyond sequence modeling, methods like MiME ( Choi et al. 2018 ) embraced the hierarchical structure of clinical codes and visits, learning embeddings that reflect both temporal and taxonomic relationships. The rise of transformer-based architectures has dramatically reshaped the landscape. The Transformer model ( Vaswani 2017 ) introduced a self-attention mechanism that allows for global context modeling, independent of sequence length. This innovation catalyzed the development of pre-trained language models such as BERT ( Devlin 2018 ) , which revolutionized NLP by introducing bidirectional contextual embeddings. Domain-adapted variants—ClinicalBERT ( Alsentzer et al. 2019 ) , MedBERT ( Vasantharajan et al. 2022 ) , and SciBERT ( Beltagy et al. 2019 ) —have demonstrated superior performance on clinical and biomedical NLP tasks, benefitting from continued pretraining on corpora like MIMIC-III, PubMed, and PMC articles. Importantly, these models have shifted the paradigm from feature engineering to representation learning. Text-derived embeddings capture latent semantics that often elude structured features, enabling more nuanced phenotyping and risk stratification. Several studies, such as ( Lee et al. 2024a ) , have shown that these text-based representations not only outperform traditional EHR encodings on many tasks, but also provide better generalization across institutions—a critical bottleneck in real-world clinical AI deployment. However, a recurring limitation of general-purpose LLMs is their inability to natively interpret codified clinical knowledge, such as ICD or CPT codes, which are pervasive in EHR systems ( Lee & Lindsey 2024 ) . Addressing this gap, recent models have proposed hybrid approaches that incorporate medical ontologies during pretraining. DK-BEHR ( An et al. 2025 ) embeds both medical codes and their textual definitions, fostering alignment between symbolic and natural language representations. Clinical ModernBERT ( Lee et al. 2025 ) further extends this paradigm by incorporating structured medical concepts into long-context encoders, enabling accurate modeling of multi-visit, heterogeneous clinical records. Evaluation frameworks have concurrently matured to match the sophistication of models. Rather than reporting single-point metrics, recent studies emphasize bootstrapped confidence intervals, stratified subgroup analysis, and robustness across temporal slices ( Lee et al. 2024b ) , ( Goetz et al. 2024 ) . These practices are essential to ensure clinical validity and avoid overfitting to institutional idiosyncrasies. Additionally, interest in generative modeling for synthetic EHR data is growing ( Lin et al. 2025 ) , offering a promising avenue for privacy-preserving training and benchmarking, albeit with caveats regarding fidelity and realism. The frontier of clinical AI is increasingly multimodal. Models like Gemini ( Team et al. 2023 ) integrate text, imaging, and structured data