From Generative Modeling to Clinical Classification: A GPT-Based Architecture for EHR Notes

Fariba Afrin Irany · 2026

Read From Generative Modeling to Clinical Classification: A GPT-Based Architecture for EHR Notes by Fariba Afrin Irany (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21955.txt.

Introduction

Fariba Afrin Irany explores whether generative language models can classify hospital patients from their records. Electronic health records hold vast amounts of unstructured clinical narrative. Those notes could support disease characterization, finding patient cohorts, and clinical decision support, if machines could read them well.

Long, specialized clinical text is hard to model, labeled examples are limited, and disease classes are heavily imbalanced. Irany's 2026 preprint presents a GPT-based architecture that moves from generative modeling to clinical classification of these notes. Success would help hospitals surface at-risk patients. It would help them build research cohorts. No armies of manual chart reviewers would be needed. Still, the stakes demand careful checks before any clinical use.

Catalog record

CREATORFariba Afrin Irany
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21955

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Generative Modeling to Clinical Classification: A GPT-Based Architecture for EHR Notes Fariba Afrin Irany Abstract The increasing availability of unstructured clinical narratives in electronic health records (EHRs) has created new opportunities for automated disease characterization, cohort identification, and clinical decision support. However, effectively modeling long, domain-specific clinical text remains challenging due to limited labeled data, substantial class imbalance, and the high computational cost associated with adapting large pretrained language models to downstream tasks. In this study, a GPT-based architecture for clinical text classification is presented, which adapts a pretrained decoder-only Transformer to downstream clinical tasks using a selective fine-tuning strategy. Instead of updating all model parameters, the majority of the pretrained GPT-2 backbone is frozen, and training is restricted to the final Transformer block, the final layer normalization, and a lightweight classification head. This design significantly reduces the number of trainable parameters while preserving the representational capacity required for modeling complex clinical language. This study evaluates the proposed approach on radiology reports from the MIMIC-IV-Note dataset using uncertainty-aware CheXpert-style labels derived directly from report text. Experiments are conducted across multiple problem formulations, including multi-label classification of radiographic findings, binary per-label classification under different uncertainty assumptions, and aggregate disease outcome prediction. Across varying sample sizes, the model demonstrates stable convergence behavior and strong classification performance, particularly in multi-class settings where non-mention and negated findings are prevalent. Overall, the results suggest that selective fine-tuning of pretrained generative language models offers an efficient and effective pathway for clinical text classification, enabling scalable adaptation to real-world EHR data while substantially reducing computational complexity. Keywords: clinical natural language processing; large language models; GPT; radiology report classification; uncertainty-aware labeling; MIMIC-IV Introduction The widespread adoption of electronic health record (EHR) systems has led to an unprecedented growth in the volume of unstructured clinical text. Clinical narratives such as progress notes, discharge summaries, operative reports, and radiology interpretations capture detailed information about patient history, physician reasoning, and care trajectories that is often unavailable in structured fields. Extracting meaningful information from these free-text documents is essential for a wide range of downstream applications, including disease phenotyping, cohort discovery, outcome prediction, quality assessment, and clinical decision support. However, the unstructured and highly specialized nature of clinical language poses substantial challenges for automated analysis. Traditional natural language processing approaches in the clinical domain have relied heavily on rule-based systems, domain-specific lexicons, or manually engineered features. While these methods can be effective for narrowly defined tasks, they are often brittle, difficult to scale, and unable to capture long-range dependencies or nuanced contextual meaning. Statistical machine learning methods improved flexibility but still required substantial feature engineering and struggled with the variability and complexity of clinical narratives. These limitations have motivated the adoption of representation learning approaches that can automatically learn contextual features directly from raw text. Transformer-based language models have emerged as a dominant paradigm for representation learning in natural language processing. By relying on self-attention mechanisms, Transformers are able to model long-range dependencies and contextual interactions between tokens more effectively than recurrent or convolutional architectures. Large language models pretrained on massive text corpora using self-supervised objectives have demonstrated remarkable generalization capabilities, enabling transfer learning across a wide range of downstream tasks. Generative pretrained models, such as GPT-2, learn rich contextual representations by predicting tokens autoregressively, allowing them to capture syntactic structure, semantic relationships, and discourse-level patterns in text. Despite these advances, directly applying large pretrained language models to clinical text introduces several practical and methodological challenges. Clinical datasets are typically orders of magnitude smaller than the corpora used for pretraining, increasing the risk of overfitting during downstream training. Moreover, the computational cost of fine-tuning all parameters of a large language model can be prohibitive, particularly in academic or clinical environments with limited computational resources. Full fine-tuning also increases memory usage during training due to the need to store gradients and optimizer states for all parameters, further constraining scalability. An additional challenge arises from the domain mismatch between general-language pretraining data and clinical text. While pretrained models capture broad linguistic knowledge, clinical narratives exhibit unique characteristics, including specialized terminology, abbreviations, telegraphic writing styles, and institution-specific conventions. Effective adaptation therefore requires balancing domain-specific learning with preservation of general-language representations. Updating too many parameters risks catastrophic forgetting of pretrained knowledge, while updating too few parameters may limit the model’s ability to adapt. To address these challenges, this work explores a selective fine-tuning strategy for adapting GPT-2 to unstructured clinical text classification tasks. Instead of updating the entire model, the majority of pretrained parameters are frozen, and optimization is restricted to a carefully chosen subset of layers. Specifically, only the final Transformer block, the final layer normalization, and a lightweight task-specific classification head are updated during fine-tuning. This design leverages the hierarchical nature of Transformer representations, where lower layers tend to encode general linguistic features and higher layers capture more abstract, task-relevant information. A key contribution of this work is a detailed and transparent analysis of model parameterization and computational complexity under this selective fine-tuning regime. While selective fine-tuning is often motivated intuitively, its concrete impact on parameter count and training-time complexity is rarely made explicit. In this paper, the number of frozen and trainable parameters in the GPT-2 (small) architecture is quantified, demonstrating that more than ninety-four percent of the model parameters remain fixed during training. A further analysis examines how restricting backpropagation to a single Transformer block substantially reduces training-time complexity and memory requirements while still enabling effective task adaptation. This work is intentionally written to be accessible to readers who may be new to large language models and Transformer architectures. Rather than assuming familiarity with internal model mechanics, step-by-step explanations are provided to clarify how parameters are distributed across model components, why specific terms arise in the complexity analysis, and how architectural design choices influence computational efficiency. By making these details explicit, selective fine-tuning strategies are demystified, and practical guidance is provided for researchers seeking to apply large language models in data-limited or resource-constrained settings. Overall, this study demonstrates that efficient adaptation of generative pretrained language models to clinical text is possible without full-model fine-tuning. By combining selective parameter optimization with a clear understanding of architectural complexity, the proposed approach offers a practical and computationally efficient pathway for deploying large language models in clinical natural language processing applications. This work makes the following contributions: • A parameter-efficient selective fine-tuning strategy for GPT-based clinical text classification that updates only a small subset of model parameters. • A systematic analysis of training dynamics and computational trade-offs associated with selective fine-tuning. • An empirical evaluation on large-scale MIMIC-IV radiology reports demonstrating competitive performance under constrained computational resources. Literature Review This literature review situates the proposed selective fine-tuning framework within four intersecting research areas: (i) Transformer-based language models and large-scale pretraining, (ii) transfer learning and domain adaptation for NLP, (iii) parameter-efficient fine-tuning and computationally efficient adaptation, and (iv) clinical NLP and modeling unstructured EHR narratives. Transformer Architectures and Large-Scale Language Model Pretraining The Transformer architecture introduced self-attention as the core mechanism for sequence modeling, enabling efficient parallel computation and strong performance on machine translation and beyond 57 . Subsequent work clarified optimization and architectural details relevant to deep Transformer training, including the role of normalization and

Read the complete text