Selective Fine-Tuning of GPT Architectures for Parameter-Efficient Clinical Text Classification

1 st Fariba Afrin Irany, 2 nd Sampson Akwafuo · 2026

Read Selective Fine-Tuning of GPT Architectures for Parameter-Efficient Clinical Text Classification by 1 st Fariba Afrin Irany, 2 nd Sampson Akwafuo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14183.txt.

Introduction

Irany and Akwafuo adapt GPT-2 to clinical text sorting with a light, selective fine-tuning scheme. Health records hold rich free-text notes, but labeled data are scarce and full tuning is costly. Their method freezes most weights and trains only the last block, the last norm layer, and a small sorting head. That cuts trainable parameters while keeping the model's learned context skills. Tests use radiology reports from MIMIC-IV-Note with labels in the CheXpert style.

Catalog record

CREATOR1 st Fariba Afrin Irany, 2 nd Sampson Akwafuo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23082
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.14183

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Selective Fine-Tuning of GPT Architectures for Parameter-Efficient Clinical Text Classification 1 st Fariba Afrin Irany, 2 nd Sampson Akwafuo Abstract The rapid expansion of electronic health record (EHR) systems has generated large volumes of unstructured clinical narratives that contain valuable information for disease identification, patient cohort discovery, and clinical decision support. Extracting structured knowledge from these free-text documents remains challenging because clinical language is highly specialized, labeled datasets are limited, and full fine-tuning of large pretrained language models can require substantial computational resources. Efficient adaptation strategies are therefore essential for practical clinical natural language processing applications. This study proposes a parameter-efficient selective fine-tuning framework for adapting GPT-2 to clinical text classification tasks. Instead of updating the entire pretrained model, the majority of network parameters are frozen, and only the final Transformer block, the final layer normalization module, and a lightweight classification head are updated during training. This design substantially reduces the number of trainable parameters while preserving the contextual representation capabilities learned during pretraining. The proposed approach is evaluated using radiology reports from the MIMIC-IV-Note dataset with automatically derived CheXpert-style labels. Experiments on 50,000 radiology reports demonstrate that selective fine-tuning achieves approximately 91% classification accuracy while updating fewer than 6% of the model parameters. Comparative experiments with head-only training and full-model fine-tuning show that the proposed method provides a favorable balance between predictive performance and computational efficiency. These results indicate that selective fine-tuning offers an efficient and scalable framework for clinical text classification. I Introduction The widespread adoption of electronic health record (EHR) systems has generated a large volume of unstructured clinical text, including progress notes, discharge summaries, operative reports, and radiology reports. These narratives contain rich descriptions of patient conditions, diagnostic reasoning, and treatment decisions that are often not captured in structured database fields. As a result, automated analysis of clinical text has become increasingly important for tasks such as disease phenotyping, cohort identification, outcome prediction, healthcare quality assessment, and clinical decision support. However, clinical narratives present several challenges for computational analysis. Clinical language is highly specialized and frequently contains abbreviations, incomplete sentences, and institution-specific terminology. Early clinical natural language processing (NLP) systems relied primarily on rule-based methods and manually engineered features. Although such approaches can perform well in limited scenarios, they are difficult to scale and often fail to capture the complex contextual relationships present in long clinical documents. Recent advances in deep learning have significantly improved NLP capabilities. In particular, Transformer-based architectures have become the dominant paradigm for modeling sequential text data. Large pretrained language models learn contextual representations from massive text corpora and can be adapted to downstream tasks through transfer learning. Generative Pretrained Transformer models such as GPT-2 capture syntactic and semantic relationships across long text sequences using autoregressive language modeling [ 4 ] . Despite these advances, applying large pretrained models to clinical text introduces practical challenges. Clinical datasets are typically much smaller than the datasets used for model pretraining, which increases the risk of overfitting during full-model fine-tuning. In addition, updating all parameters of large language models requires substantial computational resources and memory. These limitations can make full-model fine-tuning inefficient in many academic and healthcare research environments. To address these challenges, this study investigates a parameter-efficient selective fine-tuning strategy for adapting GPT-based models to clinical text classification. Instead of updating the entire model, most pretrained parameters are frozen while only the final Transformer block, the final layer normalization layer, and a lightweight classification head are optimized. This approach leverages the hierarchical representation structure of Transformer models while substantially reducing training complexity. The proposed framework is evaluated on radiology reports from the MIMIC-IV clinical database using automatically derived CheXpert-style labels. Experimental results demonstrate that selective fine-tuning achieves competitive classification performance while updating only a small fraction of model parameters, making it a practical approach for clinical NLP applications under limited computational resources. II Related Work Recent advances in natural language processing have been driven by the development of Transformer-based architectures and large-scale pretrained language models. The Transformer architecture introduced the self-attention mechanism, enabling efficient modeling of long-range dependencies and parallel sequence processing [ 2 ] . Subsequent research improved the training stability and optimization behavior of Transformer models through refinements in normalization strategies and architectural design [ 5 ] . These developments enabled the emergence of large pretrained language models trained using self-supervised objectives that learn contextual linguistic representations transferable to a wide range of downstream tasks. Early work on contextual representation learning demonstrated the effectiveness of pretrained embeddings for improving performance in natural language processing tasks. Models such as ELMo [ 10 ] and ULMFiT [ 11 ] showed that pretrained language representations significantly improve downstream performance when labeled datasets are limited. Later architectures expanded both the scale and capability of pretrained models. Generative pretrained Transformer models introduced autoregressive language modeling at scale [ 3 , 4 , 8 ] , while bidirectional models such as BERT demonstrated the effectiveness of masked language modeling for capturing contextual relationships across sequences [ 9 ] . Several extensions including RoBERTa, XLNet, ALBERT, and T5 further improved model scalability, training efficiency, and transfer learning performance [ 12 , 13 , 14 , 15 ] . These models established a general paradigm in which large pretrained models are adapted to downstream tasks through fine-tuning. Transfer learning has therefore become a central paradigm in modern NLP because pretrained models can be adapted to specialized tasks using relatively small labeled datasets. However, domain mismatch between general-language training corpora and specialized domains often requires additional adaptation strategies. Domain-adaptive pretraining and task-adaptive pretraining approaches have been shown to significantly improve model performance in domain-specific settings [ 7 ] . These strategies are particularly important in biomedical and clinical natural language processing, where specialized terminology and writing conventions differ substantially from general-language corpora. As a result, pretrained language models are increasingly applied to clinical text classification, information extraction, and healthcare prediction tasks using electronic health record data. In parallel with advances in language modeling, computational frameworks have been developed to analyze large-scale healthcare, epidemiological, and complex network datasets. Data-driven modeling techniques have been used to analyze disease transmission patterns, epidemiological trends, and health system data integration using large heterogeneous datasets [ 20 , 22 ] . Network-based analytical methods have also been applied to identify structural patterns and community relationships in multilayer data environments [ 21 ] . Optimization and simulation approaches have been explored for improving healthcare logistics, emergency response planning, and resource allocation in complex systems [ 23 ] . Mathematical modeling approaches have also been applied to estimate infectious disease transmission dynamics and public health risk using computational epidemiology frameworks [ 24 ] . These studies highlight the growing role of scalable computational modeling and data-driven analytics in modern healthcare research. Despite their strong performance, applying large language models in clinical settings is challenging due to limited labeled data and high computational costs for full fine-tuning. Parameter-efficient methods such as adapters and low-rank adaptation address this issue by updating only a small subset of parameters while keeping most pretrained weights fixed [ 17 , 16 ] . III Methodology Experiments use the GPT-2 (small) architecture, which contains 12 Transformer blocks, a hidden size of 768, and about 124 million

Read the complete text