Systematic Comparative Analysis of Large Pretrained Language Models on Contextualized Medication Event Extraction

Tariq Abdul-Quddoos, Xishuang Dong, and Lijun Qian · 2025

Read Systematic Comparative Analysis of Large Pretrained Language Models on Contextualized Medication Event Extraction by Tariq Abdul-Quddoos, Xishuang Dong, and Lijun Qian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-19224.txt.

Introduction

Tariq Abdul-Quddoos and two co-authors compare six large pre-trained language models on a clinical task: pulling medication events and their context out of hospital notes. The data come from a 2022 national clinical language challenge, built on annotated electronic health records. The models range from general BERT to versions trained on biomedical and clinical text, plus a long-document model. Each was fine-tuned for three jobs: spotting medications, detecting events, and classifying the context around them.

Models pre-trained on clinical data proved better at detecting medications and events. General-domain BERT Base, though, was the most effective at classifying event context, a split result that resists a simple more-domain-training story. The authors also detail how the long records were broken into pieces each model could accept. Scores use precision, recall, and F1, giving a practical guide for clinical text mining.

Catalog record

CREATORTariq Abdul-Quddoos, Xishuang Dong, and Lijun Qian
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-19224

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Systematic Comparative Analysis of Large Pretrained Language Models on Contextualized Medication Event Extraction Tariq Abdul-Quddoos, Xishuang Dong, and Lijun Qian Abstract Attention-based models have become the leading approach in modeling medical language for Natural Language Processing (NLP) in clinical notes. These models outperform traditional techniques by effectively capturing contextual representations of language. In this research a comparative analysis is done amongst pre-trained attention based models namely Bert Base, BioBert, two variations of Bio+Clinical Bert, RoBerta, and Clinical Longformer on task related to Electronic Health Record (EHR) information extraction. The tasks from Track 1 of Harvard Medical School’s 2022 National Clinical NLP Challenges (n2c2) are considered for this comparison, with the Contextualized Medication Event Dataset (CMED) given for these task. CMED is a dataset of unstructured EHR’s and annotated notes that contain task relevant information about the EHR’s. The goal of the challenge is to develop effective solutions for extracting contextual information related to patient medication events from EHR’s using data driven methods. Each pre-trained model is fine-tuned and applied on CMED to perform medication extraction, medical event detection, and multi-dimensional medication event context classification. Processing methods are also detailed for breaking down EHR’s for compatibility with the applied models. Performance analysis has been carried out using a script based on constructing medical terms from the evaluation portion of CMED with metrics including recall, precision, and F1-Score. The results demonstrate that models pre-trained on clinical data are more effective in detecting medication and medication events, but Bert Base, pre-trained on general domain data showed to be the most effective for classifying the context of events related to medications. I Introduction An Electronic Health Record (EHR) holds records of patient health information generated by encounters in any care delivery setting, including information on patient demographics, progress notes, problem lists, medications, vital signs, past medical history, immunizations, laboratory data, and radiology reports [ 1 ] . The application of machine learning on EHR’s serves to place the burden of extracting information on data driven models rather than people. The National Clinical NLP Challenges (N2C2), hosted by Harvard Medical School, focuses on the study of applying data driven approaches to mining clinical information holding a number of challenges since 2006. Track 1 of the 2022 N2C2 has task researchers with developing solutions for extracting information related to the context of medication mentions in EHR’s. For this challenge the Contextualized Medication Event Dataset(CMED) has been released and is a dataset capturing relevant context needed to understand medication changes in clinical narratives [ 2 ] , containing EHR’s and annotated notes with task relevant information. In the competition, a number of methods were used with large language transformer based architectures, with some popular choices being Bert, RoBerta, DeBerta, Longformer, and Text-To-Text Transfer Transformer. In combination with these models additional architecture add-on methods consisted of multiple linear layers with combined losses and knowledge graphs [ 3 ] . In this work several pre-trained attention based language models undergo comparative analysis with information extraction task on CMED. The specific applied models are Bert Base, BioBert [ 4 ] , two variations of Bio+Clinical Bert [ 5 ] , RoBerta Base [ 6 ] , and Clinical LongFormer [ 7 ] with pre-training from general, biomedical, and clinical domain corpora. The models are fine-tuned on CMED and used for the three task listed in section II . This study examines the performance of the applied models on the CMED dataset along with the differences in performance with regard to architecture differences, pre-training methods, and pre-training corpora. Pre- and post- processing methods are also of presented in this study given the wide range of syntactic structures used in EHR’s, it can be difficult to judge how best the process an entire record. The contributions of this work are as follow: 1) A comparative analysis of pre-trained large langauge attention models performance of EHR related task; 2) Fine-tuning pre-trained models on CMED training data for medication detection,medication event detection, and multi-dimensional medication event context classification; 3) Processing methods for breaking down EHR’s into format compatible with applied models. The rest of this paper is organized as follows. In Section II the three modeling task are detailed, in section III the applied modeling achitecture’s are detailed along with the differnces between applied pre-trained models, in section IV an overview of processing methods is given and results for experiments are analyzed, in section V an overview of related works is given, and in section VI this work is summarized and future work of interest is detailed. II 2022 National Clinical NLP Challenges Track 1 of the 2022 N2C2 has task researchers with developing solutions for generating information related to the context of medication mentions in EHR’s using data driven methods. A description of each task is given below. II-A Task 1: Medication Detection Task 1 for the 2022 N2C2 has task researchers with identifying medication mentions in EHR’s. This task has been well studied by NLP researchers with it also being a task in the 2018 N2C2 [ 8 ] . Knowing what medications and where they are in EHR’s are an essential step for further medication related information extraction. II-B Task 2: Medication Event Classification Task 2 for the 2022 N2C2 task researchers with classifying identified medication mentions from task 1 as having events associated with them. An event refers to any change that has to do with a particular medication within its context. The three classes for this task are given as disposition, no disposition, and undetermined. Disposition refers to an event occurring with the associated medication, no disposition refers to no event occurring for the associated medication, and undetermined refers to if annotators cannot determine if an event has occurred or not [ 2 ] . II-C Task 3: Multi-dimensional Medication Event Context Classification Task 3 for the 2022 N2C2 task researchers with classifying the context of medications that have been labeled with disposition in task 2. The context is classified across 5 dimensions, those dimensions being action, temporality, certainty, and actor, and negation [ 2 ] . Action refers to type of change being discussed and has the following 7 classes: Start, Stop, Increase, Decrease, Unique Dose, Other Change, and Unknown. Temporality refers to when the change occurred and has the following 4 classes: Past, Present, Future, Unknown. Certainty refers to if a change was implemented or just discussed and has the following 4 classes: Certain, Hypothetical, Conditional, and Unknown. Actor refers to who initiated the change and has the following and has the following 3 classes: Physician, Patient, Unknown. Negation refers to if the medication event is negated and has the following two classes: negated and not negated. III Method In this work pre-trained attention based large language encoders are the primary modeling component studied. Each pre-trained encoder can be easily fine-tuned by switching the output layer once weights are initialized in pre-training to one suitable for the applied task. All models are pre-trained on a Masked Language Modeling(MLM) task allowing them to capture general representations of language distributions, significantly reducing training time for researchers and increasing performance on downstream task. The modeling pipeline is shown in figure 1 .Words are first converted to subword tokens with each

Read the complete text