NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical Text
Prajwal Kailas, Max Homilius, Rahul C. Deo, Calum A. MacRae · 2024
Read NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical Text by Prajwal Kailas, Max Homilius, Rahul C. Deo, Calum A. MacRae (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11477.txt.
Introduction
Prajwal Kailas and co-authors present a method to code medical notes by computer in this 2024 study. Doctors' free-text notes often describe a patient more exactly than the formal diagnosis codes used for care, research, and billing. Coding by hand is slow, and codes are often wrong or incomplete, so a sound auto-coder has real value.
Their system, NoteContrast, trains a language model on notes and code lists at the same time, pulling matched pairs close and pushing others apart. It learned from a large real-world set of hospital records. On three shared test sets built from intensive care notes, the joint training beat past best systems. The authors offer it as a step toward records that learn from their own text.
Catalog record
| CREATOR | Prajwal Kailas, Max Homilius, Rahul C. Deo, Calum A. MacRae |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-11477
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical Text Prajwal Kailas, Max Homilius, Rahul C. Deo, Calum A. MacRae Abstract Accurate diagnostic coding of medical notes is crucial for enhancing patient care, medical research, and error-free billing in healthcare organizations. Manual coding is a time-consuming task for providers, and diagnostic codes often exhibit low sensitivity and specificity, whereas the free text in medical notes can be a more precise description of a patient’s status. Thus, accurate automated diagnostic coding of medical notes has become critical for a learning healthcare system. Recent developments in long-document transformer architectures have enabled attention-based deep-learning models to adjudicate medical notes. In addition, contrastive loss functions have been used to jointly pre-train large language and image models with noisy labels. To further improve the automated adjudication of medical notes, we developed an approach based on i) models for ICD-10 diagnostic code sequences using a large real-world data set, ii) large language models for medical notes, and iii) contrastive pre-training to build an integrated model of both ICD-10 diagnostic codes and corresponding medical text. We demonstrate that a contrastive approach for pre-training improves performance over prior state-of-the-art models for the MIMIC-III-50, MIMIC-III-rare50, and MIMIC-III-full diagnostic coding tasks. These authors contributed equally † † volume: 225 † † year: 2023 † † submitted: 225 † † published: 225 † † workshop: Machine Learning for Health (ML4H) 2023 keywords medical text, diagnostic coding, contrastive training. 1 Introduction Accurate and automated diagnostic annotations of medical notes have become increasingly important in healthcare systems for improving the efficiency of care and enabling large-scale real-world data analyses. While diagnostic codes are vital for healthcare providers for tracking disease incidence and billing, manual coding is often cumbersome, time-consuming, and prone to errors. Machine learning methods have been developed to automate the diagnostic coding of medical notes. This is a complex task, as frequently multiple codes are applicable for a single note or single condition, and there are over 60,000 medical codes of varying specificity in the hierarchical International Classification of Diseases (ICD-10) system. While the large number of different codes allows for detailed and specific documentation of medical conditions, some codes are used distinctively by individual clinicians, or in different locations, and many are used infrequently. Often, only the most relevant codes for billing purposes are used per medical encounter. As a result, accurate coding can be seen as a multi-label problem with noisy training labels and a long tail of rarely applied diagnostic codes. Recent work has used long-document transformers and contrastive pre-training to annotate notes spanning thousands of tokens. However, these approaches typically depend on pre-defined biomedical ontologies like the Unified Medical Language System (UMLS) ( Yang et al., 2022 ; Yuan et al., 2022 ) , or ICD-9 and ICD-10 hierarchies ( Xie et al., 2019 ; Cao et al., 2020 ) to derive meaningful distances between different diagnoses, and often involve complex preprocessing. We sought to combine a data-driven contextual embedding of diagnostic codes with a straightforward contrastive pre-training objective to improve the automated annotation of medical notes. For this we developed an approach based on i) contextual embedding models for diagnostic codes based on a large real-world data set ii) large language models for medical notes and iii) contrastive pre-training to build an integrated model of both ICD-10 codes and corresponding medical text. We show that this contrastive approach incorporating real-world data significantly improved performance over prior state-of-the-art approaches using static sources of biomedical information in the MIMIC-50, MIMIC-50 rare and MIMIC-full benchmarks. 2 Related work 2.1 Diagnostic Coding Automatic diagnostic coding is a multi-label classification task assigning ICD codes to medical notes, typically ranging from several hundred to more than 2000 words per note. In addition, the label space is large and sparse, with over 60,000 codes in the most recent version of ICD-10, and a long tail of rarely diagnosed conditions. Several studies have proposed different methods to address this problem. Complex patterns between text and ICD codes were learned using variations of LSTM networks, dilated convolutions, residual connections, and per-label attention ( Mullenbach et al., 2018 ; Li and Yu, 2020 ; Ji et al., 2020 ; Vu et al., 2020 ) . Label representations were further improved by using graph convolution networks to capture the hierarchical structure of diagnostic codes ( Xie et al., 2019 ; Cao et al., 2020 ; Michalopoulos et al., 2022 ) . Shared representations can be further improved by extracting representations from low- and high-frequency codes via self-distillation ( Zhou et al., 2021 ) , and UMLS-based code synonyms have been used to provide more comprehensive knowledge than capturing code hierarchies ( Yuan et al., 2022 ) . Other studies have proposed various techniques to improve coding accuracy, such as using pre-trained biomedical language models with segment pooling to encode longer texts ( Huang et al., 2022 ) , exploiting the discourse structure of notes by utilizing section and reconciled label embeddings ( Zhang et al., 2022 ) , and incorporating tree-based features constructed from structured electronic health record (EHR) data such as lab values and medications as additional embedding vectors ( Liu et al., 2022 ) . Self-alignment learning with a hierarchical contrastive loss has been used to inject knowledge from biomedical ontologies, and prompt-based fine-tuning has been shown to be a powerful approach for predicting diagnostic codes ( Yang et al., 2022 ) . 2.2 Language Models for Biomedical Text There are many pre-trained transformer-based language models for clinical and biomedical tasks, including those trained on masked language modeling (MLM) as the original BERT architecture ( Devlin et al., 2019 ) and other pre-training objectives. Notable MLM models in the biomedical domain include SciBERT ( Beltagy et al., 2019 ) , BioBERT ( Lee et al., 2019 ) , ClinicalBERT ( Alsentzer et al., 2019 ) , and BioLM ( Lewis et al., 2020 ) , which are trained on the semantic scholar , scientific abstracts from PubMed and PMC or the MIMIC-III dataset ( Johnson et al., 2016 ) . Most of these models work at a sentence level and can handle up to 512 tokens, but for longer document-level tasks, the Clinical Longformer and Clinical BigBird ( Li et al., 2022 ) models can handle up to 4096 tokens and are based on Longformer ( Beltagy et al., 2020 ) and BigBird ( Zaheer et al., 2020 ) architectures, respectively. 2.3 Medical Code Representations Transformer models have also been used to encode temporal sequences of diagnostic codes across multiple hospital visits. These models are pre-trained with a masked language model objective on diagnostic, billing, and procedure code sequences. BEHRT ( Li et al., 2020 ) is a BERT model that uses additional position and segment embeddings to distinguish between visits and encode temporal information. It was trained on EHR data from 1.6 million patients and tested on three disease prediction tasks (disease diagnosis on the next visit, within 6 months, and within 12 months). MedBERT ( Rasmy et al., 2021 ) is conceptually similar to BEHRT but encodes both ICD-9 and ICD-10 codes and uses an additional serialization embedding to maintain the relative order of diagnostic codes within a visit. Lastly, G-BERT ( Shang et al., 2019