Memorize and Rank: Elevating Large Language Models for Clinical Diagnosis Prediction
Mingyu Derek Ma, Xiaoxuan Wang, Yijia Xiao, Anthony Cuturrufo, Vijay S Nori, Eran Halperin , 2 ,2, Wei Wang · 2025
Read Memorize and Rank: Elevating Large Language Models for Clinical Diagnosis Prediction by Mingyu Derek Ma, Xiaoxuan Wang, Yijia Xiao, Anthony Cuturrufo, Vijay S Nori, Eran Halperin , 2 ,2, Wei Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17326.txt.
Introduction
Ma and co-authors introduce MERA, a model that uses large language models to predict diagnoses from a patient's medical history. Early detection guides timely care, yet patient data are scarce and the list of possible diseases is huge. Their 2025 model bridges plain language medical knowledge with the codes used in records.
Fine tuning teaches the model to memorize disease concepts in words. A layered contrast method then ranks candidate diseases, easing the large choice problem. Tests on two versions of the MIMIC intensive care datasets set new top scores for the task. The authors say generative models gain a lot from this memorization and ranking design.
Catalog record
| CREATOR | Mingyu Derek Ma, Xiaoxuan Wang, Yijia Xiao, Anthony Cuturrufo, Vijay S Nori, Eran Halperin , 2 ,2, Wei Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-17326
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Memorize and Rank: Elevating Large Language Models for Clinical Diagnosis Prediction Mingyu Derek Ma, Xiaoxuan Wang, Yijia Xiao, Anthony Cuturrufo, Vijay S Nori, Eran Halperin , 2 ,2, Wei Wang Abstract Clinical diagnosis prediction models, when provided with a patient’s medical history, aim to detect potential diseases early, facilitating timely intervention and improving prognostic outcomes. However, the inherent scarcity of patient data and large disease candidate space often pose challenges in developing satisfactory models for this intricate task. The exploration of leveraging Large Language Models (LLMs) for encapsulating clinical decision processes has been limited. We introduce Mera , a clinical diagnosis prediction model that bridges pertaining natural language knowledge with medical practice. We apply hierarchical contrastive learning on a disease candidate ranking list to alleviate the large decision space issue. With concept memorization through fine-tuning, we bridge the natural language clinical knowledge with medical codes. Experimental results on MIMIC-III and IV datasets show that Mera achieves the state-of-the-art diagnosis prediction performance and dramatically elevates the diagnosis prediction capabilities of generative LMs. 1 University of California, Los Angeles 2 Optum AI {ma, xw27, yijia.xiao, acc, weiwang}@cs.ucla.edu [email protected], [email protected] 1 Introduction Electronic Health Records (EHR), containing patient status and diagnoses, embody valuable domain expertise and clinical operation patterns ( Caufield et al. 2019 ) . Clinicians make diagnosis judgments based on their extensive medical knowledge, acquired through years of education from textbooks and literature, as well as their accumulated experience derived from clinical practice. Clinical diagnosis prediction aims to predict patients’ diseases that are highly likely to be diagnosed in the upcoming hospital admission by analyzing the patients’ past diagnoses. The input and output are both presented in sequences of medical codes, which do not directly convey semantic information nor disease property. The resulting AI-enhanced diagnosis system ( Morid, Sheng, and Dunbar 2023 ) may enable early warning of diseases ( Rochefort, Buckeridge, and Forster 2015 ) , optimized clinical resource allocation ( Yadav et al. 2013 ) , and better risk estimation for sustainable insurance ( Hsu et al. 2016 ) . Two primary challenges in diagnosis prediction have driven various research efforts ( Wornow et al. 2023b ) but remain unsolved. First, what would be the best practice to incorporate clinical knowledge into the model? Existing works initialize concept embeddings from natural language descriptions ( Wu et al. 2023b ; Bornet et al. 2023 ) , or enrich patient representation with external disease ontologies ( An et al. 2023 ; Cheong et al. 2023 ) . However, a significant gap persists between the primary knowledge modality, i . e . natural language, and the model’s hidden representation. Second, how can we handle the large candidate space when making predictions and exploit the supervisory signals induced from this candidate space? The commonly used International Classification of Diseases (ICD) coding system encodes 13k+ diseases ( Cartwright 2013 ) . Existing works typically treat the task as k -way classification where k is the number of possible diseases, and then apply cross entropy loss for each disease individually. These approaches often overlook the dependencies among diseases and the structural nuances within the diagnosis coding system. Generative Language Models (LM), especially the Large Language Models (LLM), are trained to predict the next token, adhere to task instructions ( Brown and et al. 2020 ; Ma et al. 2024a ) , and align with human preferences ( Ouyang and et al 2022 ) . These models exhibit superior capabilities in language understanding and reasoning, as shown by their performance on science-based benchmarks ( Ma et al. 2024b ; Wu et al. 2023a ; Zhang et al. 2024 ) . During the pretraining stage, LLMs assimilate a large amount of knowledge extracted from literature and online corpora. However, there remains an underexplored domain in using LLM for clinical diagnosis prediction, due to the aforementioned gap between natural language and medical code, as well as the disparity between the token-level optimization process and the large candidate outcome space. These challenges impede the effective application of generative LMs to diagnosis prediction tasks, even as the state-of-the-art models predominantly rely on graph neural networks without fully harnessing natural language knowledge ( Yang et al. 2023b ; Wu et al. 2023b ; An et al. 2023 ) . Fine-tuning generative LM LLaMA2 ( Touvron and et al. 2023 ) directly on diagnosis prediction yields almost 20-point lower recall@20 than GNN-based existing best model ( Yang et al. 2023b ) as shown in Table 1 . There are some studies that use transformer-based LM for clinical outcome prediction, but they either do not support structured data as input ( Niu et al. 2024 ; Wang et al. 2023a ) , not compatible with mainstream LLMs ( Rupp, Peter, and Pattipaka 2023 ; Guo et al. 2023 ) , or only work for narrow output space with few classes ( Wang et al. 2023a ; Shoham and Rappoport 2023 ) . To tackle these challenges, we propose Mera , an LLM designed for clinical diagnosis prediction that incorporates a comprehensive understanding of clinical knowledge by leveraging relationships among medical codes and offers extensive supervision over the output space. The patient’s historical diagnosis results are formulated as linear sequences and the LLM is tasked with generating a probability distribution for the diagnosis results in the subsequent visit. Compared with the ordinary paradigm that optimizes the probability of generating the correct token, we optimize the outcome directly. To enhance the inter-visit causal reasoning, we employ contrastive learning to compel the model to distinguish true diagnoses from false ones. The contrastive learning process is extended to multiple levels in the hierarchical organization of the medical codes within the ICD coding ontology. The model is learned to distinguish the true diagnoses from a pool of potential diagnoses while the pool is increasingly relevant to the true ones. To regularize the diagnosis predictions to follow intra-visit diagnosis patterns, we develop a teaching-forcing strategy to optimize the medical code ranking, assuming partial diagnoses of the visit are known. To allow the model to grasp the comprehensive clinical semantics and diagnosis property of each medical code, we fine-tune the LM to “memorize” the mapping between medical codes and their natural language definitions. Consequently, this process bridges the gap between raw codes and their contextual medical meanings and equips the LM to capture the intricate code dependencies that are crucial for precise diagnosis assessments. We validate the effectiveness of Mera in general diagnosis and heart failure prediction tasks on the patient records in MIMIC-III ( Johnson et al. 2016 ) and MIMIC-IV ( Johnson et al. 2023 ) datasets. Mera yields significant improvements over the existing state-of-the-art models across tasks on all datasets while having almost perfect memorization of bidirectional medical code-definition mapping. An extensive analysis of leading LLM’s medical code understanding and diagnosis prediction capabilities is conducted, and we observe that GPT-4 is still far behind fine-tuned models on both tasks. We further conduct ablation studies to validate the effectiveness of the proposed novel design choices. 2 Preliminaries 2.1 Task Formulations Mera can be applied for any task whose output is a collection of candidates belonging to a pre-defined decision space. We introduce widely used diagnosis prediction settings as