Diagnosing our datasets: How does my language model learn clinical information?
Furong Jia, David Sontag, Monica Agrawal · 2025
Read Diagnosing our datasets: How does my language model learn clinical information? by Furong Jia, David Sontag, Monica Agrawal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15024.txt.
Introduction
Furong Jia, David Sontag, and Monica Agrawal audit where medical chat models get their knowledge in a 2025 paper. Models never trained on hospital records still answer clinical questions well. Their smarts came from giant scraped text piles, and nobody had checked how the piles' makeup shapes the answers.
The team tested models on clinical jargon and on shaky medical claims, then counted matching material in the training piles. Jargon skill tracked how often terms appeared in the piles, yet terms thick in real hospital notes are often rare there. Worse, a real slice of the piles backs disputed claims that models then repeat. The audit argues for deliberate menu planning in training data.
Catalog record
| CREATOR | Furong Jia, David Sontag, Monica Agrawal |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15024
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Diagnosing our datasets: How does my language model learn clinical information? Furong Jia, David Sontag, Monica Agrawal Abstract Large language models (LLMs) have performed well across various clinical natural language processing tasks, despite not being directly trained on electronic health record (EHR) data. In this work, we examine how popular open-source LLMs learn clinical information from large mined corpora through two crucial but understudied lenses: (1) their interpretation of clinical jargon, a foundational ability for understanding real-world clinical notes, and (2) their responses to unsupported medical claims. For both use cases, we investigate the frequency of relevant clinical information in their corresponding pretraining corpora, the relationship between pretraining data composition and model outputs, and the sources underlying this data. To isolate clinical jargon understanding, we evaluate LLMs on a new dataset MedLingo . Unsurprisingly, we find that the frequency of clinical jargon mentions across major pretraining corpora correlates with model performance. However, jargon frequently appearing in clinical notes often rarely appears in pretraining corpora, revealing a mismatch between available data and real-world usage. Similarly, we find that a non-negligible portion of documents support disputed claims that can then be parroted by models. Finally, we classified and analyzed the types of online sources in which clinical jargon and unsupported medical claims appear, with implications for future dataset composition. † † volume: 287 † † year: 2025 † † workshop: Conference on Health, Inference, and Learning (CHIL) 2025 Data and Code Availability This paper leverages publicly available pre-training corpora, the Clinical Acronym Sense Inventory (CASI) dataset, and the MIMIC-IV dataset ( Moon et al. 2014 ; Johnson et al. 2023 ; Johnson et al. 2020 ) . The code, our new benchmark MedLingo , and analysis results can be found here: https://github.com/Flora-jia-jfr/diagnosing_our_datasets Institutional Review Board (IRB) This research does not require IRB approval. 1 Introduction In recent years, there has been significant warranted excitement around the application of large language models (LLMs) to diverse clinical applications, including information extraction, summarization, question answering, and trial matching ( Li et al. 2024a ; Van Veen et al. 2024 ; Agrawal et al. 2022 ; Zakka et al. 2024 ; Jin et al. 2024 ) . Researchers have found promising performance with both off-the-shelf general domain models (e.g., GPT and Llama families), as well as models fine-tuned specifically with biomedical corpora, such as PubMed, clinical guidelines, and medical question answering datasets ( Chen et al. 2023 ; Christophe et al. 2024 ) . Recent research has actually found that general domain models can perform just as well as these medically fine-tuned counterparts on standard benchmarks, despite being trained only on general online corpora ( Jeong et al. 2024a ; Li et al. 2024b ) . This raises major questions around where and how open-source LLMs are learning clinical information, given that they are not trained on EHR text. Figure 1: An overview of our analysis: 1) Benchmarking models on their knowledge of the clinical jargon and debunked medical claims. 2) Estimating the prevalence of clinical keywords in the pretraining corpora and examining its correlation with model performance, and 3) Investigating the sources of clinical data in pretraining corpora, both for jargon and unsupported medical claims. In this paper, we aim to better understand this phenomenon by characterizing the composition of clinical information in standard open-source training corpora and the relation to LLM behavior. Given that these corpora are generally multiple terabytes, it is most feasible to investigate this question through narrow well-defined tasks that enable us to probe the corpora for specific knowledge ( Kandpal et al. 2023 ) . Therefore we study model behavior and dataset composition through the lens of two tasks that are amenable to probing: (i) clinical jargon understanding and (ii) unsupported medical claims (overview in Figure 1 ). Clinical jargon understanding is particularly topical, as a recent systematic review found that only 5% of over 500 recent studies on LLMs in medicine have used real patient care data in their evaluation ( Bedi et al. 2024 ) . The rest rely often on synthetic or stylized clinical vignettes, as in licensing exams ( Raji et al. 2025 ) . While these evaluations may be able to portend medical reasoning capabilities, they don’t necessarily extend to tasks that require EHR note understanding. In particular, there is a significant distribution shift between clinical note text and biomedical text more broadly. Clinicians often have limited time to generate clinical documentation and therefore resort to shorthand (Figure 2 ). Therefore, we probe (i) LLMs for clinical jargon understanding and (ii) their training corpora for co-occurrences of both clinical shorthand and their expansion, from which these LLMs could have learned. Figure 2: Example of the difference between language in clinical notes vs. benchmarks. On the flip side, it is also important to understand how LLMs may be acquiring potentially dangerous information from these mined online corpora. Generation of unsupported medical claims poses risks when models are used in patient-facing applications, and there are existing concerns around model fragility and safety for high-stakes medical applications. For example, LLMs are sensitive to whether generic or brand names are used, and even small injections of incorrect information can propagate through models ( Gallifant et al. 2024 ; Alber et al. 2025 ) . Given the rise of unsupported medical claims overall online, it is interesting to see how this may extend to common pretraining corpora. Therefore, we probe for pairs of keywords (e.g. ‘Covid’, ‘5g’) that correspond to unsupported medical claims, both in models and in training corpora. In this work, we analyze how open-source LLMs acquire clinical information through the lenses of both clinical jargon and unsupported medical claims. We tackle this by benchmarking model performance, identifying the frequency of the knowledge in training corpora, and investigating the composition of clinical information in those corpora (Figure 1 ). Specifically, we make the following contributions: 1. Direct Evaluation of Clinical Jargon Knowledge: We introduce an evaluation framework and dataset centered on clinical jargon from real-world clinical notes. With this isolated assessment of how well LLMs understand real-world clinical text, we analyze how this performance relates to the frequency of clinical jargon in training corpora. 2. Investigation over unsupported medical claims: We probe LLMs for their generation of unsupported medical claims under different prompting techniques and connect this to how frequently these claims are supported vs. refuted in these corpora. 3. Analysis of Sources in Pretraining Corpora: Finally, we go past frequency alone to understand the sources from which this clinical information (and unsupported medical claims) is learned, which could inform future training corpora. 2 Related Work Clinical Jargon Understanding Unfortunately, many tasks in medical NLP don’t test on actual clinical text, but even when models are evaluated specifically on clinical text interpretation, the tasks don’t necessarily require a deep understanding of clinical jargon ( Jeong et al. 2024b ) . For example, MedNLI aims to test whether a given clinical premise supports a hypothesis. However, in practice, one can perform well even without access to the premise, due to shallow heuristics that are artifacts from dataset construction ( Herlihy and Rudinger 2021 ) . Similarly, synthetic GPT-4 generated questions often result in artificially simple datasets ( Bai et