ScienceMeter : Tracking Scientific Knowledge Updates in Language Models

Yike Wang Shangbin Feng Yulia Tsvetkov Hannaneh Hajishirzi · 2025

Read ScienceMeter : Tracking Scientific Knowledge Updates in Language Models by Yike Wang Shangbin Feng Yulia Tsvetkov Hannaneh Hajishirzi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24302.txt.

Introduction

Yike Wang, Shangbin Feng, Yulia Tsvetkov, and Hannaneh Hajishirzi present a framework for tracking how language models update their science knowledge in a 2025 paper. Models are now used to support research, but science moves fast and stored facts go stale. Updating a model can also erase old knowledge or fail to take in new claims.

Their framework scores three things: how much old knowledge survives an update, how well new claims are learned, and how well the model projects to future claims. Testing five update methods across ten fields, they found even the best kept only part of the old knowledge and projected poorly. The work sets a clear bar for a hard, neglected problem.

Catalog record

CREATORYike Wang Shangbin Feng Yulia Tsvetkov Hannaneh Hajishirzi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24302

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ScienceMeter : Tracking Scientific Knowledge Updates in Language Models Yike Wang Shangbin Feng Yulia Tsvetkov Hannaneh Hajishirzi Abstract Large Language Models (LLMs) are increasingly used to support scientific research, but their knowledge of scientific advancements can quickly become outdated. We introduce ScienceMeter , a new framework for evaluating scientific knowledge update methods over scientific knowledge spanning the past, present, and future. ScienceMeter defines three metrics: knowledge preservation , the extent to which models’ understanding of previously learned papers is preserved; knowledge acquisition , how well scientific claims from newly introduced papers are acquired; and knowledge projection , the ability of the updated model to anticipate or generalize to related scientific claims that may emerge in the future. Using ScienceMeter , we evaluate the scientific knowledge of LLMs through claim judgment and generation tasks on a curated dataset across ten domains. We evaluate five representative knowledge update approaches and find that the best-performing knowledge update methods can preserve only 85.9% of existing knowledge, acquire 71.7% of new knowledge, and project 37.7% of future knowledge, underscoring that developing robust scientific knowledge update mechanisms is both crucial and challenging. 1 Introduction LLMs are being widely used to aid scientific research ( Liang et al., 2024 ; Luo et al., 2025 ; Hsu et al., 2024 ; Qi et al., 2023 ; Jansen et al., 2025 ) , with the potential to enable even greater future discoveries ( Ahn, 2024 ; Si et al., 2024 ) . However, due to the rapid pace of scientific advancements ( Kuhn, 1962 ) and the static nature of pre-trained LLMs ( Bommasani et al., 2021 ) , their scientific knowledge quickly becomes stale. We posit that effective scientific knowledge updates in LLMs must do more than add new information, but preserve existing knowledge, incorporate new findings, and enable generalization to reason about future or yet-undiscovered knowledge. Although generic update strategies have been explored, e.g., continual pre-training ( Gururangan et al., 2020 ) , instruction-tuning ( Wei et al., 2021 ) , or RAG ( Shi et al., 2023 ) , it is unclear whether these methods support these goals. To fill this gap, we propose ScienceMeter —a new framework for evaluating how LLMs update and reason over scientific knowledge. As shown in Figure 1 , our approach centers on tracking scientific knowledge updates as trajectories along three axes: preservation of prior knowledge (the parametric knowledge already encoded in the LLM), acquisition of new knowledge introduced through knowledge update methods, and projection of future knowledge not yet available to the model but can be inferred. Past discoveries serve as the foundation for future advancements and remain valuable for researchers seeking historical context, validation, or reinterpretation of previous findings, while the latter evaluates the utility of knowledge updates in enabling models to internalize new knowledge, moving beyond mere factual memorization to understand the underlying principles and patterns that govern such knowledge. This capability can facilitate hypothesis generation ( Qi et al., 2023 ) and the formulation of novel ideas ( Si et al., 2024 ) —key future usages of AI for science. Inspired by SciFact ( Wadden et al., 2020 ) , ScienceMeter operationalizes scientific knowledge as atomic scientific claims , i.e., atomic verifiable statements expressing a finding about one aspect of a scientific entity or process, which can be verified against a single source. While prior work in general domains often represents knowledge as factoid information or structured entity tuples ( Tang et al., 2024 ; Yin et al., 2024 ) , we argue that scientific claims are more appropriate and meaningful units of knowledge in scientific contexts, as they better capture the core insights and implications of research beyond isolated numerical values. In ScienceMeter , we curate a dataset spanning ten rapidly evolving scientific fields. As LLMs become increasingly integrated into these domains, it is essential to evaluate whether existing knowledge update methods can support their progress. Related scientific literature is grouped chronologically to represent prior, new, and future knowledge. To evaluate scientific knowledge, we focus on two tasks, claim judgment and claim generation , and emphasize both factual accuracy and the model’s confidence to better reflect the rigor of the scientific domain. Specifically, we categorize the model’s knowledge of a claim as correct (factually accurate and confident), incorrect (factually inaccurate and confident), or unknown (not confident) and quantify the percentage of two types of errors in preservation, acquisition, and projection, respectively. We evaluate LLMs’ scientific knowledge updates using five methods spanning training, inference, or both. Experimental results across standard and frontier models highlight that the best-performing knowledge update method achieves on average only 85.9% on knowledge preservation, 71.7% on knowledge acquisition, and 37.7% on knowledge projection in the domain of Computer Science . While inference-time update methods tend to be effective for large models, smaller models require training-based approaches to achieve comparable performance. Cross-domain analysis reveals that performance on these objectives is correlated, with knowledge preservation and projection heavily influenced by domain volatility, while the availability of domain knowledge during pretraining has limited impact. Moreover, even applying on domain-adapted scientific LLMs struggle to balance these objectives, underscoring persistent challenges in updating scientific knowledge in LLMs. Figure 1: We propose an evaluation framework, ScienceMeter , along with novel metrics to quantify the reliability and usefulness of scientific knowledge updates in LLMs: preservation of existing scientific claims and their linkage to existing literature, acquisition of new claims from emerging research, and projection of future claims. For example, when we update an LLM with a new paper introducing the features of Microglia, our framework evaluates the acquisition of this new knowledge, as well as the preservation of existing knowledge about the fundamentals of Microglia learned from previous literature, and the ability of LLMs to effectively use its parametric knowledge to extrapolate future knowledge on Microglia, such as its potential roles in Alzheimer’s disease. 2 ScienceMeter To systematically evaluate scientific knowledge updates in LLMs, ScienceMeter integrates three core components: (1) a carefully curated multi-domain dataset of scientific papers and scientific claims (§ 2.1 ); (2) evaluation of a model’s scientific knowledge through claim judgment and generation tasks, assessed by both factual accuracy and model confidence (§ 2.2 ); and (3) novel metrics for evaluating knowledge update methods that aggregate claims from past/present/future data sets (§ 2.3 ). An overview of ScienceMeter is in Figure 2 . Figure 2: An overview of ScienceMeter : (1) We curate chronologically organized datasets of scientific papers and claims; (2) define claim judgment and generation tasks to evaluate scientific knowledge, incorporating both factual accuracy and model confidence; and (3) introduce metrics for evaluating scientific knowledge updates. 2.1 Dataset Construction Paper Collection We identify 10 rapidly evolving scientific domains, including Computer Science , Medicine , Biology , Materials Science , Psychology , Business , Political Science , Environmental Science , Agricultural and Food Sciences , and Education . For each domain, we randomly retrieve 1,000 journal or conference papers (excluding review or survey papers) published at least three months before the knowledge cutoff date of the given model through Semantic Scholar ( Ammar et al., 2018 ) , an openly accessible academic search platform that indexes publisher-authorized metadata and papers. To obtain more recent knowledge on

Read the complete text