Pitfalls in Evaluating Language Model Forecasters
Daniel Paleka, Shashwat Goel, Jonas Geiping, Florian Tramèr · 2025
Read Pitfalls in Evaluating Language Model Forecasters by Daniel Paleka, Shashwat Goel, Jonas Geiping, Florian Tramèr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00723.txt.
Introduction
Some reports claim that large language models can forecast future events as well as skilled humans. Daniel Paleka and co-authors urge care with such claims in a 2025 paper. They argue that testing a model's forecasting skill is much trickier than it looks.
The authors sort the traps into two groups. The first is leakage, where facts from after the forecast date sneak into the test and inflate scores. The second is the gap between test scores and real-world skill, which may not carry over. With examples from past studies, they show how these flaws can prop up weak claims. The paper calls for stricter test designs before anyone trusts a model forecaster.
Catalog record
| CREATOR | Daniel Paleka, Shashwat Goel, Jonas Geiping, Florian Tramèr |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00723
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pitfalls in Evaluating Language Model Forecasters Daniel Paleka, Shashwat Goel, Jonas Geiping, Florian Tramèr Abstract Large language models (LLMs) have recently been applied to forecasting tasks, with some works claiming these systems match or exceed human performance. In this paper, we argue that, as a community, we should be careful about such conclusions as evaluating LLM forecasters presents unique challenges. We identify two broad categories of issues: (1) difficulty in trusting evaluation results due to many forms of temporal leakage, and (2) difficulty in extrapolating from evaluation performance to real-world forecasting. Through systematic analysis and concrete examples from prior work, we demonstrate how evaluation flaws can raise concerns about current and future performance claims. We argue that more rigorous evaluation methodologies are needed to confidently assess the forecasting abilities of LLMs. 1 Introduction Forecasting, the task of assigning probabilities to future events, represents a critical capability for decision-making across various domains. Several recent studies have explored the potential of Large Language Models (LLMs) as forecasting systems, sometimes even suggesting that LLMs can already rival human performance ( Halawi et al. 2024 ; Phan et al. 2024 ; Schoenegger et al. 2024 ) . However, this paper identifies significant concerns in the trustworthiness of reported results for both existing and future LLM forecasting systems. We expand on several issues that are partially known in the forecasting community (e.g., ( Bosse et al. 2024 ; Sempere and Lawsen 2021 ; Arnott et al. 2018 ) ), but have not been comprehensively analyzed for LLM forecasters. Through concrete examples, we illustrate subtle challenges in forecasting evaluation, and discuss how these issues may have led to overly optimistic assessments of LLM forecasting abilities in prior work. The challenges we identify fall into two broad categories: 1. Trusting evaluation results : Various forms of data leakage make it difficult to ensure that models truly predict future events rather than using information after the forecast date. 2. Extrapolating from evaluation results to real-world performance : Good performance on forecasting benchmarks may not necessarily correspond to good forecasting abilities due to issues like benchmark gaming and distributional biases. We provide an overview of these challenges below, and analyze them in more depth in Sections 2 - 3 . Challenge 1: Establishing trustworthy evaluation results. The gold standard for evaluating a forecaster involves running it on unresolved questions, waiting until the questions resolve, and then scoring the predictions. However, this approach is impractical for rapid model evaluation. Thus, researchers typically resort to backtesting or retrodiction ( Zou et al. 2022 ) , where the forecasting system is given knowledge as of some past time T and asked to forecast events between time T and the present. Although appealing in principle, backtesting introduces several issues: • Logical leakage: The very nature of backtesting can logically constrain possible answers. Consider a time-traveler analogy: If someone from 2035 asks you to predict if we will find alien life before 2040, you can deduce that the answer must be "yes"–otherwise, the time traveler would not yet have definitive evidence to grade your prediction. We show that a significant percentage of questions in prior forecasting benchmarks permit similar logical deductions of the correct answer. • Unreliable date-restricted retrieval : Many forecasting systems incorporate retrieval components (e.g., search engines) that are restricted to data available at time T . Yet, date metadata on documents is often inaccurate, allowing future data to leak into the system. More subtly, the retrieval model itself has typically been trained on future data, causing leakage through learned associations. For example, searching for "January 6th" with a date restriction to 2020 returns documents with an abnormally strong association to U.S. politics (by 2020 standards). • Over-reliance on model cutoff dates : Researchers often assume that models have no knowledge beyond their reported training cutoff dates. However, cutoffs are more of a guideline than a guarantee, and evidence suggests that models possess knowledge of some events beyond these dates. Challenge 2: Extrapolating from benchmark performance. Even with a sound evaluation, translating results into real-world forecasting ability faces additional issues: • Piggybacking on human forecasts : Many forecasting datasets originate from human prediction platforms. Thus, human forecasts are likely available to the LLM (in its training data or through its retrieval system). Claims that LLMs “match human performance” may then be circular: the models might simply be copying human forecasts rather than demonstrating independent ability. • Gaming benchmarks through betting : Unlike many AI tasks, forecasting benchmarks can reward strategic gambling over accurate uncertainty estimation. For example, consider forecasting U.S. politics in 2025, from a vantage point in 2023 with a 50/50 prior on the presidential election. The benchmark-optimal strategy might be to commit to one outcome and condition all predictions on that guess. With 50% chance, this yields excellent performance; with 50% chance, it fails completely. This has a much higher chance of topping the benchmark than a calibrated strategy. • Skewed data distribution : Questions on forecasting platforms often focus on topics that competitive forecasters find interesting, creating potential distribution biases. When curating benchmarks for backtesting, these biases can be exacerbated by constraints on which questions can be resolved within the evaluation timeframe. Although data biases exist in many ML benchmarks (e.g., ImageNet’s focus on dog breeds still produces transferable visual features), there is little evidence that performance on current forecasting benchmarks yields generalizable forecasting capabilities. • Metrics are counterintuitive : Typical metrics used to evaluate forecasters can fail to incentivize good forecasting. For example, calibration can penalize useful forecasting, while Brier scores over-emphasize performance on questions with a high base rate. Looking ahead: Challenges in optimizing better forecasters. Current LLM forecasters primarily leverage existing models’ knowledge and reasoning capabilities. As the field advances, a natural next step will be applying optimization pressure specifically to improve forecasting performance. However, temporal correlations in data make this optimization challenging. Naively training on question-answer pairs over a time period creates temporal leakage, as early samples in training can leak information relevant to later ones (e.g., "Who will win the election?" followed by "Who will win the primaries?"). Temporally sorting training samples still fails to properly simulate the task of predicting events further in the future. By developing these arguments, we argue that the monitoring of forecasting capabilities is appreciably harder than the evaluation of knowledge about the past or present, as it has unique issues on top of all the existing issues with machine learning evaluation ( Leech et al. 2024 ) . Forecasting future events is a fascinating and challenging task for LLMs, with wide-ranging implications, and we hope to convince the community to devote more attention to evaluating it carefully. 2 Challenge 1: Establishing Trustworthy Evaluation Results The ideal way to evaluate a forecasting system is to ask questions about future events, collect predictions, and then score the model’s answers as the events resolve over time. However, such an evaluation can last months or years. Instead, researchers rely on backtests , which constrains a system to only access knowledge at some past time T , and then ask it questions that resolved between time T and the present ( Tashman 2000 ) . This allows for immediate feedback, which is essential