QuantSightBench: Evaluating LLM Quantitative Forecasting with Prediction Intervals
Jeremy Qin, Maksym Andriushchenko, [0.6em] ELLIS Institute Tübingen, [0.3em] Tübingen AI Center · 2026
Read QuantSightBench: Evaluating LLM Quantitative Forecasting with Prediction Intervals by Jeremy Qin, Maksym Andriushchenko, [0.6em] ELLIS Institute Tübingen, [0.3em] Tübingen AI Center (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15859.txt.
Introduction
Qin and Andriushchenko present QuantSightBench, a benchmark that grades language models on quantitative forecasting through prediction intervals rather than single-point guesses or simple choices. They force a sense of scale, fit across confidence levels, and tuning across a full range of results. Testing 11 frontier and open-weight models, they find none reaches 90 percent coverage. The best, Gemini 3.1 Pro at 79.1 percent, still falls over 10 points short, with models too sure of themselves at extreme sizes.
Catalog record
| CREATOR | Jeremy Qin, Maksym Andriushchenko, [0.6em] ELLIS Institute Tübingen, [0.3em] Tübingen AI Center |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33398 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.15859
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
QuantSightBench: Evaluating LLM Quantitative Forecasting with Prediction Intervals Jeremy Qin, Maksym Andriushchenko, [0.6em] ELLIS Institute Tübingen, [0.3em] Tübingen AI Center Abstract Forecasting has become a natural benchmark for reasoning under uncertainty. Yet existing evaluations of large language models remain limited to judgmental tasks in simple formats, such as binary or multiple-choice questions. In practice, however, forecasting spans a far broader scope. Across domains such as economics, public health, and social demographics, decisions hinge on numerical estimates over continuous quantities, a capability that current benchmarks do not capture. Evaluating such estimates requires a format that makes uncertainty explicit and testable. We propose prediction intervals as a natural and rigorous interface for this purpose. They demand scale awareness, internal consistency across confidence levels, and calibration over a continuum of outcomes, making them a more suitable evaluation format than point estimates for numerical forecasting. To assess this capability, we introduce a new benchmark QuantSightBench, and evaluate frontier models under multiple settings, assessing both empirical coverage and interval sharpness. Our results show that none of the 11 evaluated frontier and open-weight models achieves the 90% coverage target, with the top performers Gemini 3.1 Pro (79.1%), Grok 4 (76.4%), and GPT-5.4 (75.3%) all falling at least 10 percentage points short. Calibration degrades sharply at extreme magnitudes, revealing systematic overconfidence across all evaluated models. Leaderboard Code 1 Introduction Forecasting, the prediction of future outcomes under uncertainty, is foundational to decision-making across nearly every domain. In high-stakes settings such as epidemiology, climate policy, and macroeconomic planning, forecast quality directly shapes interventions that affect millions of lives. But the relevance of forecasting extends well beyond institutional contexts. Individuals routinely make forecast-dependent decisions: estimating future expenses, anticipating price changes, or gauging demand for a product. In each case, producing a good forecast requires gathering information from diverse sources, synthesizing evidence, and iteratively refining one’s estimates, capabilities that are increasingly described as agentic. Large language models, with their growing ability to search, reason, and synthesize information, are already being consulted for exactly these kinds of judgments ( 14 ; 26 ; 16 ) . Understanding how well they perform is therefore necessary to ensure their reliable use in decision-making, as models that convey poorly calibrated uncertainty risk silently misinforming decisions at every scale. Beyond its practical relevance, forecasting is also particularly well suited as a benchmark for evaluating LLM capabilities. Foresight, the ability to anticipate outcomes that have not yet occurred, is a demanding test of reasoning that integrates world knowledge, contextual understanding, and uncertainty estimation. Moreover, forecasting tasks are naturally grounded in future events, which provides two key properties for rigorous evaluation: ground truth labels are obtained automatically once outcomes are resolved, and temporal separation between training data and evaluation targets offers a principled mechanism for preventing data leakage when the benchmark is carefully constructed ( 17 ) . However, existing benchmarks evaluate forecasting through a narrow lens by primarily testing whether models can predict discrete event outcomes through binary or multiple-choice formats, which are susceptible to evaluation shortcuts ( 2 ) , leaving out a broad class of tasks across economics, public health, and social demographics where decisions depend not on whether something will happen, but on how much. Figure 1: Empirical coverage under the agentic setting. Coverage reflects the proportion of true values captured by 90% prediction intervals. No model achieves the target, indicating systematic overconfidence across all evaluated models. To address this gap, we propose prediction intervals as an evaluation format for numerical forecasting in LLMs. A prediction interval is a bounded numerical range at a specified confidence level, formalizing how people naturally reason about uncertain quantities: not as point estimates, but as ranges (”between 2% and 4%,” ”around 10 to 15 million”). Notably, this intuition already surfaces in frontier models. When asked to forecast a quantity such as the population of Montreal in 2027, models like GPT-5.1 spontaneously produce interval-valued responses rather than single point estimates, suggesting that this format aligns not only with human reasoning but also with how models naturally express numerical uncertainty. Beyond being intuitive, prediction intervals offer a richer and more robust evaluation format than point forecasts by making uncertainty explicit, while remaining more practical to elicit and evaluate than full probability distributions ( 8 ; 7 ) . They also impose meaningful demands on model capability: producing calibrated intervals requires scale awareness, internal consistency across confidence levels, and calibration over a continuum making this task a good testbed for capability evaluation. To this end, we make the following contributions: • We introduce QuantSightBench, a benchmark for evaluating LLM numerical forecasting capability through prediction intervals across diverse domains. • We evaluate models spanning frontier and open-weight families under multiple settings (zero-shot, context-grounded, and agentic), providing a controlled comparison of how evaluation setup affects calibration. • We analyze calibration across models, settings, and scales, identifying systematic overconfidence and key failure modes such as scale sensitivity. Question Background Answer What will be the annual silver supply deficit (in million ounces) in 2025? The silver market faces structural deficits when demand exceeds supply, impacting prices and industrial availability. 117.6 What is the annual climate finance commitment agreed upon by developed nations for developing countries by 2035? Developed nations have pledged financial support to help developing countries transition to clean energy and adapt to climate change. 300 How many fatalities were confirmed in the Air India Flight AI 171 crash by October 2025? Air India Flight AI 171 crashed shortly after takeoff from Ahmedabad, leading to a major aviation disaster investigation. 260 Table 1: Forecasting question examples from the benchmark. 2 Related Work 2.1 Forecasting with LLMs Forecasting has a long history across diverse fields, from time-series methods in econometrics ( 1 ) and weather prediction ( 19 ) to probabilistic forecasting in finance ( 4 ) . A growing body of work has explored the use of large language models for applications in time-series forecasting, leveraging their pattern recognition and contextual reasoning capabilities to predict structured numerical sequences ( 12 ; 9 ) . More recently, judgmental forecasting, where predictions are expressed in natural language rather than derived from statistical models, has emerged as an increasingly popular benchmark for evaluating LLM reasoning under uncertainty. Platforms such as ForecastBench ( 14 ) , FutureX ( 26 ; 16 ) , and the Metaculus AI Benchmarking Series provide dynamic leaderboards that track model performance over time. These benchmarks predominantly evaluate models on binary or multiple-choice questions sourced from prediction markets. However, this evaluation paradigm is not without limitations, 17 identified methodological concerns including data leakage and distributional biases inherent to prediction market questions. To diversify beyond prediction markets, recent work has also explored automated question generation from news articles ( 5 ; 10 ; 22 ) , and the OpenForecast pipeline ( 3 ) scaled this approach through a fully automated framework that uses static news snapshots to mitigate leakage risks. Other works such as 18 also looked at different evaluation methodology such as using consistency checks that measures logical coherence across related predictions without requiring ground truth resolution. Despite this progress, evaluation across all of these efforts remains limited to purely judgmental forecasting. 2.2 Uncertainty Estimation in LLMs A growing body of work has studied