Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting
Hui Dai, Ryan Teehan, Parsa Torabian, Mengye Ren · 2026
Read Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting by Hui Dai, Ryan Teehan, Parsa Torabian, Mengye Ren (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27668.txt.
Introduction
Hui Dai, Ryan Teehan, Parsa Torabian, and Mengye Ren align language-model forecasts with human doubt through a Beta-Bernoulli calibrator. The calibrator turns any model's point forecast into a spread over event odds, guided by yes-no outcomes and crowd forecasts. It beats post-hoc tuning and forecast-tuned models while staying lightweight, and its epistemic uncertainty predicts forecast error better than stated confidence.
Catalog record
| CREATOR | Hui Dai, Ryan Teehan, Parsa Torabian, Mengye Ren |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60998 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27668
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting Hui Dai, Ryan Teehan, Parsa Torabian, Mengye Ren Abstract Probabilistic forecasting estimates the likelihood of uncertain future events. To improve LLM forecasting, existing methods typically learn from binary outcomes to output verbalized forecasts. However, while aggregated human forecasts contain rich information in both the crowd probability estimate and the degree of agreement among forecasters, how to utilize these signals remains underexplored. To address this, we propose the Beta-Bernoulli Calibrator (BBC), which converts an initial point estimate forecast from any model into a distribution over event likelihood, using supervision from both binary outcomes and human forecasts. BBC models event likelihood p ∼ Beta ( α , β ) p\sim\text{Beta}(\alpha,\beta) and outcome y ∼ Bernoulli ( p ) y\sim\text{Bernoulli}(p) , with the mean as the calibrated point forecast and the variance as the epistemic uncertainty. Our results show that BBC generally provides better calibrated and more accurate forecasts than both traditional post-hoc calibration methods and models fine-tuned specifically for forecasting, while remaining lightweight and having good generalization. We also show that the epistemic uncertainty captured by BBC is a more reliable predictor of forecasting error than verbalized confidence. A Beta-Bernoulli Calibrator for LLM Forecasting Hui Dai et al. 1 Introduction Making predictions about the future is an integral part of everyday decision-making. Individuals check weather forecasts to adjust travel plans, companies calculate the odds of a product’s success, and governments shape policy around economic and national security forecasts ( 24 ; 39 ) . Given large language models’ (LLMs) broad knowledge and reasoning capabilities, there is increasing interest in using LLMs for forecasting, typically by prompting the model to output a verbalized estimate of an event’s likelihood ( 21 ; 49 ; 45 ) . However, even state-of-the-art models struggle to outperform skilled human forecasters ( 21 ) . To improve forecasting capabilities, prior work has investigated supervised fine-tuning via distillation on subsets where the model outperforms humans ( 15 ) , as well as reinforcement learning (RL) using signals from realized outcomes ( 6 ; 41 ) . However, these approaches are resource-intensive and typically cannot be applied to black-box models. Moreover, human forecasts contain rich information about human sentiment and uncertainty, as they capture both the aggregate estimate of an event’s likelihood and the amount of consensus among the pool of forecasters. In spite of this, incorporating this information remains underexplored, and current methods do not capture the degree of consensus among the human forecasters. In this work, we ask: beyond eliciting verbalized probabilities, how can we calibrate model forecasts using supervision from both binary outcomes and human forecasts? Figure 1 : Overview of the Beta-Bernoulli Calibrator (BBC). Given a forecasting question and an initial verbalized forecast from an input LLM, BBC outputs a mixture of Beta distributions over the event probability. BBC is itself a small language model with an MLP head that predicts the Beta parameters, trained using supervision from both binary outcomes and human forecasts. The mean of the predicted distribution serves as the calibrated point forecast and the variance as epistemic uncertainty. As illustrated in Figure 1 , we propose a Beta-Bernoulli framework in which the event probability p is modeled as a Beta distribution, p\sim\text{Beta}(\alpha,\beta) , and the observed outcome y is a realization from a Bernoulli trial, y\sim\text{Bernoulli}(p) . In practice, we relax this single Beta to a mixture of K Betas for added flexibility. Taking the forecasting question and an initial verbalized forecast as input, the calibrator outputs the Beta parameters. Notably, this framework is model-agnostic. It is implemented using a small, open-source language model (the calibrator) to refine the initial textual forecast provided by a separate input LLM. This allows us to calibrate any input LLM’s beliefs without access to its internal representations, ensuring universal applicability and reducing training overhead. When learning from only binary outcomes, we show that the Beta–Bernoulli objective reduces to binary cross-entropy (BCE), a proper scoring rule that incentivizes truthful probability estimation. To incorporate signals beyond the binary outcome, we use human forecasts as distributional supervision for the Beta distribution. This enables the model to represent both the predicted event probability via the Beta mean, and epistemic uncertainty about that probability via the Beta variance. We evaluate our framework on data from prediction platforms Metaculus and Polymarket. Compared to uncertainty estimation and post-hoc calibration methods, our Beta-Bernoulli Calibrator (BBC) generally provides better-calibrated forecasts with stronger discrimination performance. We find that utilizing human forecasts as auxiliary supervision consistently improves discrimination compared to training only on binary outcomes. Moreover, this lightweight post-hoc adjustment even outperforms models that are fine-tuned specifically for forecasting, and provides further improvements when applied to them. In addition, we validate that BBC’s epistemic uncertainty is a strong predictor of forecasting errors, while verbalized confidence is a noisier signal. Finally, we test our calibrator’s generalization on the external Kalshi dataset, and observe consistent performance gain. Therefore, we present the following contributions: • Beta-Bernoulli Calibrator. We propose BBC, a lightweight, model-agnostic calibrator that converts an initial probability forecast into a distribution over event likelihood. This effectively captures both aleatoric and epistemic uncertainty in event forecasting. • Humans as Distributional Supervision. Human forecasts are a rich source of data that have so far been underutilized. In addition to providing an aggregated estimate of an event’s probability, the degree of consensus among the forecasts provides information about human sentiment and uncertainty. We use these forecasts as distributional supervision, which significantly improves AUC and allows us to go beyond only learning from binary outcomes. 2 Related work Traditional calibration and evidential methods. Earlier work on uncertainty calibration mainly focused on post-hoc calibration of classifier outputs ( 10 ; 32 ) . Parametric methods such as Platt scaling ( 34 ) and temperature scaling ( 14 ) learn global parameters to rescale prediction scores across all samples. For nonparametric methods, histogram binning ( 47 ) uses the empirical outcome frequencies in bins as calibrated scores, and isotonic regression ( 48 ) learns a monotonic piecewise constant function to transform uncalibrated scores. Post-hoc calibration relates to BBC’s role as a calibrator, while Evidential Deep Learning (EDL) relates to its probabilistic output parameterization. EDL models a Dirichlet over categorical probability predictions, with the Beta as the binary special case ( 36 ; 7 ) . However, our framework differs in two ways: (i) rather than collecting evidence directly from task inputs (like a cat image) in an end-to-end classifier, BBC is a stagewise calibrator that adjusts on top of another model’s natural language output, benefiting from its reasoning capability; (ii) rather than learning only from deterministic class labels, we introduce learning from human forecast distributions, which provides additional supervision and helps address the identifiability issue discussed in Section 4.3 . Uncertainty estimation and calibration in LLMs. In the context of LLMs, the focus shifts from calibrating classifier scores to estimating the reliability of natural language generations ( 37 ) . Most work has concentrated on tasks such as mathematics (e.g. GSM8K ( 8 ) ) and reasoning (e.g. HotpotQA ( 46 ) ). In these settings, the model generates an answer and the elicited confidence score should reflect the probability that