Proper Calibeating
Dean P. Foster, Sergiu Hart · 2026
Read Proper Calibeating by Dean P. Foster, Sergiu Hart (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26703.txt.
Introduction
Foster and Hart extend two forecasting standards, calibration and calibeating, from the quadratic scoring rule to every bounded proper scoring rule. They show calibration always implies proper calibration, but calibeating does not always imply proper calibeating. They then give methods that guarantee proper calibeating, prove complete calibeating is always proper, and tie proper calibration to universal no regret in decision making.
Catalog record
| CREATOR | Dean P. Foster, Sergiu Hart |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 108419 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26703
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Proper Calibeating Thanks: Previous version: December 2025. A general presentation that includes results from this paper is available at http://www.ma.huji.ac.il/hart/pres.html#calib-beat-p Dean P. Foster, Sergiu Hart Abstract The classic concept of “calibrated forecasts” and its more recent refinement, “calibeating,” are defined with respect to the standard quadratic scoring rule. We extend these notions to the class of proper scoring rules (for which the best forecast is the true distribution) and define proper-calibration and proper-calibeating by requiring the errors to converge to zero uniformly over all bounded proper scoring rules. We first establish that calibration always implies proper-calibration, whereas calibeating need not imply proper-calibeating. Second, we show how to guarantee proper-calibeating and proper-multicalibeating. Finally, we demonstrate the equivalence between proper-calibration and universal no regret when best replying to forecasts in decision-making under uncertainty. 1 Introduction Forecasting the probability of future events is a foundational problem. Forecasters issue probabilistic predictions that are then compared with the outcomes that materialize. How should one evaluate such a forecaster? A natural approach is to use a scoring rule : a penalty function that assigns a loss to each forecast–outcome pair. A scoring rule is deemed proper if the loss is minimized when the forecast equals the underlying probability distribution of the outcome. The standard and most widely used proper scoring rule is the quadratic rule (Brier 1950; see Savage 1971, Schervish 1989, and the comprehensive treatment in Gneiting and Raftery 2007). A classic result (Sanders 1963; Murphy 1972; DeGroot and Fienberg 1983) is that the Brier quadratic score \mathcal{B} decomposes as \mathcal{B}=\mathcal{K}+\mathcal{R}, where \mathcal{K} is the calibration score—measuring how close the forecasts are to the realized frequencies—and \mathcal{R} is the refinement score—measuring how informative the partitioning of the outcomes into bins is, where the partitioning is determined by the announced forecasts (see Section 2.2 for the precise setup and definitions). A surprising result of Foster and Vohra (1998) is that one can generate forecasts that are guaranteed to be calibrated (i.e., \mathcal{K} converges to zero), no matter what the outcomes turn out to be (for the ensuing literature, see the survey of Olszewski 2015 1 1 1 For work on calibration that appeared after this survey, see Foster and Hart (2018, 2021, 2023), Hart (2025), Marx, Kuleshov, and Ermon (2024), Okoroafor, Sun, and Kleinberg (2024), Qiao and Valiant (2021). ). In Foster and Hart (2023), we introduced the notion of “calibeating”: beating a reference forecaster by achieving a Brier score no worse than its refinement score---thus gaining calibration without losing expertise, which, as we argued there, is reflected in the induced partitioning into bins and the resulting refinement score. We showed that calibeating can be achieved in several ways: by a simple deterministic online procedure; by a stochastic procedure that is itself calibrated; and by a deterministic continuously calibrated procedure. All these results, however, were stated exclusively for the quadratic score. 2 2 2 In Appendix A.9 of the full version of the paper, Foster and Hart (2022), we show how a modification of our simple procedure yields calibeating with respect to the logarithmic proper scoring rule. However, there is an extensive class of proper scoring rules (in particular, every decision-making setting under uncertainty generates one; see below and Section 6 ). A fundamental concern is thus whether these guarantees are robust or merely artifacts of the quadratic functional form. If a forecaster’s performance evaluation is sensitive to the specific choice of a proper loss function, its theoretical and practical relevance may well be questionable. This raises the natural question: If a procedure is calibrated (or calibeats) under the quadratic score, do these guarantees persist under every other proper scoring rule? This paper answers that question for proper scoring rules that are bounded. 3 3 3 This excludes the logarithmic scoring rule. Some of the proper-calibeating results are limited to Lipschitz proper scoring rules. The answer turns out to be positive for calibration, and negative for calibeating. We will call a procedure proper-calibrated (respectively, proper-calibeating ) if the corresponding guarantee holds simultaneously for all bounded proper scoring rules (see Section 3 for the precise definitions). Calibration entails proper-calibration. We show, first, that calibration transfers for free: every calibrated procedure is automatically proper-calibrated. The key observation is that the calibration score under any bounded proper scoring rule is bounded by a multiple of the square root of the quadratic calibration score. Consequently, all established calibrated forecasting procedures---the stochastic procedures, as well as the deterministic ones that are continuously calibrated 4 4 4 See Foster and Hart (2021), with precursors Kakade and Foster (2004), Foster and Kakade (2006), Foster and Hart (2018). —are already proper-calibrated, with no modification needed. Calibeating does not entail proper-calibeating. Our second result is perhaps a surprise: unlike calibration, calibeating does not transfer to proper-calibeating. We exhibit a concrete example (Section 5.1 ) in which a forecasting sequence \mathbf{c} calibeats a reference sequence \mathbf{b} under the quadratic scoring rule, yet fails to calibeat \mathbf{b} under, for instance, the 2 -spherical proper scoring rule (which is bounded, and even Lipschitz). Both sequences \mathbf{b} and \mathbf{c} are perfectly calibrated; the failure is purely in the refinement comparison. The reason for the asymmetry between calibration and calibeating is instructive. Calibration scores shrink whenever the quadratic calibration score shrinks, regardless of which proper scoring rule is used; but refinement scores under different proper rules need not move in tandem. Achieving proper-calibeating. We next provide three positive results on proper-calibeating (i.e., calibeating simultaneously under all bounded proper scoring rules). First, we show that a stochastic procedure that calibeats a finer binning—one that cross-classifies forecasts by the bins of both the reference forecaster and the procedure itself—is proper-calibeating as well as proper-calibrated. A key insight is that it is not enough to calibeat the reference forecaster and to be calibrated separately; what matters is calibeating the joint binning, which keeps track of how the two sets of bins interact (Appendix A.2 elaborates further on this). Second, the very simple deterministic procedure of Theorem 3 of Foster and Hart (2023)—which forecasts the running average action in the reference forecaster’s bin—is shown to be proper-calibeating for every Lipschitz proper scoring rule (which we term “proper-Li-calibeating”). We further show that this result does not hold for all bounded proper scoring rules. Third, we establish the existence of a deterministic procedure that is simultaneously proper-Li-calibeating and continuously proper-calibrated; this is obtained via an approximate decomposition of the score (that generalizes from the quadratic case; see Appendix A.7 in Foster and Hart 2026). All three results extend to the simultaneous proper-calibeating of multiple reference forecasters, by cross-classifying across all of their bins. Decision-making under uncertainty. Forecasts are commonly used in decision-making under uncertainty, where the probabilities of the various states of nature are unknown (see Section 6 ). The forecast is utilized to make a decision that optimizes the decision-maker’s expected utility. This process induces a scoring rule, where the loss is taken as the realized disutility of the decision that is optimal with respect to the forecast. The scoring rule is proper: when the forecast is “correct,” i.e., equal to the true distribution, expected utility is maximized and expected loss is minimized. We show that, in a sequential decision-making setting, the calibration score of