Understanding Model Calibration - A gentle introduction and visual exploration of calibration and the expected calibration error (ECE)
Maja Pavlovic · 2025
Read Understanding Model Calibration - A gentle introduction and visual exploration of calibration and the expected calibration error (ECE) by Maja Pavlovic (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19047.txt.
Introduction
Maja Pavlovic, writing alone, gives a gentle guide to model calibration in this 2025 piece. A model is calibrated when its stated confidence matches how often it is right. That match is a core part of whether a model can be trusted.
Pavlovic walks through the standard meaning of calibration with pictures and plain steps. The guide then covers expected calibration error, a common score for this trait. It shows flaws in that score and how those flaws led to new notions of calibration. Each new notion needs its own score. The piece is a primer, not a deep review or a guide to fixing models.
Catalog record
| CREATOR | Maja Pavlovic |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-19047
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding Model Calibration - A gentle introduction and visual exploration of calibration and the expected calibration error (ECE) Maja Pavlovic Abstract To be considered reliable, a model must be calibrated so that its confidence in each decision closely reflects its true outcome. In this blogpost we’ll take a look at the most commonly used definition for calibration and then dive into a frequently used evaluation measure for model calibration. We’ll then cover some of the drawbacks of this measure and how these surfaced the need for additional notions of calibration, which require their own new evaluation measures. This post is not intended to be an in-depth dissection of all works on calibration, nor does it focus on how to calibrate models. Instead, it is meant to provide a gentle introduction to the different notions and their evaluation measures as well as to re-highlight some issues with a measure that is still widely used to evaluate calibration. 1 What is Calibration? Calibration makes sure that a model’s estimated probabilities match real-world likelihoods. For example, if a weather forecasting model predicts a 70% chance of rain on several days, then roughly 70% of those days should actually be rainy for the model to be considered well calibrated ( Dawid, 1982 ; DeGroot & Fienberg, 1983 ) . This makes model predictions more reliable and trustworthy , which makes calibration relevant for many applications across various domains. Figure 1: Reliability Diagram Now, what calibration means more precisely depends on the specific definition being considered. We will have a look at the most common notion in machine learning (ML) formalised in ( Guo et al., 2017 ) and termed confidence calibration in ( Kull et al., 2019 ) . But first, let’s define a bit of formal notation for this blog. In this blogpost we consider a classification task with K possible classes, with labels Y\in\{1,...,K\} and a classification model \hat{p}:\mathscr{X}\rightarrow\Delta^{K} , that takes inputs in \mathscr{X} (e.g. an image or text) and returns a probability vector as its output. \Delta^{K} refers to the K-simplex, which just means that the elements of the output vector must sum to 1 and that each estimated probability in the vector is between 0 & 1. These individual probabilities ( or confidences ) indicate how likely an input belongs to each of the K classes. Figure 2: Notation - input example sourced from ( Uma et al., 2021 ) 1.1 (Confidence) Calibration A model is considered confidence-calibrated if, for all confidences c, the model is correct c proportion of the time: \mathbb{P}(Y=\text{arg max}(\hat{p}(X))\;|\;\text{max}(\hat{p}(X))=c)=c\;\;\>\>\forall c\in[0,1]\;, where (X,Y) is a datapoint and \hat{p}:\mathscr{X}\rightarrow\Delta^{K} returns a probability vector as its output. This definition of calibration, ensures that the model’s final predictions align with their observed accuracy at that confidence level ( Guo et al., 2017 ) . The left chart below visualises the perfectly calibrated outcome (green diagonal line) for all confidences using a binned reliability diagram ( Guo et al., 2017 ) . On the right hand side it shows two examples for a specific confidence level across 10 samples. Figure 3: Confidence Calibration For simplification, we assume that we only have 3 classes as in figure 2 and we zoom into confidence c=0.7 , see image above. Let’s assume we have 10 inputs here whose most confident prediction ( max ) equals 0.7 . If the model correctly classifies 7 out of 10 predictions ( true ), it is considered calibrated at confidence level 0.7 . For the model to be fully calibrated this has to hold across all confidence levels from 0 to 1. At the same level c=0.7 , a model would be considered miscalibrated if it makes only 4 correct predictions. 2 Evaluating Calibration - Expected Calibration Error (ECE) One widely used evaluation measure for confidence calibration is the Expected Calibration Error (ECE) ( Naeini et al., 2015 ; Guo et al., 2017 ) . ECE measures how well a model’s estimated probabilities match the observed probabilities by taking a weighted average over the absolute difference between average accuracy ( acc ) and average confidence ( conf ). The measure involves splitting all n datapoints into M equally spaced bins: ECE=\sum_{m=1}^{M}\frac{\mathopen{|}B_{m}\mathclose{|}}{n}\mathopen{|}acc(B_{m})-conf(B_{m})\mathclose{|}, where B is used for representing ”bins” and m for the bin number, while acc and conf are: {acc(B_{m})=\frac{1}{\mathopen{|}B_{m}\mathclose{|}}\sum_{i\in B_{m}}{\displaystyle\bm{1}}(\hat{y}_{i}=y_{i})\;\>\text{\&}\;\>conf(B_{m})=\frac{1}{\mathopen{|}B_{m}\mathclose{|}}\sum_{i\in B_{m}}\hat{p}(x_{i})} \hat{y}_{i} is the model’s predicted class ( arg max ) for sample i and y_{i} is the true label for sample i . {\displaystyle\bm{1}} is an indicator function, meaning when the predicted label \hat{y}_{i} equals the true label y_{i} it evaluates to 1, otherwise 0. Let’s look at an example, which will clarify acc , conf and the whole binning approach in a visual step-by-step manner. 2.1 ECE - Visual Step by Step Example In the image below, we can see that we have 9 samples indexed by i with estimated probabilities \hat{p}(x_{i}) (simplified as \hat{p}_{i} ) for class cat (C) , dog (D) or toad (T) . The final column shows the true class {y}_{i} and the penultimate column contains the predicted class \hat{y}_{i} . Figure 4: ECE - Toy Example Only the maximum probabilities, which determine the predicted label are used in ECE ( Guo et al., 2017 ) . Therefore, we will only bin samples based on the maximum probability across classes ( see left table in below image ). To keep the example simple we split the data into 5 equally spaced bins M=5 . If we now look at each sample’s maximum estimated probability, we can group it into one of the 5 bins ( see right side of image below ). Figure 5: Table 2 & Binning Diagram We still need to determine if the predicted class is correct or not to be able to determine the average accuracy per bin. If the model predicts the class correctly (i.e. y_{i}=\hat{y}_{i} ), the prediction is highlighted in green; incorrect predictions are marked in red: Figure 6: Table 3 & Binning Diagram We now have visualised all the information needed for ECE and will briefly run through how to calculate the values for bin 5 ( B_{5} ). The other bins then simply follow the same process, see below. Figure 7: Table 4 & Example for bin 5 We can get the empirical probability of a sample falling into B_{5} , by assessing how many out of all 9 samples fall into B_{5} , see \mathbf{(\;1\;)} . We then get the average accuracy for B_{5} , see \mathbf{(\;2\;)} and lastly the average estimated probability for B_{5} , see \mathbf{(\;3\;)} . Repeat this for all bins and in our small example of 9 samples we end up with an ECE of 0.10445 . A perfectly calibrated model would have an ECE of 0. 2.1.1 Expected Calibration Error Drawbacks The images of binning above provide a visual guide of how ECE could result in very different values if we used more bins or perhaps binned the same number of items instead of using equal bin widths. Such and more drawbacks of ECE have been highlighted by several works early on ( Kumar et al., 2018 ; Nixon et al., 2019 ; Gupta et al., 2021 ; Zhang et al., 2020 ; Roelofs