Revisiting Reweighted Risk for Calibration: AURC, Focal and Inverse Focal Loss

Han Zhou, Sebastian G. Gruber, Teodora Popordanoska, Matthew B. Blaschko · 2025

Read Revisiting Reweighted Risk for Calibration: AURC, Focal and Inverse Focal Loss by Han Zhou, Sebastian G. Gruber, Teodora Popordanoska, Matthew B. Blaschko (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23463.txt.

Introduction

Han Zhou, Sebastian Gruber, Teodora Popordanoska, and Matthew Blaschko revisit a family of training losses that reweight examples, and tie them to how honest a model's confidence is. Their paper appeared in two thousand twenty-five. Losses like focal loss were built for calibration, yet their link to calibration error stayed murky. Selective classification, where models may abstain, turns out to be the bridge.

The team shows that shrinking calibration error and doing well at selective prediction are close cousins. Improving selective risk where confidence is low naturally polishes calibration. Their proposed loss shares focal loss's reweighting spirit but lets users choose the confidence score. A bin-based trick makes training efficient. Across data sets and architectures, the method matched strong rivals. The work gives calibration a firmer theoretical footing.

Catalog record

CREATORHan Zhou, Sebastian G. Gruber, Teodora Popordanoska, Matthew B. Blaschko
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23463

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Revisiting Reweighted Risk for Calibration: AURC, Focal and Inverse Focal Loss Han Zhou, Sebastian G. Gruber, Teodora Popordanoska, Matthew B. Blaschko Abstract Several variants of reweighted risk functionals, such as focal loss, inverse focal loss, and the Area Under the Risk–Coverage Curve (AURC), have been proposed for improving model calibration; yet their theoretical connections to calibration errors remain under-explored. In this paper, we revisit a broad class of weighted risk functions and find a principled connection between calibration error and selective classification. We show that minimizing calibration error is closely linked to the selective classification paradigm and demonstrate that optimizing selective risk in low-confidence regions naturally improves calibration. Our proposed loss shares a similar reweighting strategy with dual focal loss but offers greater flexibility through the choice of confidence score functions (CSFs). Furthermore, our approach utilizes a bin-based cumulative distribution function (CDF) approximation, enabling efficient gradient-based optimization with 𝒪 ⁡ ( n ​ M ) \mathcal{O}(nM) complexity for n n samples and M M bins. Empirical evaluations demonstrate that our method achieves competitive calibration performance across a range of datasets and model architectures. 1 Introduction Over the past decade, advances in deep learning have enabled neural networks to achieve, and in some cases surpass, human-level performance on tasks ranging from image recognition ( Haggenmüller et al., 2021 ) to natural language understanding ( Achiam et al., 2023 ; Liu et al., 2024a ) . However, achieving high accuracy does not necessarily ensure that the predicted confidence scores align with the true probabilities of outcomes, a phenomenon known as miscalibration ( Guo et al., 2017 ) . Miscalibration can lead to critical errors in high‐stakes applications where trustworthy uncertainty estimates are essential. To address and quantify this gap, the research community has introduced a range of calibration error metrics ( Brier, 1950 ; Guo et al., 2017 ; Kumar et al., 2019 ; Popordanoska et al., 2022 ) to evaluate the discrepancy between predicted confidences and observed outcome frequencies. Recent work has explored both trainable ( Karandikar et al., 2021 ; Kumar et al., 2018 ; Krishnan and Tickoo, 2020 ; Bohdal et al., 2023 ; Popordanoska et al., 2022 ) and post-hoc ( Guo et al., 2017 ; Ding et al., 2021 ; Kull et al., 2019 ; Wenger et al., 2020 ; Ma and Blaschko, 2021 ) approaches for neural network calibration, achieving impressive reductions in terms of those empirical calibration errors. Among trainable methods, focal loss ( Lin et al., 2017 ) —which emphasizes hard examples during training—has gained popularity for its calibration benefits. Conversely, inverse focal loss ( Wang et al., 2021 ) , which focuses on the easy examples more, has also been shown to improve calibration in certain settings. This raises a key question: Which weighting scheme is most effective for improving model calibration? We address this question by examining the relationship between selective classification ( Geifman and El-Yaniv, 2017 ) and calibration error. We revisit the exisiting theoretical results that links calibration error minimization to the principles of selective classification, showing that optimal calibration strategies naturally emerge from selective weighting based on confidence rankings. Building on this connection, we propose a loss function similar to the AURC ( Geifman et al., 2019 ) , but focused specifically on selective risk in low-confidence regions that require careful calibration. Unlike AURC, our method employs a bin-based CDF approximation that enables efficient gradient-based optimization with \mathcal{O}(nM) complexity. This adaptation ensures computational scalability while maintaining differentiability with respect to any choice of CSFs. Our main contributions are summarized as follows: • We revisit the calibration error minimization and its relation to the “abstain” strategy at the core of selective classification. • Motivated by this connection, we propose a computationally efficient, differentiable loss function based on selective classification principles, using a bin-based CDF approximation that avoids expensive sorting operations while maintaining gradient flow for any CSF choice. • Through extensive experiments, we show that our proposed method delivers competitive calibration performance across various datasets and network architectures, surpassing the majority of existing calibration-aware loss functions. 2 Related work Post-hoc calibration methods address miscalibration by learning a mapping function that transforms model outputs into calibrated posterior probabilities. Approaches like Platt scaling ( Platt and others, 1999 ) , temperature scaling (TS) ( Guo et al., 2017 ) , and local temperature scaling ( Ding et al., 2021 ) typically preserve classification accuracy by introducing a scalar temperature parameter. In contrast, other probabilistic methods—including Beta calibration ( Kull et al., 2017 ) , Dirichlet calibration ( Kull et al., 2019 ) , Gaussian processes ( Wenger et al., 2020 ) , and the Concrete distribution ( Esaki et al., 2024 ) —may achieve better calibration but often at the cost of reduced classification accuracy. Notably, Dirichlet calibration generalizes Beta calibration to multi-class settings. ( Ma and Blaschko, 2021 ) and ( Liu et al., 2024b ) proposed post-hoc calibration methods that adjust predicted probabilities based on correctness, using binary classifiers to either distinguish between correct and incorrect predictions or to directly modulate confidence accordingly. However, these approaches often involve a trade-off between calibration and predictive accuracy. We aim to develop a trainable calibration method that improves calibration performance without significantly compromising predictive accuracy. Trainable calibration methods seek to enhance the calibration of deep neural networks by explicitly modifying the training objective. One class of approaches introduces differentiable surrogate losses that directly approximate the expected calibration error, as demonstrated in ( Karandikar et al., 2021 ; Kumar et al., 2018 ; Krishnan and Tickoo, 2020 ; Bohdal et al., 2023 ; Popordanoska et al., 2022 ) . Popordanoska et al. (2022) proposed \mathrm{ECE}^{\mathrm{KDE}} , a differentiable estimator of calibration error, and introduced the KDE-XE objective, which augments cross-entropy (XE) with an ECE-KDE regularizer to balance accuracy and calibration. Alternatively, several studies replace the standard cross-entropy loss with calibration-aware objectives, including mean squared error loss ( Hui, 2020 ) , inverse focal loss ( Wang et al., 2021 ) , and focal loss ( Lin et al., 2017 ) . Focal loss, in particular, has been empirically shown to improve calibration by mitigating overconfident predictions ( Mukhoti et al., 2020 ; Charoenphakdee et al., 2021 ; Komisarenko and Kull, 2024 ) . Nevertheless, Wang et al. (2021) argue that the regularization effect of focal loss might suppress crucial information about sample difficulty, which in turn limits the effectiveness of subsequent post-hoc calibration techniques. To address this trade-off, several variants of focal loss have been proposed, such as Adaptive Focal Loss ( Ghosh et al., 2022 ) and Dual Focal Loss ( Tao et al., 2023 ) . Distinct from these prior approaches, we propose a loss function based on the selective classification framework. While selective classification is not originally intended for calibration, we show that its underlying principle is fundamentally aligned with minimizing calibration error. Selective classification augments a standard classifier with a reject option, enabling an explicit trade-off between selective risk and coverage. This technique mitigates selective risk by withholding low-confidence predictions ( Geifman and El-Yaniv, 2017 ; Geifman et al., 2019 ; Ding et al., 2020 ; Galil et al., 2023 ) . The Area Under the Risk-Coverage Curve (AURC) and

Read the complete text