Exploring the Potential of Bilevel Optimization for Calibrating Neural Networks
Gabriele Sanguin, Arjun Pakrashi, Marco Viola, Francesco Rinaldi · 2025
Read Exploring the Potential of Bilevel Optimization for Calibrating Neural Networks by Gabriele Sanguin, Arjun Pakrashi, Marco Viola, Francesco Rinaldi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13113.txt.
Introduction
Gabriele Sanguin and co-authors use two-level optimization to teach neural networks honest confidence. Modern networks often sound sure when they should not. Their 2025 paper frames training as a bilevel problem, where an inner task fits the data and an outer task tunes how trust is scored. The network thus learns to grade its own calls while it learns the calls themselves.
Trials ran on toy shapes and on a blood alcohol dataset. The rival was isotonic regression, a standard fix applied after training. The bilevel route cut calibration error while keeping accuracy intact. The authors pitch the frame as a built-in cure, not a patch. Better-graded confidence matters wherever software calls feed real choices, from clinics to courts.
Catalog record
| CREATOR | Gabriele Sanguin, Arjun Pakrashi, Marco Viola, Francesco Rinaldi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13113
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring the Potential of Bilevel Optimization for Calibrating Neural Networks Gabriele Sanguin, Arjun Pakrashi, Marco Viola, Francesco Rinaldi Abstract Handling uncertainty is critical for ensuring reliable decision-making in intelligent systems. Modern neural networks are known to be poorly calibrated, resulting in predicted confidence scores that are difficult to use. This article explores improving confidence estimation and calibration through the application of bilevel optimization, a framework designed to solve hierarchical problems with interdependent optimization levels. A self-calibrating bilevel neural-network training approach is introduced to improve a model’s predicted confidence scores. The effectiveness of the proposed framework is analyzed using toy datasets, such as Blobs and Spirals, as well as more practical simulated datasets, such as Blood Alcohol Concentration (BAC). It is compared with a well-known and widely used calibration strategy, isotonic regression. The reported experimental results reveal that the proposed bilevel optimization approach reduces the calibration error while preserving accuracy. *1 keywords Bilevel optimization ,confidence ,calibration ,neural networks † † copyrightyear: 2024 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: AICS’24: 32nd Irish Conference on Artificial Intelligence and Cognitive Science, December 09–10, 2024, Dublin, Ireland † † email: [email protected] † † address: Dipartimento di Matematica, Università degli Studi di Padova, Via 8 Febbraio 2, 35122 Padova (Italy) † † email: [email protected] † † address: School of Computer Science, University College Dublin, Belfield, Dublin 4 (Ireland) † † email: [email protected] † † address: School of Mathematical Sciences, Dublin City University, Collins Avenue Ext, Dublin 9 (Ireland) † † email: [email protected] † † corresp: Corresponding author. 1 Introduction Machine learning classification algorithms with increasingly higher discriminative power, especially neural networks, have rapidly developed in the last decade. Such advanced models are generally supposed to help humans make decisions by assisting. Although these models have a high discriminative power, sometimes they will predict something completely incorrect with a fairly high confidence score. This creates a huge problem in highly regulated and sensitive real-world applications (e.g. medical field, autonomous vehicles, healthcare diagnostics, financial forecasting, etc.) Minderer et al. (2021) . Therefore, it is important for these models to provide a meaningful confidence, based on which it can say ”I don’t know”, when they are not confident enough, so that the human expert can inspect, and make further decisions. This is sometimes called learning to reject or abstention Zhang et al. (2023) ; Hendrickx et al. (2024) . Confidence is a probabilistic score that a model assigns to each prediction of a data point, and it determines how certain the model is about the prediction. One straightforward approach to use such a confidence score to understand if the model is confident enough or not is to define an interval, a rejection window , within which if the prediction falls, it will be marked as reject . It is however hard to fix such a confidence window to decide the rejection window for such models, especially modern neural network models, because such models are known to have poor model confidence calibration Minderer et al. (2021) ; Guo et al. (2017) . Confidence calibration is the process of adjusting the confidence scores to better align with the actual likelihood of correctness. Well-calibrated confidence is crucial for effective decision making and is important for the interpretation of the model, since humans have a natural cognitive intuition for probabilities Cosmides and Tooby (1996) . Accurate confidence scores make it easier for users to comprehend how confident the model was during prediction and to establish trust on the decisions being made. Moreover, accurate confidence scores are essential if such a rejection window needs to be defined by a human expert. There are two types of confidence calibration. Firstly, post-calibration , where the output scores/probabilities of the main model are re-adjusted by another external calibration model. For such methods, the main model does not need to be modified. In Guo et al. (2017) , the authors have demonstrated how well-known post-calibration methods can be used to calibrate existing models. On the other hand, self-calibration method are algorithms which integrate the confidence calibration process into the model training itself. These methods aim to ensure that the model’s probabilities are calibrated during the training phase, without the need for a separate calibration step. The objective of this article is to present an initial study on whether it is possible to self-calibrate neural network models by exploiting bilevel optimization (BO), a mathematical framework specifically designed to solve hierarchical two-level decision-making optimization problems. BO has recently gained importance in machine learning, particularly in hyperparameter optimization and metalearning Pedregosa (2016) ; Franceschi et al. (2017) ; Franceschi et al. (2018) . The choice of such a framework is obvious where the inner-level optimization problem tackles the training of the model, while the outer-level optimization problem addresses the model confidence calibration. To the best of our knowledge, in the context of uncertainty scores, the only work in the literature that uses BO is Jain and Shenoy (2022) . Here, we define a BO framework to train two different architectures, one for classification and one for the uncertainty score (which is then tested as a rejection function), thus leading to a significant increase of the parameters to be trained. Then we focus on the study on the selective potential of such a score. The work presented in the current article is the first of its kind, which is quite distinct from the one in Jain and Shenoy (2022) . The objective of the current work is to propose a BO framework to train a single self-calibrated deep neural network, BO4SC , and provide an initial, but crucial analysis of the applicability of the approach. The article is organized as follows. Section 2 discusses the mathematical foundations of confidence estimation, calibration methods, and bilevel optimization (BO). Section 3 introduces BO4SC, a BO framework for confidence estimation in neural networks. In Section 4 the initial experiments and analysis are shown and discussed. Finally, Section 5 concludes the article. 2 Theoretical Background This section will briefly introduce and discuss the relevant parts of confidence estimation, model calibration evaluation, calibration methods, and bilevel optimization. 2.1 Confidence Estimation For a machine learning model, the standard process (referred to as standard method later in Section 4 ) to accurately predict an output involves learning a mapping function that can generalize well from training data to unseen samples. Let \{X,Y\}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{n} be a labeled dataset, where \mathbf{x}_{i} represents a data point and y_{i}\in\{1,2,\dots,C\} is its corresponding class label, where C being the total number of classes and n the number of data points. The objective is usually to learn a function f that maps an unseen data point \mathbf{x}_{t} to a predicted output \hat{y}_{t}=f(\mathbf{x}_{t}) . This function f can be efficiently and effectively trained by minimizing the empirical loss over all training data. Confidence estimation involves assigning a probabilistic score to each prediction which reflects the model’s certainty about the predicted output. Confidence is a score function \hat{p_{i}}=g(f,\mathbf{x}_{i},\hat{y_{i}}), which measures the likelihood of the prediction \hat{y}_{i} being correct given the features \mathbf{x}_{i} and the classifier f . Ideally, the confidence score should be continuous and fall within the range [0,1]