Socrates Loss: Unifying Confidence Calibration and Classification by Leveraging the Unknown

Sandra Gómez-Gálvez [email protected] of Auckland Tobias Olenyi University of AucklandTechnical University of Munich Gillian Dobbie University of Auckland, Katerina Taškova [email protected] of Auckland · 2026

Read Socrates Loss: Unifying Confidence Calibration and Classification by Leveraging the Unknown by Sandra Gómez-Gálvez [email protected] of Auckland Tobias Olenyi University of AucklandTechnical University of Munich Gillian Dobbie University of Auckland, Katerina Taškova [email protected] of Auckland (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12245.txt.

Introduction

Gómez-Gálvez and three colleagues propose Socrates Loss, a single loss function that unifies classification with confidence calibration by adding an auxiliary unknown class and a dynamic uncertainty penalty. Two-phase methods gain accuracy at the cost of unstable training and poor calibration, while single-loss methods stay stable but underperform. Socrates Loss optimizes both goals at once without scheduled losses, with theory guarding against miscalibration and overfitting, and it often converges faster.

Catalog record

CREATORSandra Gómez-Gálvez [email protected] of Auckland Tobias Olenyi University of AucklandTechnical University of Munich Gillian Dobbie University of Auckland, Katerina Taškova [email protected] of Auckland
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE113392
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.12245

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Socrates Loss: Unifying Confidence Calibration and Classification by Leveraging the Unknown Sandra Gómez-Gálvez [email protected] of Auckland Tobias Olenyi University of AucklandTechnical University of Munich Gillian Dobbie University of Auckland, Katerina Taškova [email protected] of Auckland Abstract Deep neural networks, despite their high accuracy, often exhibit poor confidence calibration, limiting their reliability in high-stakes applications. Current ad-hoc confidence calibration methods attempt to fix this during training but face a fundamental trade-off: two-phase training methods achieve strong classification performance at the cost of training instability and poorer confidence calibration, while single-loss methods are stable but underperform in classification. This paper addresses and mitigates this stability-performance trade-off. We propose Socrates Loss , a novel, unified loss function that explicitly leverages uncertainty by incorporating an auxiliary unknown class, whose predictions directly influence the loss function and a dynamic uncertainty penalty. This unified objective allows the model to be optimized for both classification and confidence calibration simultaneously, without the instability of complex, scheduled losses. We provide theoretical guarantees that our method regularizes the model to prevent miscalibration and overfitting. Across four benchmark datasets and multiple architectures, our comprehensive experiments demonstrate that Socrates Loss consistently improves training stability while achieving more favorable accuracy-calibration trade-off, often converging faster than existing methods. 1 Introduction Deep neural networks (DNNs) have achieved remarkable performance across diverse domains, yet their deployment in high-stakes applications remains constrained by their ability to reliably operate in real-world conditions. Critical applications, such as medical diagnosis ( 10 ) , nuclear security ( 2 ) , and biosecurity ( 23 ) , require models that provide both accurate, reliable, and trustworthy predictions. One important aspect of a reliable model is its ability to effectively represent its own uncertainty. To achieve this, a model needs to be calibrated , i.e., its predictive confidence matches the true likelihood of correctness ( 11 ) . In classification settings, confidence scores from softmax layers commonly serve as a proxy for uncertainty ( 1 ) , where, ideally, predictions made with 90 \% confidence should be correct 90 \% of the time. However, while modern DNNs are widely known to suffer from systematic overconfidence (i.e., higher confidence than the actual accuracy) ( 11 ) , recent evidence shows that they may also be underconfident or exhibit mixed calibration patterns depending on architectural choices ( 24 ) . As we also demonstrate in this work, modifying the loss function alone can produce underconfident or mixed-calibration models. This gap between predicted confidence and actual accuracy undermines the reliability needed for high-stakes applications. Research in confidence calibration has emerged to address this challenge by both quantifying miscalibration and developing methods to improve the alignment between predictive confidence and accuracy. Current alignment methods broadly fall into two categories: post-hoc methods ( 7 ; 8 ; 25 ; 28 ; 33 ; 18 ; 41 ; 42 ) that adjust confidences after training without modifying model parameters, and ad-hoc methods ( 13 ; 19 ; 21 ; 26 ; 27 ; 32 ; 36 ) that integrate calibration during training. While post-hoc methods offer simplicity and speed, they face significant limitations, including hyperparameter tuning and additional data, which is problematic when data is scarce ( 3 ; 16 ; 37 ) . We identify a critical logical gap: Many widely used pre-trained models are optimized for accuracy and are not well-calibrated, often lacking available data for post-hoc calibration, limiting their reliability in downstream tasks. Furthermore, post-hoc methods are fundamentally incompatible with knowledge-transfer paradigms such as transfer learning, where calibrated representations must be embedded within the model weights themselves ( 40 ) . The impact of performing transfer learning with calibrated versus uncalibrated pre-trained models remains unexplored. In contrast, ad-hoc methods integrate calibration into the training, creating their own challenges ( 20 ) , such as longer development times and a trade-off in accuracy to achieve better calibration. Current methods typically act as regularizers using data augmentation ( 13 ; 36 ) , adapted loss functions ( 21 ; 26 ; 27 ; 32 ) , or modifying the model architecture ( 19 ) . Through empirical evaluation, we identify that existing ad-hoc confidence calibration methods face a fundamental trade-off: methods that combine losses in a two-phase training achieve strong classification performance but suffer from training instability and worse calibration, while single-loss methods train stably and achieve strong calibration performance but at the cost of lower classification performance. Reliability diagrams show that final models in both cases are underconfident or overconfident, with calibration fluctuating across epochs and dependent on dataset and architecture. These insights motivate our research question: Can we design an ad-hoc calibration method based on a single, easily implementable, loss function that trains a single model while promoting training stability and maintaining calibration and classification performance across diverse datasets and architectures? In exploring the connection between ad-hoc confidence calibration and ad-hoc selective classification due to their regularization nature ( 8 ) , we analyze Self-Adaptive Training (SAT) ( 15 ) and uncover a relationship between the average confidence assigned to the unknown class and confidence calibration. While correlation does not imply causation, this observation led us to extend our research question: Does explicit uncertainty modeling through an unknown class mitigate training instability while maintaining calibration and classification performance? To this end, our contributions are threefold: • We propose Socrates Loss 1 1 1 Socrates Loss was named after the philosopher Socrates and his famous quote I know that I know nothing . , a novel ad-hoc and easy-to-implement confidence calibration method that mitigates the training stability-performance trade-off by unifying classification and confidence calibration objectives through explicit uncertainty modeling via an unknown class. By incorporating predictions for this class into the loss function via a dynamic uncertainty penalty, Socrates Loss penalizes the model for failing to recognize its own uncertainty. The loss further emphasizes hard-to-classify instances and leverages previous predictions to adaptively promote training convergence and stability. • We provide theoretical guarantees showing that Socrates Loss regularizes the weights of the network and acts as a regularized upper bound on the Kullback-Leibler divergence, thereby preventing overfitting and miscalibration while maintaining training stability. • We demonstrate empirically that Socrates Loss improves confidence calibration performance, achieving a better training stability and accuracy–calibration trade-off across multiple benchmarks, architectures, and transfer learning scenarios, without compromising accuracy. 2 Background and Related Work The level of confidence calibration can be assessed both visually and quantitatively (for formal definitions and extended discussion see Appendix B ). A widely used visualization method is the reliability diagram ( 31 ) , which plots the expected sample accuracy as a function of prediction confidence at a given training epoch, following several binning strategies ( 6 ; 11 ; 30 ) . We adopt the approach in 11 , which groups confidences into M interval bins of size 1/M . While reliability diagrams offer visual insights, evaluating confidence calibration only at the final epoch is insufficient, particularly for ad-hoc confidence calibration methods that influence training dynamics. Following 21 , we argue that ad-hoc confidence calibration should be assessed across training epochs. To jointly analyze classification and confidence calibration across epochs, we suggest the use of Pareto plots, providing a more holistic visualization of the performance trade-off. Quantitatively, the

Read the complete text