Margin and Consistency Supervision for Calibrated and Robust Vision Models
Salim Khazem · 2026
Read Margin and Consistency Supervision for Calibrated and Robust Vision Models by Salim Khazem (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05812.txt.
Introduction
Khazem presents Margin and Consistency Supervision, a design-free rule-setting frame for tuned vision models. Deep vision sorters score highly while staying poorly tuned and fragile under shifts. The method adds to cross-entropy a hinge-squared margin penalty holding a gap, plus a same-ness regularizer for shaken views. Theory links larger margins and lower response to spread and a toughness radius scaling with the ratio. Across base nets spanning CNN, transformer, it improves tuning and toughness while keeping top score, with no extra data or design changes.
Catalog record
| CREATOR | Salim Khazem |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34248 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.05812
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Margin and Consistency Supervision for Calibrated and Robust Vision Models Salim Khazem Abstract Deep vision classifiers often achieve high accuracy while remaining poorly calibrated and fragile under small distribution shifts. We present Margin and Consistency Supervision ( MaCS ), a simple, architecture-agnostic regularization framework that jointly enforces logit-space separation and local prediction stability. MaCS augments cross-entropy with (i) a hinge-squared margin penalty that enforces a target logit gap between the correct class and the strongest competitor, and (ii) a consistency regularizer that minimizes the KL divergence between predictions on clean inputs and mildly perturbed views. We provide a unifying theoretical analysis showing that increasing classification margin while reducing local sensitivity formalized via a Lipschitz-type stability proxy yields improved generalization guarantees and a provable robustness radius bound scaling with the margin-to-sensitivity ratio. Across several image classification benchmarks and several backbones spanning CNNs and Vision Transformers, MaCS consistently improves calibration (lower ECE and NLL) and robustness to common corruptions while preserving or improving top-1 accuracy. Our approach requires no additional data, no architectural changes, and negligible inference overhead, making it an effective drop-in replacement for standard training objectives. Figure 1 : Per-dataset model curves across methods. Each line corresponds to a model and traces accuracy across training objectives, highlighting method-specific gains. 1 Introduction Deep neural networks achieve impressive accuracy on vision classification benchmarks, yet they frequently exhibit problematic behaviors: overconfident predictions on ambiguous or out-of-distribution inputs ( 10 ) , brittleness to small input perturbations ( 36 ) , and poor generalization under distribution shift ( 13 ) . These issues limit deployment in safety-critical applications where reliable uncertainty quantification is essential. Classical statistical learning theory provides a compelling lens for understanding these failures: classifiers with larger margins the gap between the score for the correct class and the highest-scoring alternative generalize better ( 2 ; 1 ) . Intuitively, larger margins provide a “buffer zone” that makes predictions more robust to noise and distribution shift. Similarly, locally stable classifiers that produce consistent predictions under small input changes are inherently more robust ( 5 ) . Motivated by these insights, we propose Margin and Consistency Supervision ( MaCS ), a unified training objective that encourages both properties. MaCS augments cross-entropy with two complementary terms: 1. Margin Loss: A hinge-squared penalty encouraging the logit margin \gamma(x)=f_{y}(x)-\max_{j\neq y}f_{j}(x) to exceed a target threshold \Delta , promoting well-separated representations. 2. Consistency Loss: A KL-divergence term measuring prediction stability under mild perturbations (noise and blur), encouraging smooth decision boundaries. The combined objective is simply: \mathcal{L}_{\textit{MaCS}}=\mathcal{L}_{\text{CE}}+\lambda_{m}\mathcal{L}_{\text{margin}}+\lambda_{c}\mathcal{L}_{\text{cons}}, (1) which can be applied to any classifier without architectural changes. Contributions. Our main contributions are: (i) We propose MaCS , a simple, architecture-agnostic regularization framework combining margin maximization and consistency regularization. (ii) We provide a unifying theoretical analysis connecting margin and local sensitivity to generalization guarantees and a provable robustness radius bound. (iii) We conduct extensive experiments on 6 datasets and 7 architectures spanning CNNs and Vision Transformers, showing consistent improvements in calibration, robustness, and accuracy. We release a fully reproducible codebase requiring no additional data or architectural changes. Code is released and available at : https://github.com/salimkhazem/marco . 2 Related Work Margin-Based Learning. The connection between large margins and generalization has deep roots in statistical learning theory ( 41 ; 2 ) . For neural networks, margin-based objectives have been explored primarily in metric learning and face recognition ( 23 ; 6 ; 42 ) , where angular margins in embedding space improve discriminability. 8 study margin maximization for general classification but focus on adversarial robustness rather than calibration. Label smoothing ( 35 ) and focal loss ( 22 ) implicitly affect margins but do not explicitly enforce a target gap. Consistency Regularization. Consistency-based methods have proven highly effective for semi-supervised learning ( 3 ; 34 ) , where they enforce that predictions remain stable under data augmentation. Similar principles appear in self-training ( 44 ) , domain adaptation ( 9 ) , and knowledge distillation ( 16 ) . Mixup ( 47 ) and its variants encourage smooth interpolations between training examples. We adapt consistency regularization for fully-supervised learning, using it to encourage smooth decision boundaries via explicit KL-divergence penalties. Robustness and Calibration. Adversarial training ( 26 ) and certified defenses ( 5 ; 21 ) address worst-case robustness but often sacrifice clean accuracy. Calibration methods include post-hoc temperature scaling ( 10 ) , explicit calibration losses ( 20 ; 28 ) , and mixup variants ( 38 ) . MaCS addresses accuracy, robustness, and calibration jointly through a single training objective without requiring separate post-hoc calibration. Consistency-Based Robustness. Virtual Adversarial Training (VAT) ( 27 ) uses adversarial perturbations with KL consistency for semi-supervised learning. TRADES ( 46 ) extends this to adversarial robustness by balancing clean and adversarial accuracy. Recent work DiGN ( 39 ) applies Gaussian noise consistency in fully supervised settings with theoretical control over gradient norms. In contrast to worst-case (adversarial) perturbations, MaCS uses mild perturbations to encourage local smoothness while simultaneously enforcing explicit logit-margin constraints, targeting calibration and corruption robustness rather than adversarial robustness. Data Augmentation for Robustness. AugMix ( 14 ) combines diverse augmentation chains with Jensen-Shannon consistency and is a strong baseline for corruption robustness. Subsequent methods including DeepAugment ( 12 ) , PixMix ( 15 ) , and AugMax ( 43 ) further improve robustness through sophisticated augmentation strategies. IPMix ( 18 ) extends these ideas with improved augmentation pipelines. These methods are complementary to MaCS : we show that combining MaCS with AugMix yields additive improvements (Table 5 ), suggesting MaCS can serve as a base layer for augmentation-based robustness stacks. Notably, AugMix explicitly avoids overlap with CIFAR-C corruptions in its augmentation set, while MaCS uses simple noise/blur; we analyze this overlap in Section 5.4 . Certified and Lipschitz Methods. Certified Radius Maximization (CRM) ( 45 ) and Lipschitz-constrained networks ( 40 ) provide formal guarantees by controlling model smoothness. While MaCS does not provide certified robustness, our theoretical framework (Section 4 ) shares the intuition that controlling local sensitivity enables robustness; MaCS trades formal guarantees for practicality and preserved accuracy. 3 Method Figure 2 provides an overview of the MaCS training pipeline. Figure 2 : Overview of MaCS training. The model processes both clean input x and perturbed input \tilde{x}=T(x) . The total loss combines cross-entropy, a margin penalty encouraging \gamma(x)\geq\Delta , and a KL-based consistency term enforcing prediction stability. 3.1 Problem Setup Consider a classifier f(x;\theta):\mathbb{R}^{d}\to\mathbb{R}^{K} mapping inputs to logits over K classes. For a labeled example (x,y) with y\in\{1,\ldots,K\} , the standard cross-entropy loss is: \mathcal{L}_{\text{CE}}(x,y)=-\log\frac{\exp(f_{y}(x))}{\sum_{j=1}^{K}\exp(f_{j}(x))}. (2) 3.2 Margin Loss The logit margin for a sample (x,y) is the gap between the correct class logit and the strongest competitor: \gamma(x)=f_{y}(x)-\max_{j\neq y}f_{j}(x). (3) Positive margins indicate correct classification; larger margins suggest more confident and robust predictions. We encourage margins to exceed a target \Delta>0 using a hinge-squared penalty: \mathcal{L}_{\text{margin}}(x,y)=\max(0,\Delta-\gamma(x))^{2}. (4) The squared form provides smooth gradients and penalizes small margins more aggressively than a linear hinge. We use \Delta=1 in all experiments. 3.3 Consistency Loss To encourage local stability, we measure the KL divergence between predictions on clean and perturbed inputs: \mathcal{L}_{\text{cons}}(x)=D_{\text{KL}}\big(p(x)\,\|\,p(\tilde{x})\big), (5) where p(x)=\text{softmax}(f(x)) and \tilde{x}=T(x) is a perturbed version of x . The perturbation T applies