Your Classifier Can Do More: Towards Balancing the Gaps in Classification, Robustness, and Generation

Kaichao Jiang, He Wang, Xiaoshuai Hao, Xiulong Yang, Ajian Liu, Qi Chu, Yunfeng Diao, Richang Hong · 2025

Read Your Classifier Can Do More: Towards Balancing the Gaps in Classification, Robustness, and Generation by Kaichao Jiang, He Wang, Xiaoshuai Hao, Xiulong Yang, Ajian Liu, Qi Chu, Yunfeng Diao, Richang Hong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19459.txt.

Introduction

Kaichao Jiang and co-authors try to give one image model three talents at once: sharp labelling of images, toughness against attacks meant to fool it, and the power to make new images. Two existing kinds each miss part of the trio. Energy-based joint models label and generate well but break easily under attack. Models hardened by attack training resist well yet lose accuracy and cannot generate.

The team compares how each kind scores clean, attacked, and made-up samples on an energy scale. Each kind closes a different gap on that scale, which hints that lining up all three sample types could close the whole triangle. Their method, EB-JDAT, trains with an energy objective that pulls the three groups together. On CIFAR-10, CIFAR-100, and ImageNet subsets it reached top toughness while keeping near-original accuracy and image quality rivalling the generative models.

Catalog record

CREATORKaichao Jiang, He Wang, Xiaoshuai Hao, Xiulong Yang, Ajian Liu, Qi Chu, Yunfeng Diao, Richang Hong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19459

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Your Classifier Can Do More: Towards Balancing the Gaps in Classification, Robustness, and Generation Kaichao Jiang, He Wang, Xiaoshuai Hao, Xiulong Yang, Ajian Liu, Qi Chu, Yunfeng Diao, Richang Hong Abstract Joint Energy-based Models (JEMs) are well known for their ability to unify classification and generation within a single framework. Despite their promising generative and discriminative performance, their robustness remains far inferior to adversarial training (AT), which, conversely, achieves strong robustness but sacrifices clean accuracy and lacks generative ability. This inherent trilemma—balancing classification accuracy, robustness, and generative capability—raises a fundamental question: Can a single model achieve all three simultaneously? To answer this, we conduct a systematic energy landscape analysis of clean, adversarial, and generated samples across various JEM and AT variants. We observe that AT reduces the energy gap between clean and adversarial samples, while JEMs narrow the gap between clean and synthetic ones. This observation suggests a key insight: if the energy distributions of all three data types can be aligned, we might bridge their performance disparities. Building on this idea, we propose Energy-based Joint Distribution Adversarial Training (EB-JDAT), a unified generative-discriminative-robust framework that maximizes the joint probability of clean and adversarial distribution. EB-JDAT introduces a novel min–max energy optimization to explicitly aligning energies across clean, adversarial, and generated samples. Extensive experiments on CIFAR-10, CIFAR-100, and ImageNet subsets demonstrate that EB-JDAT achieves state-of-the-art robustness while maintaining near-original accuracy and competitive generation quality of JEMs, effectively achieving a new trade-off frontier between accuracy, robustness, and generation. The code is released at https://github.com/yujkc/EB-JDAT. 1 Introduction Figure 1 : Comparisons of SOTA AT-based methods on CIFAR-10 in terms of accuracy and robustness (AutoAttack). Our method achieves the best robustness while maintaining competitive standard accuracy. Recently, Joint Energy-based Models (JEMs) [ 16 , 47 , 49 ] have garnered significant attention for their ability to integrate generative modeling within a discriminative framework. This approach effectively bridges classification and generation from an energy-based perspective and exhibits surprising inherent robustness. However, JEMs still exhibit limited robustness compared to Adversarial Training (AT) [ 31 ] . On the other hand, AT [ 23 , 17 , 4 , 55 ] is widely recognized as the most effective method for improving the robustness of discriminative classifiers. However, it typically sacrifices accuracy on clean data and lacks generative capability [ 54 ] . Consequently, the inherent gap in classification, robustness and generation raises a natural question: Can a single model simultaneously achieve high classification accuracy, adversarial robustness, and generative performance? – a goal that has been rarely explored. Figure 2 : Distributions of the E_{\theta}(\textbf{x},y) of adversarial samples for PGD-20 vs. generated samples vs. clean samples on CIFAR-10. indicates generated samples, indicates adversarial samples, indicates clean samples. To answer this question, we investigate the underlying causes of the performance gap between AT [ 31 , 54 ] and JEMs [ 16 , 49 ] in terms of accuracy, generative capability, and adversarial robustness from an energy perspective. We compute the energy distributions of clean, adversarial, and generated samples, with results shown in Fig. 2 . Further, we compute the mean and variance of the one-to-one energy differences between clean and adversarial samples, as reported in Tab. 1 . The energy distributions shows that AT reduces the energy gap between clean and adversarial samples, thereby bringing robustness. In contrast, JEMs reduce the gap between clean and generated samples, which brings generative capability and higher accuracy. Based on these insights, we draw a key conclusion: if the energy distributions of three data types can be aligned, we might unify the strengths of AT and JEMs, resolving the inherent trilemma . Table 1 : Mean and variance of one‐to‐one energy differences between clean and adv. Results in bold indicate the best. Model Mean \downarrow Variance \downarrow PGD(%) \uparrow Standard 10.18 3.27 0.06 PGD-AT [ 31 ] 1.46 0.83 55.08 TRADES [ 54 ] 1.01 0.45 56.10 JEM [ 16 ] 5.93 7.30 7.28 SADAJEM [ 49 ] 2.66 3.83 43.64 EB-JDAT 0.89 0.31 66.12 Inspired by this idea, we align the energy distributions of clean, adversarial and generated data through a joint energy-based model of p_{\theta}(\mathbf{x},\tilde{\mathbf{x}},y) , with \mathbf{x} clean samples, \tilde{\mathbf{x}} adversarial examples, y class labels and \theta the model parameters. Through Bayesian decomposition, p_{\theta}(\mathbf{x},\tilde{\mathbf{x}},y) can be further factorized as the data distribution p_{\theta}(\mathbf{x}) , adversarial distribution p_{\theta}(\tilde{\mathbf{x}}\mid\mathbf{x}) and p_{\theta}(y\mid\tilde{\mathbf{x}},\mathbf{x}) . While p_{\theta}(\mathbf{x}) can be computed via approximated sampling methods [ 33 ] and p_{\theta}(y\mid\tilde{\mathbf{x}},\mathbf{x}) is simply a cross-entropy objective for robust classification, computing p_{\theta}(\tilde{\mathbf{x}}\mid\mathbf{x}) is not straightforward because the full adversarial distribution is not observed during the training phase. However, our key observation is that adversarial perturbations almost surely move samples out of the original data manifold, which lies in high-density areas, into low-density areas where the classifier is more likely to fail, as illustrated by the weakly robust classifiers JEM [ 16 ] and SADAJEM [ 49 ] in Fig. 2 . Motivated by this observation, we propose a new min-max energy optimization to approximate p_{\theta}(\tilde{\mathbf{x}}\mid\mathbf{x}) . During the sampling phase, we search for high-energy adversarial examples, then minimize the energy gap between adversarial and clean samples during the training phase, thereby pulling the adversarial samples back into low-energy regions. We name our proposed method as Energy-based Joint Distribution Adversarial Training, or EB-JDAT. Extensive experiments across multiple datasets and attacks show that EB-JDAT achieves 68.76%, 35.63% and 32.40% robustness (under AutoAttack [ 8 ] ) on CIFAR-10, CIFAR-100 and ImageNet subset, outperforming the SOTA AT methods by +10.78%, +4.70% and +7.88% respectively, while maintaining near original clean accuracy and reasonable generative capability, which is competitive to most advanced JEMs. 2 Related Work Adversarial Defense. Since the discovery of adversarial vulnerabilities in deep learning models [ 52 , 30 , 50 , 3 , 35 , 37 , 18 ] , numerous defense strategies have been proposed to mitigate this security challenge. Prominent approaches include input denoising [ 27 ] , gradient regularization [ 28 , 22 ] , defensive distillation [ 20 ] , and adversarial training (AT) [ 31 , 13 , 55 ] . Recent studies confirm that AT remains among the most effective defense methods [ 36 ] , significantly enhancing model robustness by incorporating adversarial examples into the training process. Notably, [ 31 ] first formulated AT as a saddle-point optimization problem (PGD), laying the foundational framework for subsequent robust training methodologies. As adversarial attack methods have evolved, various novel strategies have emerged, refining conventional AT methods [ 23 , 17 , 4 , 13 , 55 ] . However, many of these strategies prioritize robustness at the expense of clean accuracy. which is the principal evaluation metric in visual classification, highlighting an inherent trade-off in AT. To mitigate this issue, [ 54 ] proposed TRADES, which constrains the divergence in output probability between clean and adversarial examples. Additionally, [ 6 ] combined Jacobian regularization with model ensembling, introducing the Jacobians ensemble approach that effectively balances robustness and accuracy against universal adversarial perturbations. Furthermore, DUCAT [ 42 ] assigns a dummy class to each original class to absorb hard adversarial examples during training and remaps them back at inference to mitigating the trade-off. Although these approaches represent notable advancements, they still suffer from substantial

Read the complete text