Robust Representation Consistency Model via Contrastive Denoising

Jiachen Lei, Julius Berner, Jiongxiao Wang, Zhongzhu Chen, Zhongjia Ba, Kui Ren, Jun Zhu, Anima Anandkumar · 2025

Read Robust Representation Consistency Model via Contrastive Denoising by Jiachen Lei, Julius Berner, Jiongxiao Wang, Zhongzhu Chen, Zhongjia Ba, Kui Ren, Jun Zhu, Anima Anandkumar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13094.txt.

Introduction

Jiachen Lei and co-authors make image classifiers harder to fool, in a 2025 paper. Attackers can tweak an image slightly so a model misreads it. One defense smooths the model's answers with noise and can certify a safe radius, but it runs slowly and weakens at larger attacks. Diffusion models help clean the noise, at heavy cost. The team reframes the cleaning task in a hidden feature space.

Their model learns to keep features steady along the diffusion path, by matching nearby points in time. After fine-tuning, one prediction does both the cleaning and the classifying, which cuts run cost sharply. On standard image sets, the method beat diffusion-based defenses on certified accuracy at every radius tested. The gain grew at larger radii, while inference cost fell by a wide factor.

Catalog record

CREATORJiachen Lei, Julius Berner, Jiongxiao Wang, Zhongzhu Chen, Zhongjia Ba, Kui Ren, Jun Zhu, Anima Anandkumar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-13094

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Robust Representation Consistency Model via Contrastive Denoising Jiachen Lei, Julius Berner, Jiongxiao Wang, Zhongzhu Chen, Zhongjia Ba, Kui Ren, Jun Zhu, Anima Anandkumar Abstract Robustness is essential for deep neural networks, especially in security-sensitive applications. To this end, randomized smoothing provides theoretical guarantees for certifying robustness against adversarial perturbations. Recently, diffusion models have been successfully employed for randomized smoothing to purify noise-perturbed samples before making predictions with a standard classifier. While these methods excel at small perturbation radii, they struggle with larger perturbations and incur a significant computational overhead during inference compared to classical methods. To address this, we reformulate the generative modeling task along the diffusion trajectories in pixel space as a discriminative task in the latent space. Specifically, we use instance discrimination to achieve consistent representations along the trajectories by aligning temporally adjacent points. After fine-tuning based on the learned representations, our model enables implicit denoising-then-classification via a single prediction, substantially reducing inference costs. We conduct extensive experiments on various datasets and achieve state-of-the-art performance with minimal computation budget during inference. For example, our method outperforms the certified accuracy of diffusion-based methods on ImageNet across all perturbation radii by 5.3% on average, with up to 11.6% at larger radii, while reducing inference costs by 85 × \times on average. Codes are available at: https://github.com/jiachenlei/rRCM . 1 Introduction Figure 1: Performance vs. Inference Latency. . Marker sizes correspond to relative model sizes. Deep neural networks (DNNs) have achieved unprecedented success in various visual applications. Yet, they are still vulnerable to small adversarial perturbations. This imposes a threat to the deployment of DNNs in real-world systems, in particular for security-critical scenarios, such as human face identification and autonomous driving. To counteract this issue, numerous efforts in terms of both empirical and certified defenses have been made to improve the robustness of DNNs against adversarial perturbations. While empirical defenses train DNNs to be robust to known adversarial examples ( Mądry et al., 2017 ) , they can be easily compromised by employing stronger or unknown perturbations. In contrast, to end the mouse-and-cat game of iterative improvements of attacks and defenses, certified defenses focus on developing strategies that provide certifiable and formal robustness guarantees. However, this also makes the design of such certified defenses much more challenging. (a) (b) Figure 2: Illustration of our pre-training method and model forward pass. (a) Pre-training method. After pre-training, the projector is discarded, and the encoder is fine-tuned alongside a linear head using class labels. (b) Model forward pass. Noticeably, during certification, our model serves as the base classifier (as described in Section 2 ) and predicts the class label of each perturbed sample in a single forward pass. Among certified defenses, randomized smoothing with Gaussian noise ( Cohen et al., 2019 ) is currently considered the “gold standard” , providing a scalable way of certifying model robustness against adversarial perturbations with bounded \ell_{2} -norm. To date, various randomized smoothing-based methods have been proposed ( Jeong & Shin, 2020 ; Carlini et al., 2022 ) . Among these works, diffusion model-based methods ( Carlini et al., 2022 ) stand out with superior performance by integrating trained diffusion models into randomized smoothing. They first apply the denoising process of a diffusion model to remove Gaussian noise added to images. Then, using the purified samples, they predict the class label using a separate classifier. For brevity, we refer to these approaches as diffusion-based methods in the following discussions. Despite the success, there exists a gap between achieving low latency and strong performance for diffusion-based methods. To maintain a competitive performance, they either increase the number of sampling steps ( Xiao et al., 2022 ) and/or implement majority voting during class prediction ( Xiao et al., 2022 ; Zhang et al., 2023 ) , suffering from even higher computational demands during inference (e.g., as high as 52 minutes 1 1 1 On a single A800 GPU, we report the time by certifying DensePure ( Xiao et al., 2022 ) on a single image from ImageNet with N = 10k smoothing noises. ). Furthermore, while leveraging the basic denoising property of diffusion models, previous approaches achieve consistent prediction across perturbed and clean samples through two independent models, resulting in a cumbersome prediction process and increased model maintenance overhead. We show that the framework of diffusion models itself already offers an effective solution in this regard: it establishes a unique connection between perturbed and clean samples along the trajectories of the probability flow (PF) of the denoising process. In this context, perturbed samples can be seen as points on the same trajectory of the denoising process but at different time steps, with the clean sample being the initial point 2 2 2 As is common practice, we call the clean image on the reverse sampling trajectory the “initial point” , as opposed to the one sampled from the Gaussian prior at the beginning of the reverse sampling process. . This motivates our approach of directly optimizing for consistent semantics across noise-perturbed and clean samples on the same trajectory of the diffusion process, leading to a unified model that supports consistent one-step prediction. In contrast, classical methods ( Cohen et al., 2019 ; Jeong & Shin, 2020 ; Salman et al., 2019a ) train models directly on noisy samples, primarily relying on heuristic strategies. These approaches fail to thoroughly exploit the intrinsic relationships between noisy and clean images, limiting their potential to achieve higher levels of certified robustness. Our approach: We close the gap of diffusion-based methods in terms of the tradeoff between performance and efficiency. In particular, we achieve performance that is better than classical randomized smoothing methods at a fraction of the cost of existing diffusion-based methods. This is made possible by directly optimizing model robustness based on structured connections between clean and perturbed samples. With the above analysis, we reformulate model robustness against noise perturbations as consistency between predictions of clean and perturbed samples. Specifically, our framework decomposes the training into two stages: pre-training and fine-tuning. During pre-training, the model learns to align representations across points along the deterministic trajectory from the Gaussian prior to the data distribution. To accomplish this, we reformulate the original generative image denoising task into a discriminative task in latent space and propose to align the representations of temporally adjacent points on the same trajectory via pair-wise instance discrimination. Based on the learned consistent representations, the model is then fine-tuned in a supervised manner to predict class labels given perturbed samples as input. This integrates denoising and classification into a single model and enables one-shot image classification, which is key to lowering the computation cost during inference. We term our model Robust Representation Consistency Model (rRCM) . In our experiments, we demonstrate that our method improves the certified accuracy of Carlini et al. (2022) at all perturbation radii by 5.3% on average, with up to 11.6% at larger radius, while at the same time reducing the computation cost by 85 \times on average during inference. In comparison with classical methods ( Cohen et al., 2019 ; Jeong & Shin, 2020 ; Salman et al., 2019a ; Zhai et al., 2020 ; Jeong et al., 2021 ) , our rRCM model either matches or surpasses

Read the complete text