Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation

Boyu Han Qianqian Xu Shilong Bao Zhiyong Yang, Ruochen Cui Xilin Zhao Qingming Huang · 2026

Read Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation by Boyu Han Qianqian Xu Shilong Bao Zhiyong Yang, Ruochen Cui Xilin Zhao Qingming Huang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04803.txt.

Introduction

Han and colleagues improve CLIP visual pictures with contrast-guided diffusion rebuild. CLIP's visual encoder limits downstream scores through weak class split and fine-grained sight. Recent spread methods base image rebuild on CLIP visual tokens, yet may hurt class split. The paper folds contrast signals into diffusion-based rebuild for fuller pictures. A naive mix suffers slope conflict and weak scores. Diffusion Contrastive Reconstruction, DCR, unifies the learning to balance tuning.

Catalog record

CREATORBoyu Han Qianqian Xu Shilong Bao Zhiyong Yang, Ruochen Cui Xilin Zhao Qingming Huang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85483
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.04803

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation Boyu Han Qianqian Xu Shilong Bao Zhiyong Yang, Ruochen Cui Xilin Zhao Qingming Huang Abstract The limited understanding capacity of the visual encoder in Contrastive Language-Image Pre-training (CLIP) has become a key bottleneck for downstream performance. This capacity includes both Discriminative Ability (D-Ability), which reflects class separability, and Detail Perceptual Ability (P-Ability), which focuses on fine-grained visual cues. Recent solutions use diffusion models to enhance representations by conditioning image reconstruction on CLIP visual tokens. We argue that such paradigms may compromise D-Ability and therefore fail to effectively address CLIP’s representation limitations. To address this, we integrate contrastive signals into diffusion-based reconstruction to pursue more comprehensive visual representations. We begin with a straightforward design that augments the diffusion process with contrastive learning on input images. However, empirical results show that the naive combination suffers from gradient conflict and yields suboptimal performance. To balance the optimization, we introduce the Diffusion Contrastive Reconstruction (DCR) , which unifies the learning objective. The key idea is to inject contrastive signals derived from each reconstructed image, rather than from the original input, into the diffusion process. Our theoretical analysis shows that the DCR loss can jointly optimize D-Ability and P-Ability. Extensive experiments across various benchmarks and multi-modal large language models validate the effectiveness of our method. The code is available at https://github.com/boyuh/DCR . 1 Introduction Figure 1 : (a) Contrastive learning for D-Ability . (b) Reconstructive learning for P-Ability . (c) Our Diffusion Contrastive Reconstruction (DCR) for harmonizing D-Ability and P-Ability . (d) Performance overview of DCR and other methods on the OpenAI CLIP ViT-L@224 backbone. Contrastive Language-Image Pre-training (CLIP) 65 is widely used as a frozen module to provide visual representations for various downstream tasks 93 ; 104 ; 96 . However, its self-supervised text-image alignment is relatively coarse, which limits the model’s understanding capability and has become a major bottleneck to broader deployment 31 ; 80 . The CLIP’s understanding capacity has two complementary aspects 30 ; 60 . The first is Discriminative Ability (D-Ability) , which requires the model to separate categories with clear boundaries, keeping samples from the same class close while pushing different classes apart 38 ; 65 . It is crucial for recognition 89 ; 83 ; 54 ; 79 ; 97 ; 21 , retrieval 56 ; 96 ; 50 , and open-vocabulary transfer 105 ; 93 ; 47 ; 78 ; 27 . The second is Detail Perceptual Ability (P-Ability) , which preserves information about color, direction, quantity, and structure 75 . It supports multimodal question answering 28 , instruction following 48 , and vision-centric reasoning 74 ; 36 . Traditional methods fine-tune the vision encoder to learn better representations under CLIP feature supervision. As shown in Fig. 1 (a), they often rely on various forms of contrastive learning 105 ; 82 ; 8 ; 51 . With the rise of diffusion models 98 ; 24 ; 100 , recent studies turn to generative feedback for stronger enhancement 76 ; 77 ; 57 ; 42 . As shown in Fig. 1 (b), the diffusion model takes CLIP visual features as conditions and reconstructs the image. When the reconstructed image matches the original, the visual representation has captured all the information present in the input. This process builds the reconstruction loss \mathcal{L}_{\text{rec}} through a Mean-Square Error (MSE) objective, thereby improving P-Ability . However, due to the lack of class supervision, it brings little gain or may even harm D-Ability , as shown in Fig. 1 (d). A natural question arises: Can we achieve a representation that balances discriminative ability and detailed perceptual ability? Motivated by the fact that contrastive learning increases class separability, we propose a straightforward variant that adds a contrastive loss \mathcal{L}_{\text{con}} on top of the reconstruction objective to strengthen D-Ability . However, this naive design does not achieve a good balance in practice, as shown in Fig. 1 (d). The challenge lies in the fact that the two objectives focus on different aspects: one targets CLIP feature separability, while the other relies on image-level reconstruction consistency. This mismatch causes the easier objective to dominate the other. We examine the behavior of the contrastive loss \mathcal{L}_{\text{con}} and the reconstruction loss \mathcal{L}_{\text{rec}} during training and find that a gradient conflict indeed exists between them. Fig. 2 (c) shows that their gradient directions often diverge, and the conflict grows over time. As a result, \mathcal{L}_{\text{con}} gradually dominates while \mathcal{L}_{\text{rec}} stalls, which leads to unstable convergence, as illustrated in Fig. 2 (a)(b). A more detailed analysis is provided in Section 4.1 . We address this challenge with a new framework called Diffusion Contrastive Reconstruction (DCR) . The key improvement is to replace image-level consistency with contrastive supervision on the reconstructed images , as shown in Fig. 1 (c). Specifically, for each image, the anchor is the reconstruction from its own visual features. The positive is the reconstruction conditioned on its augmented view, and the negatives are reconstructions conditioned on features from other images in the mini-batch. A contrastive loss is then applied to these samples, with the diffusion model remaining frozen. This design leads to a single optimization objective and naturally avoids gradient conflicts. We further provide a theoretical analysis under a mild assumption and show that our method can satisfy contrastive constraints (Thm. 1 ) and reconstruction consistency (Thm. 2 ) at the same time. Finally, comprehensive empirical studies consistently speak to the efficacy of our method. Our main contributions are summarized as follows: • This paper rethinks diffusion-based reconstruction methods and reveals that they improve the P-Ability of visual representations, while D-Ability shows little gain or may even degrade. • We find that directly adding contrastive learning to supplement D-Ability leads to gradient conflicts. Therefore, this paper proposes the DCR loss, which uses a single objective to avoid gradient conflict, achieving a balance between D-Ability and P-Ability . • Our theoretical analysis and extensive experiments across 6 CLIP backbones and diverse vision benchmarks demonstrate the effectiveness of our method in enhancing visual representations. (a) Contrastive loss \mathcal{L}_{\text{con}} . (b) Reconstruction loss \mathcal{L}_{\text{rec}} . (c) Cosine similarity between gradient \mathbf{g}_{\text{con}} and \mathbf{g}_{\text{rec}} . Figure 2 : Training results of linearly combining contrastive learning and reconstructive learning (Naive Method) on the OpenAI CLIP ViT-L@224 backbone. The contrastive loss \mathcal{L}_{\text{con}} dominates the gradients, and most training steps exhibit gradient conflicts. 2 Related Work Vision Encoders. Vision encoders serve as the visual backbone of MLLMs 48 ; 49 ; 26 ; 81 , transforming raw images into high-level feature representations. Among various architectures 65 ; 4 ; 62 ; 70 , CLIP 65 is one of the most widely adopted models. It also provides transferable representations for a wide range of tasks, including detection 71 ; 58 ; 101 ; 29 ; 1 ; 92 ; 91 ; 90 ; 18 ; 55 , segmentation 103 ; 88 ; 85 ; 44 ; 20 , video understanding 86 ; 72 ; 46 ; 2 , and generation 66 ; 59 ; 63 ; 12 ; 19 ; 39 ; 40 ; 64 ; 14 ; 15 ; 35 ; 34 . Recent studies focus on enhancing CLIP’s understanding capability to improve downstream

Read the complete text