Boosting Generative Image Modeling via Joint Image-Feature Synthesis
Theodoros Kouzelis, Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, Nikos Komodakis · 2025
Read Boosting Generative Image Modeling via Joint Image-Feature Synthesis by Theodoros Kouzelis, Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, Nikos Komodakis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16064.txt.
Introduction
Theodoros Kouzelis and co-authors present a way to train image generators that also learn what their pictures mean, in a 2025 paper. Standard latent diffusion models make fine images but learn little about the content of a scene. The team ties generation to feature learning by having one model produce both the image code and a set of meaning features at once.
The features come from a strong self-taught vision encoder, and the model learns to denoise the image code and the features side by side. The change needs only small edits to a standard transformer design. As a bonus, the learned features can steer generation at run time, a trick the authors call representation guidance. In both prompted and unprompted tests, images came out better and training converged faster than with plain diffusion.
Catalog record
| CREATOR | Theodoros Kouzelis, Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, Nikos Komodakis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-16064
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Boosting Generative Image Modeling via Joint Image-Feature Synthesis Theodoros Kouzelis, Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, Nikos Komodakis Abstract Latent diffusion models (LDMs) dominate high-quality image generation, yet integrating representation learning with generative modeling remains a challenge. We introduce a novel generative image modeling framework that seamlessly bridges this gap by leveraging a diffusion model to jointly model low-level image latents (from a variational autoencoder) and high-level semantic features (from a pretrained self-supervised encoder like DINO). Our latent-semantic diffusion approach learns to generate coherent image–feature pairs from pure noise, significantly enhancing both generative quality and training efficiency, all while requiring only minimal modifications to standard Diffusion Transformer architectures. By eliminating the need for complex distillation objectives, our unified design simplifies training and unlocks a powerful new inference strategy: Representation Guidance, which leverages learned semantics to steer and refine image generation. Evaluated in both conditional and unconditional settings, our method delivers substantial improvements in image quality and training convergence speed, establishing a new direction for representation-aware generative modeling. Project page and code: https://representationdiffusion.github.io/ Figure 1: ReDi : Our generative image modeling framework bridges the gap between generative modeling and representation learning by leveraging a diffusion model that jointly captures low-level image details (via VAE latents) and high-level semantic features (via DINOv2 ). Trained to generate coherent image–feature pairs from pure noise, this unified latent-semantic dual-space diffusion approach significantly boosts both generative quality and training convergence speed. 1 Introduction 100K 400K 1M 2M 4M 7M 3.3 5 10 20 30 40 \times 23 speed-up Training Iteration FID DiT-XL/2 DiT-XL/2+ReDi SiT-XL/2 SiT-XL/2+ReDi 100K 400K 1M 2M 4M 3.3 5 10 20 \times 6 \times 11 Training Iteration FID SiT-XL/2+REPA SiT-XL/2+ReDi SiT-XL/2+REPA+ ReDi Figure 2: Accelerated Training. Generative performance curves on Imagenet 256\times 256 without Classifier-Free Guidance. Left : Our ReDi accelerates convergence of DiT-XL/2 and SiT-XL/2 by approximately \times 23 . Right: ReDi converges \times 6 faster than REPA . When applied on top of REPA delivers a \times 11 speed-up. Latent diffusion models ( LDMs ) ( Rombach et al., 2022 ) have emerged as a leading approach for high-quality image synthesis, achieving state-of-the-art results ( Rombach et al., 2022 ; Peebles & Xie, 2023 ; Ma et al., 2024 ) . These models operate in two stages: first, a variational autoencoder ( VAE ) compresses images into a compact latent representation ( Rombach et al., 2022 ) ; second, a diffusion model learns the distribution of these latents, capturing their underlying structure. Leveraging their intermediate features, pretrained LDMs have shown promise for various scene understanding tasks, including classification ( Mukhopadhyay et al., 2023 ) , pose estimation ( Gong et al., 2023 ) , and segmentation ( Li et al., 2023b ; Liu et al., 2023 ; Delatolas et al., 2025 ) . However, their discriminative capabilities typically underperform specialized (self-supervised) representation learning approaches like masking-based ( He et al., 2022 ) , contrastive ( Chen et al., 2020 ) , self-distillation ( Caron et al., 2021 ) , or vision-language contrastive ( Radford et al., 2021a ) methods. This limitation stems from the inherent tension in LDM training - the need to maintain precise low-level reconstruction while simultaneously developing semantically meaningful representations. This observation raises a fundamental question: How can we leverage representation learning to enhance generative modeling? Recent work by Yu et al. (2025) ( REPA ) demonstrates that improving the semantic quality of diffusion features through distillation of pretrained self-supervised representations leads to better generation quality and faster convergence. Their results establish a clear connection between representation learning and generative performance. Motivated by these insights, we investigate whether a more effective approach to leveraging representation learning can further enhance image generation performance. In this work, we contend that the answer is yes : rather than aligning diffusion features with external representations via distillation, we propose to jointly model both images (specifically their VAE latents) and their high-level semantic features extracted from a pretrained vision encoder (e.g., DINOv2 ( Oquab et al., 2024 ) ) within the same diffusion process. Formally, as shown in Figure 1 , we define the forward diffusion process as q(\mathbf{x}_{t},\mathbf{z}_{t}|\mathbf{x}_{t-1},\mathbf{z}_{t-1}) for t=1,...,T , where \mathbf{x}_{0}=\mathbf{x} and \mathbf{z}_{0}=\mathbf{z} are the clean VAE latents and semantic features, respectively. The reverse process p_{\theta}(\mathbf{x}_{t-1},\mathbf{z}_{t-1}|\mathbf{x}_{t},\mathbf{z}_{t}) learns to gradually denoise both modalities from Gaussian noise. This joint modeling approach forces the diffusion model to explicitly learn the joint distribution of both precise low-level ( VAE ) and high-level semantic ( DINOv2 ) features. We implement this approach, called ReDi ( Re presentation Di ffusion), within the DiT ( Peebles & Xie, 2023 ) and SiT ( Ma et al., 2024 ) frameworks with minimal modifications to their transformer architecture: we apply standard diffusion noise to both representations, combine them into a single set of tokens, and train the standard diffusion transformer architecture to denoise both components simultaneously. Compared to REPA , our joint modeling approach offers three key advantages. First, the diffusion process explicitly models both low-level and semantic features, enabling direct integration of these complementary representations. Second, our method simplifies training by eliminating the need for additional distillation objectives. Finally, during inference, our unified approach enables Representation Guidance - where the model uses its learned semantic understanding to iteratively refine generated images, improving quality in both conditional and unconditional generation. Our contributions can be summarized as follows: 1. We propose ReDi , a novel and effective method that jointly models image-compressed latents and semantically rich representations within the diffusion process, significantly improving image synthesis performance. 2. We provide a concrete implementation of our approach for both diffusion ( DiT ) and flow-matching ( SiT ) frameworks, leveraging DINOv2 ( Oquab et al., 2024 ) as the source of high-quality semantic representations. 3. We also introduce Representation Guidance , which leverages the model’s semantic predictions during inference to refine outputs, further enhancing image generation quality. 4. We demonstrate that our approach boosts performance in both conditional and unconditional generation, while significantly accelerating convergence (see Figure 2 ). 2 Related work Representation Learning. Various approaches aim to learn meaningful representations for downstream tasks, with self-supervised learning emerging as one of the most promising directions. Early approaches employed pretext tasks such as predicting image patch permutations ( Noroozi & Favaro, 2016 ) or rotation angles ( Gidaris et al., 2018 ) , while more recent methods utilize contrastive learning ( Chen et al., 2020 ; Van den Oord et al., 2018 ; Misra & Maaten, 2020 ) , clustering-based objectives ( Caron et al., 2020 ; Caron et al., 2018 ; Caron et al., 2019 ) , and self-distillation techniques ( Grill et al., 2020 ; Chen & He, 2021 ; Caron et al., 2021 ; Gidaris et al., 2021 ) . The introduction of transformers enabled Masked Image Modeling (MIM), introduced by BEiT ( Bao et al., 2022 ) and evolved through SimMIM ( Xie et al., 2022 ) , MAE He et al. (2022) , AttMask ( Kakogeorgiou et al., 2022 ) , iBOT ( Zhou et al., 2022 ) , and MOCA ( Gidaris et al., 2024 ) , with DINOv2 ( Oquab et al., 2024 ) achieving state-of-the-art performance through scaled models and datasets. Separately, contrastive vision-language pretraining,