Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders
Shengbang Tong, Boyang Zheng, Ziteng Wang, Bingda Tang, Nanye MaEllis Brown, Jihan Yang, Rob Fergus, Yann LeCun, Saining XieNew York University Website Code Models Data · 2026
Read Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders by Shengbang Tong, Boyang Zheng, Ziteng Wang, Bingda Tang, Nanye MaEllis Brown, Jihan Yang, Rob Fergus, Yann LeCun, Saining XieNew York University Website Code Models Data (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16208.txt.
Introduction
Shengbang Tong and coauthors scale image spread coders with a new code form. Their 2026 preprint uses Representation Autoencoders as the code base. Those coders train in a rich sense space of high size. On ImageNet they showed clear gains for spread models. The team now scales that path for text to image coders built as spread chains.
Scale tests check if rich codes keep their gains at big size. Rich codes may let the spread chain learn form fast. They may also lift the grade of pics from text prompts. The work adds a path to strong text to image tools. It tests code choice as a lever for scale.
Catalog record
| CREATOR | Shengbang Tong, Boyang Zheng, Ziteng Wang, Bingda Tang, Nanye MaEllis Brown, Jihan Yang, Rob Fergus, Yann LeCun, Saining XieNew York University Website Code Models Data |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-16208
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders Shengbang Tong, Boyang Zheng, Ziteng Wang, Bingda Tang, Nanye MaEllis Brown, Jihan Yang, Rob Fergus, Yann LeCun, Saining XieNew York University Website Code Models Data Abstract Representation Autoencoders (RAEs) have shown distinct advantages in diffusion modeling on ImageNet by training in high-dimensional semantic latent spaces. In this work, we investigate whether this framework can scale to large-scale, freeform text-to-image (T2I) generation. We first scale RAE decoders on the frozen representation encoder (SigLIP-2) beyond ImageNet by training on web, synthetic, and text-rendering data, finding that while scale improves general fidelity, targeted data composition is essential for specific domains like text. We then rigorously stress-test the RAE design choices originally proposed for ImageNet. Our analysis reveals that scaling simplifies the framework: while dimension-dependent noise scheduling remains critical, architectural complexities such as wide diffusion heads and noise-augmented decoding offer negligible benefits at scale Building on this simplified framework, we conduct a controlled comparison of RAE against the state-of-the-art FLUX VAE across diffusion transformer scales from 0.5B to 9.8B parameters. RAEs consistently outperform VAEs during pretraining across all model scales. Further, during finetuning on high-quality datasets, VAE-based models catastrophically overfit after 64 epochs, while RAE models remain stable through 256 epochs and achieve consistently better performance. Across all experiments, RAE-based diffusion models demonstrate faster convergence and better generation quality, establishing RAEs as a simpler and stronger foundation than VAEs for large-scale T2I generation. Additionally, because both visual understanding and generation can operate in a shared representation space, the multimodal model can directly reason over generated latents, opening new possibilities for unified models. * * footnotetext: Core contributor. 1 Introduction Figure 1 : RAE converges faster than VAE in text-to-image pretraining. We train Qwen-2.5 1.5B + DiT 2.4B models from scratch on both RAE (SigLIP-2) and VAE (FLUX) latent spaces for up to 60k iterations. RAE converges significantly faster than VAE on both GenEval (4.0×) and DPG-Bench (4.6×). Figure 2 : RAE decoders trained on more data (web, synthetic & text) generalize across domains. Decoders trained only on ImageNet reconstruct natural images well but struggle with text-rendering scenes (see second column). Adding web and text data greatly improves text reconstruction while maintaining natural-image quality. We also observe that both the language-supervised model and the SSL model learn representations suitable for reconstructing diverse images, including natural languages. Compared to proprietary VAEs, our RAE models achieve competitive overall fidelity. Diffusion-based generative modeling ( 19 ; 38 ; 47 ) has made rapid progress, giving rise to state-of-the-art systems across visual generative domains such as text-to-image generation ( 1 ; 46 ; 88 ) . A key factor in this success is the adoption of latent diffusion ( 65 ) , where generation occurs in a compact latent space encoded by a variational autoencoder (VAE) ( 43 ) , rather than directly in pixel space. In parallel with advances in generative modeling, visual representation learning has progressed through self-supervised learning (SSL) ( 12 ; 35 ; 34 ; 6 ) , language supervision ( 62 ; 98 ) , and their combinations ( 54 ; 83 ) . These models produce semantically structured, high-dimensional representations that generalize well across visual understanding tasks. Unlike VAE encoders, which compress images into low-dimensional latents, the representation encoders operate on high-dimensional latents that can capture much more semantically rich features. Such high-dimensional latents were previously considered too “abstract” for effective generative modeling ( 72 ; 92 ) , or outright intractable ( 48 ; 13 ) . However, a recent approach, Representation Autoencoder (RAE) ( 100 ) , has paved a path forward by training decoders on frozen representation encoders. RAE pairs a powerful frozen representation encoder with a lightweight trained decoder to reconstruct pixels from high-dimensional embeddings, enabling diffusion directly in this semantic latent space. In the highly controlled class-conditional ImageNet ( 18 ) setting, RAE demonstrates that diffusion in such frozen representation spaces can achieve more efficient and effective training than conventional VAE-based diffusion. However, ImageNet represents a best-case scenario: fixed resolution, curated content, and class-conditional generation. A critical question remains unanswered: can RAE truly scale to the complexities of freeform text-to-image generation? This setting involves broader visual diversity, open-ended compositions, and substantially larger models and compute—challenges for which high-dimensional latent diffusion remains unproven. In this work, we investigate whether RAEs can succeed at scale by training diffusion models for large-scale text-to-image (T2I) generation. We adopt SigLIP-2 ( 84 ) as the frozen representation encoder and use the MetaQuery framework ( 56 ) to train a unified T2I model, leveraging a powerful pretrained large language model (LLM) ( 61 ) . As a first step, we study decoder training beyond ImageNet supervision ( Section 2 ). Expanding from ImageNet to web-scale and synthetic aesthetic data yields only small gains on ImageNet itself, but provides moderate improvements on more diverse natural images such as YFCC ( 80 ) , showing that broader distributions enhance generalization. However, we find that text reconstruction requires targeted supervision: without text-specific data, the decoder fails to reproduce fine glyph details. Adding text-rendering data leads to substantial improvements, highlighting that data composition , not just scale, is crucial. Next, we analyze design choices in the RAE framework ( 100 ) and evaluate their importance under large-scale T2I training ( Section 3 ). We find that scale acts as a simplifier. Dimension-aware noise scheduling remains essential: removing the shift leads to substantially worse performance. The wide DDT head ( \text{DiT}^{\text{DH}} ) provides clear benefits for smaller backbones, but its advantage fades as Diffusion Transformers (DiT) scale to the billion-parameters. Finally, the effect of noise-augemented decoding is modest at T2I scale, with gains saturating quickly. We then systematically compare RAEs with SOTA VAEs under matched training conditions ( Section 4 ). We train DiTs from scratch following the conventional two-stage T2I setup ( 15 ; 60 ) : (i) large-scale pretraining with randomly initialized DiTs, and (ii) finetuning on smaller high-quality datasets. During pretraining, RAE-based models converge significantly faster and achieve higher performance on both GenEval and DPG-Bench. As shown in Fig. 1 , training a 1.5B LLM + 2.4B DiT with RAE (SigLIP-2) achieves a 4.0× speedup on GenEval and a 4.6× speedup on DPG-Bench compared to its VAE counterpart. This advantage is consistent across both language backbones (Qwen-2.5 2 1.5B–7B) and diffusion scales (DiT 0.5B–9.8B). In finetuning, RAE models continue to outperform their VAE counterparts and are less prone to overfitting. Finally, we examine unified models in which RAE enables understanding and generation to operate in the same high-dimensional semantic space ( Section 5 ). We find that adding generative training does not degrade understanding performance, and the choice of RAE vs. VAE in the generative path has little effect because both rely on the same frozen understanding encoder. Moreover, the shared latent space allows the LLM to process generated latents directly, without decoding back to pixels. We take a first exploratory step toward leveraging this property through latent-space test-time scaling, which proves both feasible and effective. Ultimately, we aim to convey one primary message: Representation Autoencoders provide a simpler and stronger foundation than VAEs for training large-scale text-to-image diffusion models. They offer a simple yet effective path to scaling