Beyond the Last Layer: Multi-Layer Representation Fusion for Visual Tokenization
Xuanyu Zhu, Yan Bai, Yang Shi, Yihang Lou, Yuanxing Zhang Jing Jin Yuan Zhou, [0.2cm] Peking University · 2026
Read Beyond the Last Layer: Multi-Layer Representation Fusion for Visual Tokenization by Xuanyu Zhu, Yan Bai, Yang Shi, Yihang Lou, Yuanxing Zhang Jing Jin Yuan Zhou, [0.2cm] Peking University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10780.txt.
Introduction
Xuanyu Zhu and colleagues give DRoRAE, a depth-routed form autoencoder joining multi-layer traits for visual tokenization. Low-level visual bits live on in the last layer as faint traces after step-by-step sense lifts, while lower layers hold them with far more truth.
DRoRAE joins all encoder layers at once through energy-capped routing and step-by-step fixes, making a rich latent that fits a frozen pretrained decoder. A three-phase split training plan first learns the join under the frozen decoder's implicit spread cap, then tunes the decoder to use the rich form fully. On ImageNet-256, DRoRAE cuts rebuild FID by about half and lifts making FID, with gains carrying to text-to-image making. The authors find a log-linear growth law between join size and rebuild quality. It marks form richness as a growth scale that can be foreseen for visual token tools.
Catalog record
| CREATOR | Xuanyu Zhu, Yan Bai, Yang Shi, Yihang Lou, Yuanxing Zhang Jing Jin Yuan Zhou, [0.2cm] Peking University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46608 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10780
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond the Last Layer: Multi-Layer Representation Fusion for Visual Tokenization Xuanyu Zhu, Yan Bai, Yang Shi, Yihang Lou, Yuanxing Zhang Jing Jin Yuan Zhou, [0.2cm] Peking University Abstract Representation autoencoders that reuse frozen pretrained vision encoders as visual tokenizers have achieved strong reconstruction and generation quality. However, existing methods universally extract features from only the last encoder layer, discarding the rich hierarchical information distributed across intermediate layers. We show that low-level visual details survive in the last layer merely as attenuated residuals after multiple layers of semantic abstraction, and that explicitly fusing multi-layer features can substantially recover this lost information. We propose DRoRAE ( D epth- Ro uted R epresentation A uto E ncoder), a lightweight fusion module that adaptively aggregates all encoder layers via energy-constrained routing and incremental correction, producing an enriched latent compatible with a frozen pretrained decoder. A three-phase decoupled training strategy first learns the fusion under the implicit distributional constraint of the frozen decoder, then fine-tunes the decoder to fully exploit the enriched representation. On ImageNet-256, DRoRAE reduces rFID from 0.57 to 0.29 and improves generation FID from 1.74 to 1.65 (with AutoGuidance), with gains also transferring to text-to-image synthesis. Furthermore, we uncover a log-linear scaling law ( R 2 = 0.86 R^{2}{=}0.86 ) between fusion capacity and reconstruction quality, identifying representation richness as a new, predictably scalable dimension for visual tokenizers analogous to vocabulary size in NLP. 1 Introduction The image tokenizer maps pixels into a compact latent space and defines the quality ceiling of modern visual generation systems 21 ; 18 . A recent line of work 30 ; 29 ; 33 ; 10 has demonstrated that leveraging pretrained vision foundation models (VFMs) such as DINOv2 17 as the tokenizer’s latent space yields substantial improvements in both reconstruction fidelity and downstream generation quality over conventional learned tokenizers trained from scratch. Despite their success, all existing VFM-based tokenizers share a common design choice: they extract features exclusively from the last layer of the encoder. While this is the natural output of any vision model, last-layer features are primarily optimized for high-level semantics rather than low-level visual details such as textures, edges, and color gradients. Recent analysis 25 reveals that low-level information survives in the last layer only as a structural consequence of residual connections, a passive pathway that becomes increasingly lossy as each successive layer superimposes semantic transformations onto the residual stream. Shallower layers, by contrast, retain this information with far greater fidelity (Figure 1 ), yet single-layer tokenizers discard it entirely. This observation suggests a natural direction: explicitly fusing features from multiple depth levels to assemble a latent representation richer than any single layer can provide. Moreover, multi-layer fusion introduces two quantifiable capacity axes, the number of fused layers and the per-layer expert capacity, which together define the representation richness of the tokenizer. An analogous concept has been explored for NLP tokenizers 12 , where increasing the input vocabulary size (representation richness in the text domain) yields predictable, log-linear improvements in downstream loss. Whether such a scaling law also exists for visual tokenizers remains an open question. Realizing multi-layer fusion in practice, however, requires addressing two challenges. (1) Content-adaptive fusion. Feature statistics vary substantially across layers, and the optimal combination is spatially dependent: textured regions benefit from shallow features while semantically uniform regions do not. Naive aggregation collapses to deep-layer dominance or introduces noise from irrelevant layers. (2) Generation compatibility. In representation-based tokenizers, the decoder is trained to invert a specific output distribution. Multi-layer fusion inevitably shifts this distribution; if unconstrained, the downstream diffusion model can no longer generate latents that the decoder reliably decodes, degrading generation even when reconstruction improves. We propose DRoRAE (Depth-Routed Representation AutoEncoder), a lightweight fusion module of \sim 29M parameters (Figure 2 ) that addresses both challenges. For content-adaptive fusion, we design an energy-constrained routing mechanism. Per-layer expert MLPs project heterogeneous layer features onto a common scale, and a learned router assigns per-token aggregation weights, including negative weights for active suppression, without the winner-take-all behavior of softmax normalization. For generation compatibility, we adopt an incremental correction formulation that injects the fused representation as a bounded perturbation to the original last-layer output. This is combined with a three-phase decoupled training strategy in which the fusion module first learns under the implicit distributional constraint of a frozen decoder, preventing arbitrary drift, and only then is the decoder fine-tuned to fully exploit the enriched latent. On ImageNet-256, DRoRAE reduces rFID from 0.57 to 0.29 and improves class-conditional generation (gFID with AutoGuidance: 1.74 \to 1.65), with gains also transferring to text-to-image synthesis. We further observe that reconstruction quality improves log-linearly with fusion module capacity ( R^{2}{=}0.86 ), confirming that an analogous scaling law holds for visual tokenizers: representation richness , jointly determined by the number of fused layers and the per-layer expert capacity, is a predictably scalable dimension paralleling vocabulary size in NLP 12 . Figure 1: Motivation. Existing representation autoencoders extract only last-layer features, where low-level details are progressively diluted by semantic transformations. DRoRAE fuses features across all layers to assemble a richer latent per spatial token. Our contributions are as follows: • We identify the single-layer information bottleneck in representation autoencoders and propose DRoRAE, a depth-routed fusion module that enriches the tokenizer latent while preserving generation compatibility through energy-constrained routing, incremental correction, and decoupled training. • DRoRAE consistently improves reconstruction (rFID: 0.57 \to 0.29), class-conditional generation (gFID w/ AG: 1.74 \to 1.65), and text-to-image synthesis on ImageNet-256, validating multi-layer fusion as a practical upgrade for representation-based tokenizers. • We conduct systematic scaling experiments across two axes, expert capacity and number of fused layers, and observe that both follow the same log-linear scaling law. This establishes representation richness as a new, predictably scalable dimension for visual tokenizers. 2 Related Work 2.1 Image Tokenizers for Latent Generation The unified model explores the relationship between understanding and generation. Image tokenizers compress images into compact latent representations on which generative models operate. Early approaches learn both the encoder and decoder from scratch. VQGAN 8 combines discrete codebooks with adversarial training; SD-VAE 21 employs a KL-regularized continuous latent space and has become the backbone tokenizer for latent diffusion models 18 ; 16 . While these learned tokenizers achieve reasonable reconstruction, their latent spaces lack explicit semantic structure, forcing the downstream diffusion model to jointly discover both visual and semantic patterns from pixel-level supervision alone. A recent line of work addresses this by aligning the latent space to pretrained visual representations. REPA 30 adds a representation alignment loss during diffusion training while retaining the original SD-VAE encoder. VA-VAE 29 distills DINOv2 17 features into a learned VAE encoder, obtaining a latent space that is both reconstructive and semantically structured. RAE 33 takes this idea further by directly freezing the pretrained DINOv2 encoder as the tokenizer and training only a decoder, so that the latent space is the pretrained representation itself. RPiAE 10 extends RAE with a principal-component-based channel expansion to decouple spatial and channel information. These representation-based tokenizers simultaneously achieve state-of-the-art reconstruction fidelity and downstream generation quality, demonstrating that the latent space structure inherited from pretrained models substantially benefits generative modeling. However, all existing representation-based tokenizers share an inherited design choice. They extract features exclusively from the