One Model, Many Budgets: Elastic Latent Interfaces for Diffusion Transformers
Moayed Haji-Ali Willi Menapace Ivan Skorokhodov Dogyun Park Anil Kag, [0.3em] Michael Vasilkovsky Sergey Tulyakov Vicente Ordonez Aliaksandr Siarohin, [1em] Rice University Snap Inc. Project Webpage: https://snap-research.github.io/elit · 2026
Read One Model, Many Budgets: Elastic Latent Interfaces for Diffusion Transformers by Moayed Haji-Ali Willi Menapace Ivan Skorokhodov Dogyun Park Anil Kag, [0.3em] Michael Vasilkovsky Sergey Tulyakov Vicente Ordonez Aliaksandr Siarohin, [1em] Rice University Snap Inc. Project Webpage: https://snap-research.github.io/elit (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12245.txt.
Introduction
Haji-Ali and co-authors give Diffusion Transformers an elastic latent interface for flexible compute budgets. Standard DiTs lock cost to image size and spread work evenly across spatial tokens. ELIT inserts a learnable latent sequence of variable length between the image tokens. Light cross-attention layers read and write between spatial tokens and latents, favoring key regions. Training drops tail latents at random, so early latents learn global structure and late ones add detail. At run time the latent count can flex to match the compute at hand.
Catalog record
| CREATOR | Moayed Haji-Ali Willi Menapace Ivan Skorokhodov Dogyun Park Anil Kag, [0.3em] Michael Vasilkovsky Sergey Tulyakov Vicente Ordonez Aliaksandr Siarohin, [1em] Rice University Snap Inc. Project Webpage: https://snap-research.github.io/elit |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67606 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.12245
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
One Model, Many Budgets: Elastic Latent Interfaces for Diffusion Transformers Moayed Haji-Ali Willi Menapace Ivan Skorokhodov Dogyun Park Anil Kag, [0.3em] Michael Vasilkovsky Sergey Tulyakov Vicente Ordonez Aliaksandr Siarohin, [1em] Rice University Snap Inc. Project Webpage: https://snap-research.github.io/elit Abstract $\dagger$ $\dagger$ footnotetext: Work partially done during an internship at Snap Inc. Diffusion transformers (DiTs) achieve high generative quality but lock FLOPs to image resolution, limiting principled latency-quality trade-offs, and allocate computation uniformly across input spatial tokens, wasting resource allocation to unimportant regions. We introduce Elastic Latent Interface Transformer (ELIT), a drop-in, DiT-compatible mechanism that decouples input image size from compute. Our approach inserts a latent interface, a learnable variable-length token sequence on which standard transformer blocks can operate. Lightweight Read and Write cross-attention layers move information between spatial tokens and latents and prioritize important input regions. By training with random dropping of tail latents, ELIT learns to produce importance-ordered representations with earlier latents capturing global structure while later ones contain information to refine details. At inference, the number of latents can be dynamically adjusted to match compute constraints. ELIT is deliberately minimal, adding two cross-attention layers while leaving the rectified flow objective and the DiT stack unchanged. Across datasets and architectures (DiT, U-ViT, HDiT, MM-DiT), ELIT delivers consistent gains. On ImageNet-1K 512px, ELIT delivers an average gain of 35.3 % 35.3\% and 39.6 % 39.6\% in FID and FDD scores. Figure 1 : Flexible compute allocation with ELIT. Starting from a vanilla DiT, we add a variable-length set of latent tokens—the latent interface —and two lightweight cross-attention layers, Read and Write. At inference, the number of latent tokens is a user-controlled knob that yields a smooth quality–FLOPs trade-off across DiT, U-ViT, HDiT, and MM-DiT backbones. 1 Introduction Recent years have seen dramatic progress in image and video generation. Owing to the simplicity of their design, architectures centered on Diffusion Transformers (DiTs) ( Peebles and Xie 2023 ) have scaled reliably and delivered state-of-the-art fidelity ( Wu et al. 2025a ; Wang et al. 2025a ; Haji-Ali et al. 2025a ) . Compute has been the primary determinant of generation quality, but continued scaling has inflated training and inference costs. This raises a central question: do DiTs utilize available computation effectively? We argue that such costs are amplified by the rigidity in the DiT design. First, a DiT typically commits to a per-step computational cost that is a fixed function of the input resolution, without accounting for latency and compute constraints. Second, we found that DiT allocates computation uniformly across image regions. In a controlled experiment, we probe the ability of a DiT to use extra compute to improve generation quality. As expected, quality improves on standard images when we increase the number of tokens by reducing the patchification size. However, when we increase the number of tokens by padding encoded image patches with zero-valued patches, we find that DiT fails to leverage the extra computation to improve generation quality. These observations suggest that compute is spent uniformly across image tokens. This is suboptimal since visual information in images is uneven: some regions are easy, others require more work. In this context, learning how to allocate computation across tokens through a flexible architecture holds the potential to dynamically reduce unnecessary computation. Figure 2 : Adaptive computation. We test whether DiT and ELIT-DiT can reallocate compute across image regions by training on synthetic inputs formed by zero-padding real images, artificially increasing the token count (◆). We compare its performance to baselines trained on real data using patch size 2\!\times\!2 and patch size 1\!\times\!1 (★). Vanilla DiT does not improve: attention in zeroed regions targets other zeroed regions (see “DiT Attention”), so extra tokens raise cost without benefits. In contrast, ELIT-DiT uses the Read layer to pull informative spatial tokens into the latent interface (see “Read Attention”), effectively filtering out zeroed areas (see “ELIT-DiT Attention”). Consequently, it leverages the larger token budget and matches the real-data baseline at equal FLOPs. Important previous work has sought forms of flexibility to make generative models more efficient. Adaptive generators allow a single model to operate at different budgets, but still spread computation uniformly across tokens ( Anagnostidis et al. 2025 ; Yu et al. 2019 ) or suffer from high complexity ( Zhao et al. 2025 ) . Masking-based methods improve training speed by skipping tokens. Yet, dropping is disabled for inference to avoid unrecoverable information loss ( Gao et al. 2023 ; Krause et al. 2025 ) , making inference compute unchanged. On the other hand, some training-free acceleration approaches reduce the computation for the least informative tokens during inference, leaving the training efficacy the same ( Liu et al. 2025c ; Yang et al. 2025 ; Jeong et al. 2025 ) . A complementary thread moves flexibility in the autoencoder by learning variable-length representations but stops short of endowing a generative model with an internal flexible representation ( Bachmann et al. 2025 ; Li et al. 2025 ) . Finally, RINs ( Jabri et al. 2023 ; Chen and Li 2023 ) learn to distribute computation non-uniformly across input tokens through a set of latent tokens, but keep inference budget fixed and depart significantly from the DiT architecture, hindering adoption. Building on the previous observations, we propose Elastic Latent Interface Transformer (ELIT) (see Figure 1 ), a minimal, DiT-compatible mechanism for representation-and-compute adaptivity. We introduce two lightweight cross-attention layers, Read and Write , that equip DiT-like architectures with a set of latent tokens we refer to as latent interface . These latent tokens act as a variable-size surface onto which to distribute input information in a flexible and learnable manner based on the difficulty of each region. Read pulls information from input tokens, which we will refer to as spatial , into the latent interface, prioritizing challenging regions. Write broadcasts updated latents back to the spatial tokens. Importantly, the number of latent tokens is user-controlled, directly setting the per-step compute budget. No changes to the training objective are necessary. We provide a thorough analysis of ELIT, showing that it successfully redistributes compute non-uniformly across input tokens across varying base architectures. Our latent interface consistently improves over a fixed-grid model with ImageNet-1k 512px FDD (Fréchet Distance on DINOv2 ( Oquab et al. 2024 ) features), improving 58.0% for DiT, 34.0% for U-ViT, 37.4% for HDiT. ELIT also allows for graceful compute-quality tradeoffs by selecting the number of latent tokens at inference time, regularly achieving better tradeoffs than sampling steps reduction while being compatible with training-free acceleration techniques ( Liu et al. 2025b ) . Additionally, variable compute enables autoguidance ( Karras et al. 2024 ) out of the box, which reduces inference cost by \approx 33\% without affecting the generation quality. In summary, this simple addition yields a framework capable of: (I) Adaptive computation. Compute is concentrated where it matters rather than spread uniformly across input regions. (II) Variable test-time compute. A single set of weights serves a spectrum of latency-quality points by selecting the number of latent tokens. (III) Improved sampling. Variable compute enables autoguidance ( Karras et al. 2024 ) out of the box. (IV) Drop-in training. We keep the design simple, retaining the vanilla rectified-flow objective and showing our method’s applicability to DiT,