JLT: Clean-Latent Prediction in Latent Diffusion Transformers

Funing Fu, Tenghui Wang, Guanyu Zhou, Junyong Cen, Qichao Zhu, [0.30em] Independent Researcher Wuhan University of Technology Hangzhou Jiyi Artificial Intelligence Co., Ltd. [0.15em] [email protected] [email protected] · 2026

Read JLT: Clean-Latent Prediction in Latent Diffusion Transformers by Funing Fu, Tenghui Wang, Guanyu Zhou, Junyong Cen, Qichao Zhu, [0.30em] Independent Researcher Wuhan University of Technology Hangzhou Jiyi Artificial Intelligence Co., Ltd. [0.15em] [email protected] [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27102.txt.

Introduction

Fu and four co-authors ask whether flow samplers should predict the clean endpoint or velocity, using JLT, a latent Transformer in a frozen auto-encoder. For squared error, the two choices are equivalent in algebra, yet a finite net assigns unequal compute under each setup. A Gaussian analysis traces the gap to the readout plus uniform target variance. Channel spectra show 90 percent of variance in 83 of 128 clean directions versus 109 velocity ones. Under a matched velocity goal, clean-endpoint work lifts ImageNet FID-50K at Base and Large scales.

Catalog record

CREATORFuning Fu, Tenghui Wang, Guanyu Zhou, Junyong Cen, Qichao Zhu, [0.30em] Independent Researcher Wuhan University of Technology Hangzhou Jiyi Artificial Intelligence Co., Ltd. [0.15em] [email protected] [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24578
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27102

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

JLT: Clean-Latent Prediction in Latent Diffusion Transformers Funing Fu, Tenghui Wang, Guanyu Zhou, Junyong Cen, Qichao Zhu, [0.30em] Independent Researcher Wuhan University of Technology Hangzhou Jiyi Artificial Intelligence Co., Ltd. [0.15em] [email protected] [email protected] Abstract Figure 1 : ImageNet 256\times 256 samples from JLT-B/1 using 50-step Heun sampling. 1 1 footnotetext: Equal contribution. Abstract Flow matching with clean-data prediction has shown that regressing the clean point can exploit low-dimensional structure more effectively than predicting an ambient noised quantity. We ask whether this principle remains useful after images are mapped into a learned latent space, where compression has already removed much of the raw pixel variability. We instantiate this comparison with JLT, a controlled 130M latent diffusion Transformer over frozen FLUX.2 VAE codes, and compare clean-latent prediction with a matched velocity-prediction DiT under the same representation, backbone, and training settings. Although x , \epsilon , and v are linearly convertible for a fixed corruption time, a local Gaussian analysis shows that velocity regression inherits an isotropic target-covariance floor and amplifies low-variance latent directions, while clean prediction damps them. On ImageNet 256\times 256 , JLT-B/1 obtains FID-50K 2.50 with classifier-free guidance, with a large matched-target gap over velocity prediction. These results suggest that prediction targets in latent diffusion are representation-dependent geometric choices, rather than interchangeable algebraic parameterizations. Code is available at https://github.com/akatsuki-neo/JLT . 1 Introduction Denoising diffusion models are motivated by reversing a corruption process, yet many successful systems do not ask the neural network to directly reconstruct the clean sample. DDPM popularized \epsilon -prediction [ 7 ] ; progressive distillation and flow-based formulations made velocity regression a standard choice [ 21 , 14 , 15 ] ; and EDM emphasized that prediction parameterization, loss weighting, preconditioning, and sampling should be disentangled as a design space [ 11 ] . Algebraically, these targets are closely related. Statistically, however, the direct output learned by a finite-capacity network can change the difficulty of the regression problem. JiT [ 12 ] makes this distinction explicit in pixel space. It argues that clean images concentrate near a low-dimensional data manifold, whereas noise and velocity targets contain ambient, off-manifold components. Directly predicting clean data can therefore let a Transformer focus on structured variation rather than reconstructing full-dimensional noise. The question we study is complementary: if the model already operates in a compressed latent space [ 18 ] , does the direct prediction target still matter? The latent setting preserves this distinction. We compare clean-latent and velocity targets under a fixed FLUX.2 VAE representation, the same Base-scale Transformer configuration, and our 250K-step (200-epoch) training setting. We name latent models in VAE-grid units: the clean-latent variants are JLT-B/1 and JLT-B/2, while the matched velocity variants are denoted DiT-B/1 and DiT-B/2; raw-pixel clean-prediction baselines remain JiT-B/16 and JiT-B/32. Under this notation, JLT-B/1 improves FID-50K from 6.56 to 2.56 over DiT-B/1, and JLT-B/2 improves it from 28.71 to 14.81 over DiT-B/2. Because the representation is shared within each pair, this separation is better viewed as a target-geometry effect than as a consequence of latent compression alone. Our main contribution is a controlled latent target study rather than a new backbone. We instantiate the study with JLT, a Base-scale latent Transformer built to isolate the prediction target in a fixed FLUX.2 VAE latent space. The first core result is empirical: under the same representation, architecture scale, training setup, and evaluation protocol, clean-latent prediction consistently outperforms matched velocity prediction. The second core result is explanatory: a local Gaussian analysis shows that velocity prediction adds an isotropic covariance floor and amplifies low-variance latent directions, whereas clean prediction attenuates those directions. Additional algebraic conversions, proof details, implementation settings, and diagnostic suggestions are deferred to the appendix. 2 Related Work Denoising objectives and prediction targets. The modern diffusion objective inherits the denoising viewpoint of earlier denoising autoencoders, where a model learns a structured signal from a corrupted observation [ 23 , 24 ] . In generative diffusion, DDPM popularized predicting the Gaussian perturbation added during the forward process [ 7 ] , and ADM showed that architectural and guidance choices can substantially improve ImageNet synthesis [ 3 , 8 ] . Subsequent parameterizations changed the direct regression target: progressive distillation uses velocity parameterization to stabilize few-step students [ 21 ] , while flow matching and rectified flow express generation as learning a transport vector field between noise and data [ 14 , 15 ] . EDM further clarified that output parameterization, loss weighting, preconditioning, and sampler design are separable choices rather than one inseparable procedure [ 11 ] . Parameterization as geometry rather than notation. Although x , \epsilon , and v can be mapped to each other algebraically, several recent analyses suggest that the target presented to the network matters under finite capacity and finite data. JiT argues from the manifold assumption that clean images occupy structured low-dimensional subsets of pixel space, whereas noise and velocity contain ambient components that are not supported by the data distribution [ 12 ] . Complementary theoretical studies relate target choice to intrinsic dimension, loss weighting, and training dynamics [ 10 , 5 ] . Our work follows this geometric interpretation but shifts the question from raw pixels to a fixed VAE latent representation: once the space is held fixed, the remaining gap between clean prediction and velocity prediction must come from the induced target distribution. Latent diffusion and Transformer backbones. Latent Diffusion Models reduce the cost of high-resolution synthesis by training the generative model in an autoencoder latent space and decoding only after sampling [ 18 ] . DiT replaces convolutional U-Nets with Vision-Transformer-style blocks over latent patches and shows that model complexity and token count correlate strongly with FID [ 4 , 22 , 17 ] . SiT then studies flow and diffusion variants on the same Transformer backbone, emphasizing controlled comparisons with fixed parameter count and GFLOPs [ 16 ] . Other Transformer-based iterative generators also explore adaptive computation and scalable token processing [ 9 ] . JLT adopts this controlled-comparison philosophy: the architecture and training scale are kept close to JiT-B, while the central ablation changes the direct target in FLUX.2 VAE latent space. Representation geometry and alignment. A parallel line of work studies how the representation space itself affects generative learning. REPA aligns diffusion Transformer hidden states with external visual representations and shows large improvements in training efficiency [ 25 ] . RiT studies frozen DINOv2 features and argues that representation-space geometry can make x -prediction well conditioned even when intrinsic dimensionality is comparable to pixels [ 26 ] . These works vary or augment the representation. By contrast, our main experiment fixes the FLUX.2 VAE latent representation and compares y_{x}=x with y_{v}=x-\epsilon inside that same space. This isolates a target-geometry effect that is orthogonal to tokenizer improvements, representation alignment, or larger backbones. 3 Method 3.1 Formulation and prediction targets Let x\in\mathbb{R}^{D} denote the clean latent produced by a fixed encoder, and let \epsilon\sim\mathcal{N}(0,I) denote Gaussian noise in the same coordinate system. We use the linear corruption path z_{t}=tx+(1-t)\epsilon,\qquad t\in[0,1]. (1) The three common direct targets are y_{x}=x,\qquad y_{\epsilon}=\epsilon,\qquad y_{v}=x-\epsilon. (2) For fixed t , x -, \epsilon -, and v -parameterizations are algebraically equivalent: once a model predicts any one target, the other endpoint

Read the complete text