Why Do DiT Editors Drift? Plug-and-Play Low Frequency Alignment in VAE Latent Space
Xiaoce Wang, Sifan Zhou, Kaifei Wang Leli Xu Xuerui Qiu Tao He Ming Li Tsinghua UniversityCarnegie Mellon UniversityPeking UniversityCASIAUniversity of Electronic Science and Technology of ChinaGuangming Laboratory · 2026
Read Why Do DiT Editors Drift? Plug-and-Play Low Frequency Alignment in VAE Latent Space by Xiaoce Wang, Sifan Zhou, Kaifei Wang Leli Xu Xuerui Qiu Tao He Ming Li Tsinghua UniversityCarnegie Mellon UniversityPeking UniversityCASIAUniversity of Electronic Science and Technology of ChinaGuangming Laboratory (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08250.txt.
Introduction
Xiaoce Wang, Sifan Zhou, Kaifei Wang, Leli Xu, and coauthors explain a slow failure. Repeated image edits with diffusion transformers drift over time. Each round adds a small semantic drift. After several turns, the picture no longer matches the user's intent. Single edits still look fine.
The authors split the pipeline into its VAE and transformer parts. The transformer injects mostly low-frequency drift in latent space. The VAE adds a steadier reconstruction bias. Their fix, VAE-LFA, needs no training. It low-pass filters the latent gap between rounds. It aligns low-frequency statistics to a moving average of past rounds. That damps the drift where it forms.
Catalog record
| CREATOR | Xiaoce Wang, Sifan Zhou, Kaifei Wang Leli Xu Xuerui Qiu Tao He Ming Li Tsinghua UniversityCarnegie Mellon UniversityPeking UniversityCASIAUniversity of Electronic Science and Technology of ChinaGuangming Laboratory |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08250
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Why Do DiT Editors Drift? Plug-and-Play Low Frequency Alignment in VAE Latent Space Xiaoce Wang, Sifan Zhou, Kaifei Wang Leli Xu Xuerui Qiu Tao He Ming Li Tsinghua UniversityCarnegie Mellon UniversityPeking UniversityCASIAUniversity of Electronic Science and Technology of ChinaGuangming Laboratory Abstract Recent advances in diffusion transformers (DiTs) have enabled promising single-turn image editing capabilities. However, multi-turn editing often leads to progressive semantic drift and quality degradation. In this work, we study this problem from a latent-space frequency perspective by decomposing the editing process into two functional components: VAE and DiT. Through systematic analysis in the VAE latent space, we uncover that the DiT introduces dominant low-frequency drift that accumulates as semantic misalignment across editing rounds, while the VAE contributes comparatively stable reconstruction bias. Based on this insight, we propose VAE-LFA (Low Frequency Alignment), a training-free, plug-and-play method that performs alignment in VAE latent space. VAE-LFA decomposes latent discrepancies across editing rounds via low-pass filtering, and aligns low-frequency statistics to an exponential moving average of previous rounds, effectively suppressing accumulated semantic drift while preserving high-frequency details. Our method requires no retraining, ground-truth priors, or access to diffusion parameters , making it applicable to both white-box and black-box DiT editors. For white-box models, VAE-LFA is seamlessly integrated into the editing pipeline by eliminating redundant VAE round trips; for black-box models, it operates via an off-the-shelf VAE to perform inter-round latent alignment. Extensive experiments demonstrate that VAE-LFA improves semantic consistency and visual fidelity across diverse multi-turn editing scenarios, including both controlled and in-the-wild images. Figure 1 : VAE-LFA improves multi-turn editing capabilities. Black-box: background visual quality degrades severely; White-box: style consistency is suboptimal. VAE-LFA improves both. 1 Introduction Recent advances in instruction-based image editing models built upon diffusion transformers (DiTs) and flow matching have achieved remarkable improvements in prompt-following, realism, and visual quality for single-round editing 25 ; 15 ; 33 ; 2 ; 7 ; 9 ; 6 ; 38 . Despite these advances, their performance degrades significantly under multi-turn ( e.g. , 5+ turns) iterative editing, often leading to severe semantic drift and quality deterioration 42 ; 17 ; 10 . This limitation poses a critical challenge for real-world deployment as users cannot be expected to articulate all editing intents with sufficient precision in a single prompt, and iterative refinement is inherent to any creative workflow. While prior works have proposed various remedies for multi-turn degradation, they either demand full-model retraining 23 ; 27 ; 1 , require internal access to denoising trajectories 10 , or rely on model-specific inference signals such as internal features or velocity fields 17 . Nevertheless, the community heavily relies on black-box editing models like GPT Image, Qwen Image 2.0 33 , Seedream 4.0 26 and GLM-Image 37 , yet these methods cannot be directly deployed on black-box or API-based editors without modification. This calls for a simple and effective remedy for editing degradation that requires no internal access. Figure 2 : Cross-architecture validation of frequency-wise drift. Relative radial-spectrum difference between DiT-only and VAE-only loops over 10 rounds: \Delta P(r)=\frac{P_{\text{DiT}}(r)-P_{\text{VAE}}(r)}{P_{\text{VAE}}(r)}\times 100\% . On both FLUX.2 and SD3, DiT transitions dominate low-frequency drift ( r<0.2 ), while VAE bias is stronger at higher frequencies. The vertical line marks r=0.2 . Moreover, fundamental questions remain unanswered: Why do DiT editors drift? Which component is the primary cause? To answer these questions, we draw upon recent evidence that frequency-based analysis in latent space provides a principled perspective 22 ; 21 . We focus on the VAE latent space , as the VAE encoder–decoder bottleneck is a fundamental component shared across virtually all modern DiT-based editors, making it a universal interface for component-level drift attribution. Specifically, we decompose the editing pipeline into two components, the VAE for encoding and decoding, and the DiT for iterative refinement, and systematically analyze frequency-wise drift in the latent space using no-operation prompts to isolate intrinsic degradation. As shown in Fig. 2 and Fig. 6 , our analysis reveals a striking asymmetry: semantic degradation is predominantly driven by accumulated low-frequency deviations introduced by the DiT over multiple iterations, whereas the VAE loop merely induces a stable, spectrally uniform reconstruction bias. This finding is counter-intuitive, challenging the prevailing hypothesis that repeated VAE cycles dominate multi-turn error accumulation 1 ; 36 . We further discover that VAE acts as an implicit regularizer that partially curbs DiT-induced low-frequency drift, rather than purely introducing degradation, as shown in Fig. 6 . Motivated by this asymmetry, we propose VAE-LFA (Low Frequency Alignment) (Fig. 3 ), a training-free and plug-and-play framework that operates directly in the VAE latent space. The key idea is to selectively constrain the low-frequency components that dominate semantic drift, while leaving high-frequency components unconstrained to preserve fine details and editing flexibility. By stabilizing the low-frequency latent statistics across editing rounds, our method effectively mitigates the accumulation of semantic misalignment without restricting the model’s ability to perform meaningful edits. Crucially, VAE-LFA requires no retraining, no ground-truth priors, and no access to internal model parameters, making it applicable to both white-box and black-box DiT-based editors. This design enables seamless integration into existing editing pipelines, either by directly operating within the model (for white-box settings) or by interfacing through an external latent space (for black-box settings), without modifying the underlying model. Extensive experiments demonstrate that VAE-LFA mitigates quality degradation across diverse editing scenarios. Across no-op, cycle, and long-chain protocols, our method improves semantic consistency and perceptual fidelity while preserving instruction-following capability (Fig. 1 , Fig. 7 , Tab. 1 , and Tab. 2 ). Overall, our contribution can be summarized as: • We present an interpretable analysis of multi-turn editing degradation from a VAE latent frequency perspective, showing that semantic drift is mainly driven by low-frequency drift. • We provide a controlled decomposition of the editing pipeline, revealing an asymmetry between DiT and VAE, where the former dominates semantic drift while the latter introduces stable reconstruction bias with partial regularization effects. • We propose VAE-LFA, a training-free and plug-and-play method that mitigates drift via low-frequency latent alignment, requiring no retraining or internal model access and supporting both white-box and black-box DiT-based editors with consistent performance improvements. Figure 3 : Overview of VAE-LFA : component-wise drift analysis and low-frequency latent alignment. 2 Related Works Instruction-based DiT editors achieve strong single-round results 25 ; 15 ; 33 , yet suffer from progressive drift under multi-turn editing 42 ; 17 ; 10 . Prior remedies require retraining 23 ; 27 , internal access 10 , or specialized architectures 17 , excluding commercial black-box models 33 ; 26 ; 37 . While the VAE bottleneck is known to introduce reconstruction bias 36 ; 14 , existing understanding of its latent space remains largely theoretical 25 ; 13 ; 8 , with limited insight into how VAE latent frequency bands govern specific semantics. Recent frequency-semantic correspondences 22 ; 21 offer a promising lens, but have not been exploited to attribute or mitigate iterative drift. In this work, we aim to explain and solve multi-turn editing degradation through a principled, interpretable analysis of the VAE latent space. By decomposing drift into frequency-specific components, we reveal which pipeline stage corrupts which semantic band, and why. Building on this causal understanding, we develop a training-free remedy that operates