Reasoning to Align: Implicit Reasoning in Diffusion Transformers for Video Editing
Yan Li, Lin Liu, Xiaopeng Zhang, Qi Tian, The Hongkong University of Science and Technology · 2026
Read Reasoning to Align: Implicit Reasoning in Diffusion Transformers for Video Editing by Yan Li, Lin Liu, Xiaopeng Zhang, Qi Tian, The Hongkong University of Science and Technology (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24674.txt.
Introduction
Yan Li and others argue that video-editing diffusion transformers fail for two structural reasons. First, editing signals feed undifferentiated into every block. Second, focus patterns get only indirect supervision from pixel reconstruction. Their RVEDiT framework fixes both. It routes learnable editing tokens from a multimodal model into shallow blocks, saving native tokens for deeper blocks. It also aligns editing focus with a parameter-sharing reference branch during training. It beats leading baselines, with the clearest gains on localized and compositional edits, at no extra inference cost.
Catalog record
| CREATOR | Yan Li, Lin Liu, Xiaopeng Zhang, Qi Tian, The Hongkong University of Science and Technology |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69474 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24674
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reasoning to Align: Implicit Reasoning in Diffusion Transformers for Video Editing Yan Li, Lin Liu, Xiaopeng Zhang, Qi Tian, The Hongkong University of Science and Technology Abstract Instruction-based video editing requires transforming a source video according to a natural-language instruction while preserving irrelevant content and remaining temporally coherent. We argue that existing Diffusion Transformer (DiT) editors struggle with this task for two structural reasons. First, conditioning signals are fed undifferentiated into all transformer blocks, forcing a single token stream to encode both global editing intent and fine-grained visual evidence. Second, the cross-attention patterns that govern the edit are supervised only indirectly through pixel-level reconstruction, leaving the model’s internal reasoning process underconstrained. To address both limitations, we propose RVEDiT , an implicit R easoning V ideo E diting DiT framework built around two complementary components. The first, Granularity-Routed Token Conditioning , introduces learnable editing tokens distilled from a multimodal LLM and routes them to shallow blocks, while reserving native visual and textual tokens for deeper blocks, thereby inducing a coarse-to-fine editing process inside the backbone. The second, Reference-Anchored Attention Alignment , employs a parameter-sharing reference branch during training and maximizes the mutual information between the attention features of the editing and reference branches, regularizing the model’s internal reasoning without incurring any additional inference cost. Experiments on standard instruction-based video editing benchmarks show that RVEDiT consistently outperforms state-of-the-art baselines, with particularly strong gains on localized and compositional edits. 1 Introduction Video has become the dominant medium of visual communication, yet editing it remains a labor-intensive craft that demands expertise in timeline-based software, keyframe manipulation, and frame-level compositing. Recent progress in large language models (LLMs) ( 7 ; 1 ) and text-conditioned diffusion models ( 20 ; 41 ) has opened a compelling alternative: instruction-based video editing , in which users describe a desired modification in natural language (e.g., “replace the dog with a corgi” or “make it look like a Van Gogh painting”) and the system produces the edited video automatically, promising to make high-quality editing accessible to a far broader population of creators, educators, and everyday users 15 . As in instruction-based image editing ( 6 ; 19 ) , video editing is not merely a text-to-pixel mapping problem, but a latent reasoning problem : given a free-form instruction and a source video, the model must infer which spatiotemporal regions to modify, what transformation to apply, and what must remain invariant across frames ( 18 ) . A natural response is to externalize this reasoning by letting a multimodal LLM emit explicit plans, masks, or programs for the diffusion model to execute ( 54 ; 52 ; 56 ) , but this outsources correctness to an external MLLM whose own biases and hallucinations propagate into the plan and cannot be corrected downstream, so the editor inherits whatever mistakes the MLLM makes. Existing systems largely sidestep the issue instead: UNet-based editors ( 50 ; 37 ; 30 ) either inflate image-level backbones along time ( 6 ) or scale triplet corpora atop unchanged architectures ( 18 ; 3 ; 61 ; 58 ) , treating the backbone as a black box, and these recipes transfer poorly to Diffusion Transformers (DiTs) ( 35 ) , whose global token mixing departs sharply from the locality bias of UNets. It therefore remains an open question how a DiT backbone can be endowed with the capacity to perform this implicit reasoning natively, so that instruction faithfulness, source preservation, and temporal coherence emerge from a single, end-to-end generative pass. We argue that current DiT pipelines fail to support such implicit reasoning for two structural reasons. First, the conditioning stream is undifferentiated, leaving no workspace for reasoning. Existing DiTs concatenate textual and visual tokens into a single joint sequence that is processed uniformly by every transformer block ( 35 ; 16 ; 36 ) , forcing the same token set to encode both the high-level editing intent and the low-level visual targets that specify which regions, textures, and motions must change or be preserved ( 19 ; 34 ) . Second, the reasoning pathway is unsupervised. Flow-matching and denoising objectives ( 20 ; 28 ) supervise only the predicted velocity field, so the cross-attention maps, which constitute the model’s reasoning trace, encoding which text token reads which visual region at which depth ( 19 ; 45 ; 9 ) , are shaped purely by gradients back-propagated through a pixel-level reconstruction loss ( 9 ; 39 ; 2 ) . Thus, the direct symptoms of an attention map are too blunt to resolve the fine-grained correspondences an instruction-faithful edit requires. In other words, the reasoning happens, but it is never taught. Motivated by this bottleneck, we propose RVEDiT to constrain both what the backbone observes and what its internal computations are encouraged to resemble . To structure the conditioning stream, we first derive a compact set of learnable editing tokens from a MLLM and use them as an abstract carrier of the editing intent. Rather than exposing every block to the same mixture of visual and textual tokens, we route information by depth: shallow blocks attend only to the edit-level tokens, while deeper blocks also receive the native visual and textual tokens for local grounding. This depth-aware routing creates a coarse-to-fine editing process inside the DiT backbone without requiring an explicit symbolic reasoning trace; we refer to it as Granularity-Routed Token Conditioning (§ 4.2 ). To supervise the resulting internal reasoning trace, we further add a parameter-sharing reference branch during training, conditioned on the ground-truth edited video and caption. By maximizing mutual information between the cross-attention maps of the editing branch and this informed reference branch at a randomly sampled depth, the model is encouraged to learn attention patterns that better reflect a successful edit. The reference branch is removed at inference, so RVEDiT adds no extra deployment cost; we call this objective Reference-Anchored Attention Alignment (§ 4.3 ). In summary, this paper makes the following contributions: • We present RVEDiT, a DiT-native framework for instruction-based video editing that formulates the task as a latent reasoning problem, identifying two structural bottlenecks in existing DiT pipelines that hinder implicit reasoning: undifferentiated conditioning across depths and unsupervised internal reasoning traces. • We propose two complementary components to endow the DiT backbone with implicit reasoning capabilities. Granularity-Routed Token Conditioning establishes a coarse-to-fine editing by routing MLLM-derived learnable editing tokens and native visual/textual tokens to different depths, while Reference-Anchored Attention Alignment actively supervises this reasoning pathway by maximizing the mutual information between the cross-attention maps of the editing branch and an informed reference branch. • Extensive experiments demonstrate that RVEDiT consistently outperforms strong open-source baselines, successfully internalizing the reasoning process to achieve significant gains on localized, compositional, and fidelity-sensitive edits, with ablations confirming the complementarity of the two proposed components. 2 Related Work 2.1 Instruction-based Video Editing Instruction-based editing has emerged as a natural interface for generative models, allowing users to modify visual content through free-form natural-language commands rather than carefully engineered source–target prompt pairs. Pioneered in the image domain by InstructPix2Pix 6 and its successors 42 ; 11 , this paradigm has been progressively transferred to video. Early attempts such as InsV2V 13 and InstructVid2Vid 38 inflate image-level instruction-tuned backbones along the