RefDecoder : Enhancing Visual Generation with Conditional Video Decoding
Xiang Fan, Yuheng Wang, Bohan Fang, Zhongzheng Ren, Ranjay Krishna · 2026
Read RefDecoder : Enhancing Visual Generation with Conditional Video Decoding by Xiang Fan, Yuheng Wang, Bohan Fang, Zhongzheng Ren, Ranjay Krishna (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15196.txt.
Introduction
Xiang Fan and four others argue that video making decoders deserve the same conditioning as their cleaning networks. Latent diffusion decoders usually run bare, losing detail and drifting from the input image. RefDecoder injects a high-fidelity reference image signal into decoding through pointer attention, handling detail-rich tokens with video latents at every growth stage. It lifts PSNR by up to 2.1 decibels over bare rivals and swaps into old systems without extra fine-tuning.
Catalog record
| CREATOR | Xiang Fan, Yuheng Wang, Bohan Fang, Zhongzheng Ren, Ranjay Krishna |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58531 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15196
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RefDecoder : Enhancing Visual Generation with Conditional Video Decoding Xiang Fan, Yuheng Wang, Bohan Fang, Zhongzheng Ren, Ranjay Krishna Abstract Video generation powers a vast array of downstream applications. However, while the de facto standard, i.e. , latent diffusion models, typically employ heavily conditioned denoising networks, their decoders often remain unconditional. We observe that this architectural asymmetry leads to significant loss of detail and inconsistency relative to the input image. To address this, we argue that the decoder requires equal conditioning to preserve structural integrity. We introduce RefDecoder , a reference-conditioned video VAE decoder by injecting high-fidelity reference image signal directly into the decoding process via reference attention. Specifically, a lightweight image encoder maps the reference frame into the detail-rich high-dimensional tokens, which are co-processed with the denoised video latent tokens at each decoder up-sampling stage. We demonstrate consistent improvements across several distinct decoder backbones ( e.g. , Wan 2.1 and VideoVAE + ), achieving up to + 2.1 +2.1 dB PSNR over the unconditional baselines on the Inter4K, WebVid, and Large Motion reconstruction benchmarks. Notably, RefDecoder can be directly swapped into existing video generation systems without additional fine-tuning, and we report across-the-board improvements in subject consistency, background consistency, and overall quality scores on the VBench I2V benchmark. Beyond I2V, RefDecoder generalizes well to a wide range of visual generation tasks such as style transfer and video editing refinement. Figure 1: (a) RefDecoder improves video VAE decoders by conditioning on a high-fidelity reference signal that bypasses the lossy VAE latent round trip. Given the encoded latents, the decoder injects fine-grained details that are not preserved in the VAE latent space, improving reconstruction fidelity and downstream generation quality. (b) RefDecoder integrates with existing video generation pipelines (Wan 2.1 shown) without requiring costly re-training of the diffusion model. 1 Introduction In the dominant paradigm of video generation, a diffusion backbone denoises latent representations guided by an input conditioning, while a VAE decoder subsequently maps these latents back to pixel space 2 ; 39 ; 52 . While substantial effort has been devoted to improving the conditioning and architecture of the diffusion backbone 28 ; 13 ; 31 , the VAE decoder has received comparatively little attention and remain unconditional in mainstream video diffusion models . It is typically treated as a standalone reconstruction module that operates without access to the primary conditioning signal at inference time 2 ; 39 ; 51 ; 17 . We observe that this architectural asymmetry creates a systematic bottleneck: even when the diffusion model faithfully preserves the reference information within the latent, the decoder must reconstruct fine-grained spatial details from a heavily compressed representation without a detailed anchor. This leads to two characteristic failure modes: (1) Progressive degradation of spatial details , where textures, edges, and high-frequency content deteriorate in frames that deviate from the conditioning image (e.g. Figure 3 baseline details); and (2) Temporal inconsistency , where appearance drifts across the sequence (e.g. Figure 4 (a) human face). These artifacts emerge during latent-to-pixel decoding rather than the diffusion process itself, exposing the decoder as an under-exploited component that fails to leverage the visual cues already provided by the reference. However, injecting the reference image into the decoder is non-trivial for several reasons. (1) Because the decoder operates through hierarchical up-sampling stages, it remains an open question at which specific stage the reference signal should be introduced. (2) The conditioning mechanism must maintain compatibility with pretrained decoder weights, ensuring the system remains functional in its original setting, i.e. , no reference image. (3) The approach should be sufficiently generic to generalize across diverse visual generation tasks and various decoder backbones. To address aforementioned problems, our key insight is that the decoder’s hidden space is far richer than the VAE latent space; encoding the reference image into this high-fidelity space and injecting it as additional self-attention tokens lets the decoder exploit details unavailable from the latent code alone. Building on this, we propose RefDecoder , a reference-conditioned video VAE decoder with two components: a reference image encoder , a lightweight convolutional network mapping the reference frame into the decoder’s hidden space as spatially aligned tokens (Sec. 2 (a)); and a conditional token decoder that, at each up-sampling stage, concatenates reference and video latent tokens along the temporal axis and jointly processes them via a shared transformer block with rotary positional embeddings (RoPE) 33 ; the tokens are then separated and up-sampled independently, preserving compatibility with the pretrained decoder (Sec. 2 (b)). To prevent the decoder from ignoring the reference, we further introduce a latent token dropout strategy that randomly zeroes video latent tokens at a variable rate, forcing the model to recover spatial details from reference tokens via attention (Sec. 2 (c)). Our newly introduced modules are architecture-agnostic and generalize to existing video VAE decoders, with the transformer block shared across decoder stages. Because only the decoder is modified and the encoder remains frozen, RefDecoder is a drop-in replacement for any existing video VAE decoder and can immediately benefit downstream diffusion models without retraining. We test RefDecoder on different video VAE backbones including Wan 2.1 39 and VideoVAE + 48 , demonstrating consistent improvements across all of them. On the Inter4K reconstruction benchmark, RefDecoder achieves over 1 dB PSNR improvement compared to the unconditional baseline. In generation tasks, on the VBench I2V benchmark 14 ; 15 , our method improves subject consistency, background consistency, motion smoothness, aesthetic quality, and overall scores. We show that the same reference injection mechanism naturally extends to decode-time style transfer . By supplying a style image as the reference, RefDecoder produces stylized video without any task-specific modification, validating the generality of conditional token decoding. Furthermore, in video editing tasks, RefDecoder enhances the fidelity of edited videos to the input reference while preserving the intended edits, demonstrating that reference conditioning as a powerful tool to mitigate the trade-off between editability and fidelity 10 . These results suggest a broader message: in image conditioned generation pipelines, the decoder is not merely a passive reconstruction module but an active participant that can—and should—leverage conditioning signals. Don’t forget the decoder. 2 Method Figure 2: Architecture overview of the RefDecoder . The reference image is encoded into tokens and injected into the decoder through shared attention blocks, enabling interactions between reference and latent tokens during decoding. We propose RefDecoder , a reference-conditioned video VAE decoder that injects reference image tokens directly into the decoding process via joint attention over concatenated reference and video tokens, enhancing spatial fidelity and temporal coherence. The overall architecture is illustrated in Fig. 2 . We build upon a standard video VAE backbone and augment its decoder with conditional token interactions, which we detail below. Reference image encoding. We start with a reference image I_{\mathrm{ref}}\in\mathbb{R}^{3\times H\times W} . Typical methods encode the reference into a low-dimensional feature space comparable to the VAE latent bottleneck ( e.g. 16 channels), and obtain hierarchical features derived therefrom. To preserve high-fidelity information, we instead project the original image patches directly into a high-dimensional feature space ( e.g. 512 dimensions), which is fed to the decoder at the very first stage, upsampled alongside the video features throughout decoding stages, and interacted with through injected attention layers at every stage