Rebalancing Reference Frame Dominance to Improve Motion in Image-to-Video Models

Wooseok Jeon, Seungho Park, Seunghyun Shin, Sangeyl Lee, [2pt] Hyeonho Jeong, Hae-Gon Jeon · 2026

Read Rebalancing Reference Frame Dominance to Improve Motion in Image-to-Video Models by Wooseok Jeon, Seungho Park, Seunghyun Shin, Sangeyl Lee, [2pt] Hyeonho Jeong, Hae-Gon Jeon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19398.txt.

Introduction

Jeon and colleagues diagnose why image-to-video models often produce near-still clips. Non-reference frames pay too much self-attention to reference-frame keys, flooding later frames with the still image and choking motion. Their DyMoS method rebalances that attention path during early denoising steps, with one scalar dialing motion strength up or down. It needs no extra training, leaves image and weights untouched, and across several leading backbones it boosts motion while preserving look and likeness.

Catalog record

CREATORWooseok Jeon, Seungho Park, Seunghyun Shin, Sangeyl Lee, [2pt] Hyeonho Jeong, Hae-Gon Jeon
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47275
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19398

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rebalancing Reference Frame Dominance to Improve Motion in Image-to-Video Models Wooseok Jeon, Seungho Park, Seunghyun Shin, Sangeyl Lee, [2pt] Hyeonho Jeong, Hae-Gon Jeon Abstract Image-to-video models often generate videos that remain overly static, compared to text-to-video models. While prior approaches mitigate this issue by weakening or modifying the image-conditioning signal, they often require additional training or sacrifice fidelity to the reference image. In this work, we identify reference-frame dominance as a key mechanism behind motion suppression. We observe that non-reference frames in I2V models allocate excessive self-attention to reference-frame key tokens, causing reference information to be over-propagated across time and suppressing inter-frame dynamics. Based on this finding, we propose DyMoS (Dynamic Motion Slider), a training-free and model-agnostic method that rebalances the attention pathway from generated frames to the reference frame during initial denoising steps. DyMoS leaves both the input image and model weights unchanged and introduces a single scalar parameter for continuous control over motion strength. Experiments across multiple state-of-the-art I2V backbones demonstrate that DyMoS consistently improves motion dynamics while maintaining visual quality and fidelity to the reference image. Figure 1 : Example videos from our method. We present DyMoS, a training-free and model-agnostic method for improving motion dynamics in image-to-video generation. (a) Comparison of generated videos from the same input image. DyMoS produces dynamic motion while preserving video quality. (b) Furthermore, DyMoS provides continuous control over motion dynamics. 1 Introduction Recent advances in generative models have been driven by the remarkable success of diffusion [ 13 , 39 , 37 , 6 ] and flow-matching [ 28 , 27 , 1 , 7 ] frameworks. Building on these paradigms, text-to-video (T2V) models [ 41 , 44 , 49 , 2 , 14 , 25 , 12 ] have made it possible to generate high-quality videos from text prompts. Image-to-video (I2V) models [ 41 , 44 , 49 , 2 , 9 , 10 , 30 , 34 , 43 , 47 , 50 , 18 ] extend this capability by conditioning generation on reference images 1 1 1 In this paper, a reference image serves as the initial frame of the generated video, rather than functioning solely as an appearance or identity reference. , enabling the creation of videos that follow text prompts while preserving the content and appearance of the input. These models are typically adapted from large-scale T2V backbones by introducing an additional image-conditioning pathway, which grounds the generation in specific visual input while retaining the motion priors learned from large-scale video data. Despite this progress, I2V models often produce videos that remain overly static compared to their T2V counterparts, even when the prompt explicitly describes motion. Prior works address this static motion bias by either modifying the image-conditioning pathway or weakening the conditioning signal during generation [ 51 , 8 , 4 ] . While these approaches can encourage stronger motion, they often require additional training or involve direct modification of the input signal. In the latter case, improved motion dynamics are achieved at the cost of fidelity to the reference image. In this work, we revisit the static motion bias by analyzing how reference image information is utilized within I2V models. Since I2V models process spatiotemporal latent tokens through self-attention, their attention maps allow us to analyze how information flows across frames during generation. By comparing paired T2V and I2V generations, we observe that query tokens in non-reference frames in I2V models allocate substantially more attention to the reference-frame key tokens than their paired T2V counterparts. We refer to this asymmetric attention pattern as reference-frame dominance . This observation implies that the static nature of I2V models may stem from excessive propagation of reference frame information through internal attention routing, rather than that from the input image itself. To test this hypothesis, we directly modulate the attention logits associated with reference-frame key tokens. Strengthening these logits makes generation more static, whereas weakening them restores motion and reduces the reference-frame dominance. This provides an intervention-based evidence that the reference-frame dominance is a primary driver of motion suppression in I2V models. Based on this insight, we present DyMoS (Dynamic Motion Slider), a simple, training-free method that mitigates the reference-frame dominance by rebalancing attention logits. DyMoS introduces a scalar bias shift to the attention logits corresponding to reference-frame key tokens during initial denoising steps, leaving the input image and model weights unchanged. This allows DyMoS to improve motion dynamics while preserving fidelity to the reference image The magnitude of this bias serves as a single parameter for continuously controlling the generation, ranging from strict reference-image preservation to highly dynamic motion. We validate the effectiveness of DyMoS in improving motion dynamics across state-of-the-art I2V models, including Wan 2.2 [ 44 ] , Wan 2.1 [ 44 ] , HunyuanVideo-1.5 [ 41 ] and CogVideoX-5B [ 49 ] . Across all backbones, DyMoS consistently improves motion dynamics while preserving reference fidelity, incurring negligible additional inference-time overhead (e.g., 1.2% on Wan 2.2). We further show that DyMoS provides continuous control over the static-to-dynamic spectrum, enabling users to adjust motion strength with a single scalar parameter. These results demonstrate that static motion bias in I2V generation can be effectively mitigated by intervening at the attention-logit level, without fine-tuning the model weights or degrading the input image. 2 Background Text-to-video models. Modern flow-based text-to-video (T2V) models [ 44 , 41 ] employ diffusion transformers (DiTs) [ 32 , 29 ] , operating in the latent space of a 3D variational autoencoder (VAE) [ 21 ] , which consists of an encoder \mathcal{E}(\cdot) and a decoder \mathcal{D}(\cdot) . An RGB video \bm{x} is first compressed into a low-dimensional latent representation via \bm{z}_{0}=\mathcal{E}(\bm{x}) . The latent \bm{z}_{0} is then perturbed along a flow-matching interpolation path \bm{z}_{t}=(1-t)\bm{z}_{0}+t\bm{\epsilon} , where t\in[0,1] and \bm{\epsilon}\sim\mathcal{N}(\bm{0},\bm{I}) . A neural network \bm{v}_{\theta}(\bm{z}_{t},t,\bm{c}) is trained to approximate a target velocity \bm{v}=\bm{\epsilon}-\bm{z}_{0} , where \bm{c} denotes the text condition embedded by a pretrained text encoder such as T5 [ 33 ] . Generation starts from a Gaussian latent \bm{z}_{T}\sim\mathcal{N}(\bm{0},\bm{I}) and progressively denoises it into a clean latent \bm{z}_{0} using the learned velocity field. The final latent is then decoded into pixel space as \hat{\bm{x}}=\mathcal{D}(\bm{z}_{0}) . During the denoising process, classifier-free guidance (CFG) [ 15 ] is commonly applied in order to improve text-conditional generation by extrapolating between the unconditional and conditional predictions: \tilde{\bm{v}}_{\theta}(\bm{z}_{t},t,\bm{c})=\bm{v}_{\theta}(\bm{z}_{t},t,\varnothing)+\omega\left(\bm{v}_{\theta}(\bm{z}_{t},t,\bm{c})-\bm{v}_{\theta}(\bm{z}_{t},t,\varnothing)\right), (1) where \omega\geq 1 is the guidance scale, and \varnothing denotes the null text condition. Self-attention mechanism. Within each DiT block, spatiotemporal latent tokens are processed by self-attention. Let S=f\cdot h\cdot w be the number of latent tokens. For each attention head, the attention weights are computed as \mathcal{A}=\operatorname{softmax}(\mathcal{L}),\qquad\mathcal{L}=\frac{QK^{\top}}{\sqrt{d_{h}}}\in\mathbb{R}^{S\times S}, (2) where Q,K\in\mathbb{R}^{S\times d_{h}} are query and key projections, d_{h} is the per-head dimension, and \mathcal{L} denotes the attention logits. Since the token sequence spans both spatial and temporal dimensions, this operation aggregates information within and across frames. Image-to-video models. Image-to-video (I2V) models are commonly obtained by fine-tuning T2V models to additionally condition on a reference image \bm{x}_{\mathrm{ref}} , which corresponds to the first frame of the generated video. The reference image is first encoded as \mathcal{E}(\bm{x}_{\mathrm{ref}}) and temporally aligned with the noisy video latent. For simplicity, we denote the resulting reference-image latent by \bm{z}_{\mathrm{ref}} . The aligned reference-image latent

Read the complete text