Reward-Forcing: Autoregressive Video Generation with Reward Feedback

Jingran Zhang, Ning Li, Yuanhao Ban, Andrew Bai, Justin Cui · 2026

Read Reward-Forcing: Autoregressive Video Generation with Reward Feedback by Jingran Zhang, Ning Li, Yuanhao Ban, Andrew Bai, Justin Cui (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16933.txt.

Introduction

Jingran Zhang, Ning Li, Yuanhao Ban, Andrew Bai, and Justin Cui add reward checks to video bots. Most video bots build clips by views from both ends. Step-by-step bots can make clips live, frame by frame. Past step bots leaned on big teacher bots. That lean caps their skill when teachers are weak. Their 2026 preprint swaps the teacher lean for reward marks.

Clips gain marks as they grow, and low marks get cut. The bot learns to chase high-mark paths on its own. Live clip tools could gain skill with no big teacher. Such tools serve ads, games, and quick film drafts.

Catalog record

CREATORJingran Zhang, Ning Li, Yuanhao Ban, Andrew Bai, Justin Cui
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-16933

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reward-Forcing: Autoregressive Video Generation with Reward Feedback Jingran Zhang, Ning Li, Yuanhao Ban, Andrew Bai, Justin Cui Abstract While most prior work in video generation relies on bidirectional architectures, recent efforts have sought to adapt these models into autoregressive variants to support near real-time generation. However, such adaptations often depend heavily on teacher models, which can limit performance, particularly in the absence of a strong autoregressive teacher, resulting in output quality that typically lags behind their bidirectional counterparts. In this paper, we explore an alternative approach that uses reward signals to guide the generation process, enabling more efficient and scalable autoregressive generation. By using reward signals to guide the model, our method simplifies training while preserving high visual fidelity and temporal consistency. Through extensive experiments on standard benchmarks, we find that our approach performs comparably to existing autoregressive models and, in some cases, surpasses similarly sized bidirectional models by avoiding constraints imposed by teacher architectures. For example, on VBench, our method achieves a total score of 84.92, closely matching state-of-the-art autoregressive methods that score 84.31 but require significant heterogeneous distillation. 1 Introduction Diffusion models 18 ; 33 have emerged as a powerful class of generative models, achieving state-of-the-art results in a wide range of domains, including image synthesis, audio generation, and molecular modeling. By iteratively denoising data from a simple noise distribution, these models are capable of producing high-fidelity samples that capture complex data distributions. Their theoretical foundation rooted in stochastic differential equations, combined with their empirical robustness, has made diffusion models a dominant approach in the generative modeling landscape. Building on this success, video diffusion models 19 ; 3 extend the capabilities of diffusion-based generation to the temporal domain. Unlike images, videos require the model to generate spatially coherent and temporally consistent sequences of frames, which significantly increases the modeling complexity. Recent advancements have adapted diffusion models to handle temporal correlations by incorporating spatiotemporal attention, motion priors, and multi-frame conditioning mechanisms. These models have demonstrated promising results in generating short to medium-length video clips with high perceptual quality. However, most existing video diffusion models are built upon bidirectional architectures 55 ; 29 , where the generation of each frame depends on information from both past and future timesteps. While effective for high quality video generation, bidirectional models are inherently unsuitable for streaming or real-time video generation, taking several minutes to generate a 5-second video. Recent efforts have explored transforming bidirectional video diffusion models into autoregressive formulations 60 ; 26 to enable low-latency and scalable generation. These methods typically rely on distillation from powerful bidirectional teachers, but their performance is often limited by the quality of the teacher model and the challenges of maintaining temporal coherence in the absence of future context. Meanwhile, reinforcement learning (RL) has emerged as a powerful paradigm for post-training fine-tuning for enhancing the quality and alignment of diffusion-based image generation. By optimizing generation policies directly with respect to reward signals—such as human preferences, aesthetic scores, or perceptual quality—RL enables fine-grained control over generative behavior beyond supervised objectives. Techniques such as Reinforcement Learning with Human Feedback (RLHF) 40 ; 32 have demonstrated success in aligning language and vision-language models, and recent works have begun applying RL to tune diffusion models for improved fidelity, diversity, or alignment with desired styles. In this work, we build upon these directions and present a novel approach that leverages autoregressive video diffusion modeling combined with reinforcement learning to achieve efficient, scalable, and high-quality streaming video generation. In summary, our contributions are as follows: • We demonstrate that the performance of existing methods for converting bidirectional video diffusion models into autoregressive models are bounded by the teacher model’s performance. • We observe that consistent motions are learned before texture, consistent with previous studies on image generation, and propose a framework that uses pure reward signals to guide autoregressive generation of high-quality videos. • Extensive experiments on VBench show that our method achieves video quality comparable to baseline autoregressive models while maintaining competitive overall scores. 2 Related work Diffusion Models for Image Generation Denoising Diffusion Probabilistic Models (DDPM) pioneered modern image generation by casting generative modeling as a Markovian noising–denoising process whose reverse dynamics are learned with a simple, noise‑conditional score network 18 . While DDPMs achieve impressive sample quality, they rely on hundreds or thousands of sequential denoising steps. Denoising Diffusion Implicit Models (DDIM) alleviate this inefficiency by interpreting the learned stochastic process as a non‑Markovian deterministic ordinary differential equation, enabling much faster inference without retraining and preserving visual fidelity 50 . Subsequent flow‑matching methods such as the Flow Matching framework further unify diffusion and continuous‑normalizing‑flow viewpoints by directly training vector fields that map noise to data in a single pass, often reducing both training variance and sampling cost 30 ; 33 . The most widely used architecture for such tasks was UNet 44 , which was later revolutionized by the introduction of DiT 41 . Due to the high computational cost of operating directly in pixel space, modern architectures typically first project the input into a latent space 43 ; 1 , where subsequent computations are performed. The final results are then decoded back into pixel space using a decoder 27 . Figure 1 : Overview of our proposed pipeline. The method first leverages a small set of ODE-based trajectories generated by a teacher model to guide the learning of motion dynamics. Subsequently, a reward model is employed to enhance the generation with fine-grained texture details. Video Diffusion Models Video diffusion models represent a significant advancement in generative AI, extending the principles of image diffusion models to handle temporal dynamics for tasks such as text-to-video generation, image-to-video synthesis, and video editing. Early extensions to video, such as Video Diffusion Models 19 , introduced 3D U-Net architectures with factorized spatio-temporal attention to ensure temporal coherence in video output with relative position embeddings 46 . Subsequent models like Imagen Video 17 employed cascaded diffusion pipelines for high-definition videos, while Make-A-Video 49 leveraged pretrained text-to-image models and unsupervised video data to learn visual appearance and motion without requiring paired text-video data. By introducing novel spatial-temporal modules and a multi-stage video generation pipeline, these methods achieve state-of-the-art results in video quality, temporal consistency, and alignment with textual input. Latent space approaches, exemplified by models such as Stable Video Diffusion 1 ; 2 , enhanced efficiency by operating on compressed representations and incorporating temporal attention mechanisms. More recent innovations include zero-shot methods like Text2Video-Zero 25 , which enable video generation without video-specific training, and Sora 39 , utilizing Diffusion Transformers on spacetime patches for scalable, high-fidelity outputs. These developments address challenges in temporal consistency and computational demands, paving the way for applications in entertainment, simulation, and content creation. The quality of generation is further enhanced by following models such as Wan 54 , HunyuanVideo 29 , and Veo 15 . Diffusion Acceleration Although diffusion models are able to perform high quality image and video synthesis, they suffer from extremely high denoising steps with high cost. In order to solve the problem, one direction is to design better ODE solvers which enable few-step sampling such as DPM-Solver 35 , GENIE 12 , and S4S 14 . An alternative approach involves distillation techniques to compress multi-step diffusion processes

Read the complete text