Video-GPT via Next Clip Diffusion

Shaobin Zhuang † Shanghai Jiao Tong University, Zhipeng Huang † University of Science and Technology of China, Ying Zhang WeChat Vision, Tencent Inc., Fangyikang Wang † Zhejiang University, Canmiao Fu WeChat Vision, Tencent Inc., Binxin Yang WeChat Vision, Tencent Inc., Chong Sun WeChat Vision, Tenc · 2025

Read Video-GPT via Next Clip Diffusion by Shaobin Zhuang † Shanghai Jiao Tong University, Zhipeng Huang † University of Science and Technology of China, Ying Zhang WeChat Vision, Tencent Inc., Fangyikang Wang † Zhejiang University, Canmiao Fu WeChat Vision, Tencent Inc., Binxin Yang WeChat Vision, Tencent Inc., Chong Sun WeChat Vision, Tenc (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12489.txt.

Introduction

Shaobin Zhuang and co-authors teach a model to treat video as a language, in a 2025 paper. Text models like GPT predict the next word; video, though, carries the space and motion that words miss. Their Video-GPT predicts the next clip instead. It cleans up a noisy clip by leaning on the clean clips before it, one step at a time, which serves both short generation and long-range forecasting.

On a physics-based video prediction benchmark, the model scored well above two strong commercial rivals. It also adapted to six mainstream video tasks, spanning making and understanding clips. The authors argue next-clip training is a clean path toward world models that grasp how scenes unfold. They present the design as simple but broadly capable.

Catalog record

CREATORShaobin Zhuang † Shanghai Jiao Tong University, Zhipeng Huang † University of Science and Technology of China, Ying Zhang WeChat Vision, Tencent Inc., Fangyikang Wang † Zhejiang University, Canmiao Fu WeChat Vision, Tencent Inc., Binxin Yang WeChat Vision, Tencent Inc., Chong Sun WeChat Vision, Tenc
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-12489

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Video-GPT via Next Clip Diffusion Shaobin Zhuang † Shanghai Jiao Tong University, Zhipeng Huang † University of Science and Technology of China, Ying Zhang WeChat Vision, Tencent Inc., Fangyikang Wang † Zhejiang University, Canmiao Fu WeChat Vision, Tencent Inc., Binxin Yang WeChat Vision, Tencent Inc., Chong Sun WeChat Vision, Tencent Inc., Chen Li WeChat Vision, Tencent Inc., Yali Wang Shenzhen Institutes of Advanced Technology, Chinese Academy of SciencesShanghai AI Laboratory Abstract GPT has shown its remarkable success in natural language processing. However, the language sequence is not sufficient to describe spatial-temporal details in the visual world. Alternatively, the video sequence is good at capturing such details. Motivated by this fact, we propose a concise Video-GPT in this paper by treating video as new language for visual world modeling. By analogy to next token prediction in GPT, we introduce a novel next clip diffusion paradigm for pretraining Video-GPT. Different from the previous works, this distinct paradigm allows Video-GPT to tackle both short-term generation and long-term prediction, by autoregressively denoising the noisy clip according to the clean clips in the history. Extensive experiments show our Video-GPT achieves the state-of-the-art performance on video prediction, which is the key factor towards world modeling (Physics-IQ Benchmark: Video-GPT 34.97 vs. Kling 23.64 vs. Wan 20.89 ). Moreover, it can be well adapted on 6 mainstream video tasks in both video generation and understanding, showing its great generalization capacity in downstream. The project page is at https://zhuangshaobin.github.io/Video-GPT.github.io/ . $\dagger$ $\dagger$ footnotetext: Work done as interns at WeChat Vision, Tencent Inc. 🖂 🖂 footnotetext: Corresponding author. 1 Introduction Over the past few years, natural language processing has been mainly driven by training large language models (LLMs) from web-scale text data. In particular, GPT series [ 57 , 58 , 8 , 2 ] have demonstrated remarkable generalization capabilities on various tasks, which further opens the possibility towards general artificial intelligence. However, the language sequence is often good at expressing high-level abstractions, while it may not be sufficient to capture rich spatial-temporal details in the visual world [ 42 , 88 ] . For example, it is difficult to describe how to tie a knot by language. Alternatively, the video sequence has been considered as a preferred candidate to describe such details, since it can record visual knowledge of our dynamical world at different spatial and temporal resolutions [ 3 , 42 ] . Hence, there is a natural question: can we treat video as new language for visual world modeling? The recent studies have shown that, video generation is a promising direction to achieve this goal [ 49 , 9 , 34 ] . The typical design is video diffusion by adding noise and denoising gradually [ 69 , 25 ] . Although such a paradigm has achieved significant progress [ 5 , 35 , 36 , 74 ] , it often suffers from difficulty in long-term future prediction that is a critical factor of world modeling [ 50 , 38 ] . To address this problem, the autoregressive attempts have been made for long-context video modeling, on analogy of next token prediction in LLMs [ 34 , 42 , 3 ] . However, compared with its diffusion counterpart, the generation performance of this paradigm still needs to be further improved. In order to leverage advantages from both paradigms, several approaches combine diffusion and autogregressive modeling in a unified transformer style [ 83 , 16 , 94 , 30 , 28 ] . But this design mainly works on the image domain, without insightful analogy between language and video. To fill this gap, we propose a concise Video-GPT, by analogy with generative pretraining design in GPT. Inspired by next token prediction in GPT, we introduce a novel next clip diffusion paradigm for our Video-GPT. Specifically, we creatively treat a clip in the video as the role of a word in the language, since both of them describe local temporal information, respectively, in the video and language sequence. However, different from a discrete word token, it is often challenging to predict a continuous video clip in the next step. To address this difficulty, we introduce a flexible diffusion design within an interleaved sequence of noisy and clean clips in the temporal order. As shown in Fig. 1 , one can randomly sample a number of clips from a training video. For each clip, we generate its noisy clip by adding noise on it, based on the forward process of diffusion. Then, we re-arrange noisy and clean clips in the temporal order, constructing an interleaved clip sequence. Consequently, for each noisy clip, we leverage the clean clips in the previous as temporal context, and reconstruct the corresponding clean clip by the diffusion loss. In this case, our Video-GPT inherits advantages from both GPT and diffusion, but within the basic unit of a video clip. This allows it to capture long-term video prediction as well as short-term video generation for effective generative pretraining on videos. Our contributions can be summarized as follows. First , we introduce a concise Video-GPT, by analogy to GPT. Different from GPT, our Video-GPT aims at modeling rich spatial-temporal information in the visual world. Second , we design a novel next clip diffusion paradigm for pretraining Video-GPT. Different from the previous works mentioned above, this hybrid paradigm allows Video-GPT to tackle both short-term generation and long-term prediction, via integrating diffusion and autoregressive design in the video clip level. Finally, our pretrained Video-GPT achieves state-of-the-art performance on Physics-IQ Benchmark and Kinetics-600, fully demonstrating its potential for visual world modeling. After fine-tuning on 6 downstream video generation and understanding tasks, it also achieved preferable performance for generalization. Figure 1: Next clip diffusion. We draw an analogy with GPT’s next token prediction and model each video clip as a visual word by denoising the next noisy clip, conditioning on the previous video. 2 Related Work Video Diffusion Models. With diffusion models demonstrating remarkable performance in image generation [ 65 , 62 , 17 , 60 ] , video diffusion models [ 26 ] also surpasses traditional methods [ 15 , 66 , 31 ] in video generation. With the emergence of large-scale text–video datasets [ 13 , 78 , 81 ] , text-to-video diffusion models [ 7 , 96 , 89 , 49 , 74 , 35 , 80 ] exhibit realistic generation capabilities and strong transferability to downstream tasks [ 93 , 29 ] . However, the supervised paradigm based on text annotations faces two challenges as it scales up. 1) Quantity of Annotated Data : Current methods require meticulous data filtering and annotation processes [ 46 , 56 ] , resulting substantially smaller data scale compared to LLM [ 18 , 87 , 32 ] . 2) Quality of Annotated Data : Text cannot fully capture the meaning of a video. While increasing the granularity of descriptions [ 12 , 6 , 67 ] alleviates this issue, the difficulty of annotating such descriptions significantly increases as video length scales up. Autoregressive Video Models. ImageTransformer [ 51 ] migrated the next token prediction approach to image generation in pixel space. Chameleon [ 72 ] and LlamaGen [ 71 ] perform pretraining on images in latent space. LVM [ 3

Read the complete text