Astraea : A GPU-Oriented Token-wise Acceleration Framework for Video Diffusion Transformers

Haosong Liu, Yuge Cheng, Zihan Liu, Aiyue Chen, Jing Lin, Yiwu Yao, Chen Chen, Jingwen Leng, Yu Feng, Minyi Guo · 2025

Read Astraea : A GPU-Oriented Token-wise Acceleration Framework for Video Diffusion Transformers by Haosong Liu, Yuge Cheng, Zihan Liu, Aiyue Chen, Jing Lin, Yiwu Yao, Chen Chen, Jingwen Leng, Yu Feng, Minyi Guo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05096.txt.

Introduction

Haosong Liu and co-authors speed up video generation by teaching a diffusion transformer to spend its effort only on the tokens that matter. These video models are powerful but slow, and earlier speedups relied on hand-set rules that transfer poorly. Their framework, Astraea, selects a subset of tokens at each step. It uses a sparse attention scheme built to run well on graphics hardware, cutting execution time in proportion to the tokens dropped. An evolutionary search then decides how many tokens each step can afford to lose.

The combined system ran up to about two and a half times faster on one graphics card, and over thirteen times faster across eight, with quality close to the full model. It also beat prior speedup methods on measured video quality by a wide margin. The authors stress that budgets are found by search per step, not guessed.

Catalog record

CREATORHaosong Liu, Yuge Cheng, Zihan Liu, Aiyue Chen, Jing Lin, Yiwu Yao, Chen Chen, Jingwen Leng, Yu Feng, Minyi Guo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05096

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Astraea : A GPU-Oriented Token-wise Acceleration Framework for Video Diffusion Transformers Haosong Liu, Yuge Cheng, Zihan Liu, Aiyue Chen, Jing Lin, Yiwu Yao, Chen Chen, Jingwen Leng, Yu Feng, Minyi Guo Abstract Video diffusion transformers (vDiTs) have made impressive progress in text-to-video generation, but their high computational demands present major challenges for practical deployment. While existing acceleration methods reduce workload at various granularities, they often rely on heuristics, limiting their applicability. We introduce Astraea , an automatic framework that searches for near-optimal configurations for vDiT-based video generation. At its core, Astraea proposes a lightweight token selection mechanism and a memory-efficient, GPU-parallel sparse attention strategy, enabling linear reductions in execution time with minimal impact on generation quality. To determine optimal token reduction for different timesteps, we further design a search framework that leverages a classic evolutionary algorithm to automatically determine the distribution of the token budget effectively. Together, Astraea achieves up to 2.4 × \times inference speedup on a single GPU with great scalability (up to 13.2 × \times speedup on 8 GPUs) while retaining better video quality compared to the state-of-the-art methods ( < < 0.5% loss on the VBench score compared to the baseline vDiT models). 1 Introduction Visual imagination has always been at the core of humanity’s nature for creativity. After the release of Sora by OpenAI in early [ 3 ] , there are numerous video generative frameworks from text input, including SenseTime’s Kling [ 2 ] , Google’s Veo [ 4 ] , and Tencent’s Hunyuan [ 1 ] . Despite the abundance of frameworks, video diffusion transformers (vDiTs) remain the core of these frameworks, widely regarded as the most effective paradigm due to its ability to generate high-fidelity videos. However, its computational demand poses a significant challenge for any industrial-level deployment. For instance, a 4-second 720 \times 1280 video clip on a Nvidia H100 GPU takes over 0.5 hours using Hunyuan [ 20 ] . This high computational cost comes from the extensive denoising steps and the large number of tokens . For example, a vDiT model often requires 50-100 denoising steps, with each step involving over millions of tokens. Various acceleration methods have been proposed to reduce computational workload at different granularities, such as denoising step reduction [ 21 , 35 , 14 ] , intermediate block caching [ 40 , 8 , 18 , 13 ] , token selection [ 43 ] , etc. However, these methods often require iteratively fine-tuning hyperparameters, i.e., which steps or blocks to skip, with a human in the loop to achieve a target performance in industrial-level deployments. Fundamentally, previous approaches primarily propose various acceleration heuristics, without addressing a key question: given a specific performance target, which tokens are worth computing at each denoising step to achieve optimal accuracy? To answer this question, we propose Astraea , a GPU-friendly acceleration framework that operates at the token level, the smallest primitive in vDiT models. Specifically, we propose a sparse diffusion inference with a lightweight token selection mechanism and GPU-efficient sparse attention to accelerate each denoising step by dynamically selecting important tokens. To determine optimal token budget allocation, we propose a search framework that determines the number of tokens that should be assigned in each denoising step to achieve the target performance. Algorithm. Our selection mechanism is largely aware of the extensive GPU memory usage in prior work [ 43 ] . Unlike prior studies, which require storing the entire attention map, our mechanism introduces negligible memory overhead by only caching previous token values. Thus, the memory requirement of our selection metric scales sub-linearly with the token length to avoid a memory explosion. In addition, we purposely design our sparse attention to be natively parallelizable, thus, it can be integrated with existing attention acceleration techniques, such as FlashAttention [ 11 , 10 ] . On existing GPUs, our sparse attention achieves linear execution delay with the number of input tokens. Search Framework. Although our algorithm can achieve linear acceleration for each denoising timestep, it is still unknown how many tokens should be allocated for individual timesteps. While numerous search techniques exist in the literature, such as neural architecture search [ 30 , 12 ] and network pruning [ 16 , 9 ] , none can be applied to vDiTs due to their substantial search times. Here, we propose a search framework based on the classic evolutionary search algorithms [ 31 , 5 ] . Our approach guarantees to achieve the target performance while achieving the minimal accuracy loss. Fig. 1: Diffusion models work by reversing a diffusion process, where they iteratively predict and remove noise at each timestep to gradually reconstruct the original images or videos. Compared to the state-of-the-art algorithms, Astraea achieves up to 2.4 \times speedup with better generation quality (less than 0.5% loss on VBench score). Also, Astraea easily scales to 13.2 \times speedup across 8 GPUs. Our contributions are summarized as follows: • We propose a lightweight token selection mechanism to reduce the computation workload of each denoising step with negligible latency and memory overhead. • We design a sparse attention computation method that achieves linear speedup on existing GPUs, as the number of selected tokens decreases. • We introduce a search framework that meets the target performance while achieving the minimal accuracy loss against prior studies. 2 Related Work Diffusion models have been widely used in video generation. It learns to generate data from Gaussian noise through a reverse Markov process (Fig. 1 ). The input of this diffusion process is a randomly generated Gaussian noise, x^{\prime}_{T} . The diffusion model recovers the original data by progressively predicting and removing noise, z^{\prime}_{t} , from x^{\prime}_{t} at each timestep t . The hope is that, through this process, the prediction of the diffusion model, x^{\prime}_{0} , can be close to the original data, x_{0} . Mathematically, the denoising step can be expressed as, x^{\prime}_{t-1}=\alpha_{t}(x^{\prime}_{t}-\beta_{t}z^{\prime}_{t})+\sigma_{t}n^{\prime}_{t},\ z^{\prime}_{t}=\Phi(x^{\prime}_{t},t), (1) where \Phi is the prediction function that predicts the noise z^{\prime}_{t} . Both \alpha_{t} and \beta_{t} are hyper-parameters. \sigma_{t}n^{\prime}_{t} is the renoising term to add randomness to the denoising step. Normally, diffusion models often require hundreds or even thousands of steps to denoise images or videos [ 34 ] . The common practice to reduce the diffusion workload is to encode the initial noising inputs into the latent space then decode the tokens at the end, as shown in Fig. 1 . Prior methods to enhance the efficiency of diffusion models fall into two categories: step reduction , which reduces the number of denoising steps, and block caching , which seeks to minimize the computational demands within each denoising step. The following paragraphs overview these two categories of acceleration techniques separately. Step Reduction. Overall, step reduction methods can be classified into two types: one requires retraining, and the other is training-free. The retraining methods [ 35 , 28 , 14 , 15 , 19 ] , such as distillation, can reduce the number of denoising steps to as few as one. However, the major downside of these methods is that they require as much training time as the original model training. For this reason, current distillation methods are primarily applied to image generation rather than video generation. For instance, Saliman

Read the complete text