RAIN: Real-time Animation of Infinite Video Stream
Zhilei Shu, Ruili Feng, Yang Cao, Zheng-Jun Zha · 2024
Read RAIN: Real-time Animation of Infinite Video Stream by Zhilei Shu, Ruili Feng, Yang Cao, Zheng-Jun Zha (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19489.txt.
Introduction
Zhilei Shu and colleagues made live animation from diffusion models practical in a 2024 system called RAIN. Streamers and virtual hosts want endless, steady animated video, but diffusion generators are slow and drift in quality over long runs, especially on home hardware. RAIN rethinks how attention is shared across frames at different noise levels and across long time spans.
By denoising many more frame tokens at once, it cuts delay while keeping long-range consistency. A standard image diffusion model, fine-tuned briefly with RAIN, then streams video in real time on a single high-end consumer card, with little loss in quality. Streams can in principle run without limit. Live creators gain a cheaper path to stable animation.
Catalog record
| CREATOR | Zhilei Shu, Ruili Feng, Yang Cao, Zheng-Jun Zha |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-19489
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RAIN: Real-time Animation of Infinite Video Stream Zhilei Shu, Ruili Feng, Yang Cao, Zheng-Jun Zha Abstract Live animation has gained immense popularity for enhancing online engagement, yet achieving high-quality, real-time, and stable animation with diffusion models remains challenging, especially on consumer-grade GPUs. Existing methods struggle with generating long, consistent video streams efficiently, often being limited by latency issues and degraded visual quality over extended periods. In this paper, we introduce RAIN, a pipeline solution capable of animating infinite video streams in real-time with low latency using a single RTX 4090 GPU. The core idea of RAIN is to efficiently compute frame-token attention across different noise levels and long time-intervals while simultaneously denoising a significantly larger number of frame-tokens than previous stream-based methods. This design allows RAIN to generate video frames with much shorter latency and faster speed, while maintaining long-range attention over extended video streams, resulting in enhanced continuity and consistency. Consequently, a Stable Diffusion model fine-tuned with RAIN in just a few epochs can produce video streams in real-time and low latency without much compromise in quality or consistency, up to infinite long. Despite its advanced capabilities, the RAIN only introduces a few additional 1D attention blocks, imposing minimal additional burden. Experiments in benchmark datasets and generating super-long videos demonstrating that RAIN can animate characters in real-time with much better quality, accuracy, and consistency than competitors while costing less latency. All code and models will be made publicly available. Figure 1 : Cross domain face morphing generation results. We achieve real-time animation of anime characters. Expression of real human can be successfully ported to anime characters, and the generation is stable, consistency and infinite long. 1 Introduction Live animation has emerged as a powerful tool for enhancing online engagement, bringing characters, avatars, and digital personas to life in real-time. Its growing significance is evident across various domains, from entertainment and gaming to virtual influencers and live-streaming platforms. By enabling dynamic, interactive experiences, live animation fosters more immersive and personalized connections, making it increasingly valuable for social media, online communication, and digital content creation. This demand for engaging real-time animation has sparked interest in developing diffusion models, the most successful image and video generative neural networks, to create smooth, vivid, and responsive animations, especially in applications that require extended live shows or continuous interaction. Despite its potential, achieving high-quality, real-time, and stable live animation with diffusion models remains a challenging task, especially when relying on consumer-grade hardware with limited computational power. Current animation methods often require several minutes to generate just a few seconds of video and are incapable of continuously synthesizing long videos that extend for several hours, as commonly needed in practical applications. Consequently, these limitations render most existing animation methods impractical for real-world live animation scenarios. Recent advances in stream-based diffusion models and diffusion acceleration methods have provided promising pathways toward addressing the challenges of real-time live animation. These methods leverage the multi-step generation nature of diffusion models, allowing stream-based diffusion models to maintain a \mathrm{StreamBatch} of frame tokens corresponding to the number of denoising steps. Each token in this \mathrm{StreamBatch} is incrementally injected with noise levels corresponding to its position, enabling efficient handling of stream inputs and outputting frames in a continuous, streaming fashion. This approach, particularly when combined with acceleration techniques, has significantly improved the speed of frame generation to reach real-time levels. However, the generation continuity and quality of this process is constrained by the size of the \mathrm{StreamBatch} , which is typically limited by the number of denoising steps. Since most acceleration methods sample in fewer than 4 steps, stream-based diffusion models often fail to fully utilize the computational power of even consumer-grade GPUs, thereby limiting their overall performance. Additionally, the relatively small size of the \mathrm{StreamBatch} hinders the model’s ability to compute attention over longer time intervals, which is essential for maintaining continuity in generated video streams. This limitation results in less influence from one frame to the next, reducing the fluidity and consistency of the animation over extended durations. Consequently, existing stream-based methods often struggle with maintaining seamless animation, resulting in latency issues or degraded visual quality, especially during long-duration outputs. In response to these challenges, this paper introduces RAIN, a pipeline solution designed to achieve real-time animation of infinite video streams using consumer-grade GPUs. Unlike previous methods that restrict the \mathrm{StreamBatch} size to match the number of denoising steps, RAIN expands this size by a factor of p=\frac{\mathrm{GPU\,Capcacity}}{\mathrm{Denoising\,Steps}} by assigning every p consecutive frame tokens into denoising groups that share the same noise level, and gradually increasing the noise level across these groups. This expansion fully utilizes the computational potential of available hardware and enables the model to capture much longer-range temporal dependencies by allowing attention over a larger sequence of frame tokens, significantly improving the consistency and continuity of the generated video streams. Additionally, while previous methods avoided cross-noise-level attention, we find that it works effectively when combined with the long-range attention across different denoise groups, where each group shares the same noise level. This synergy between long-range attention and cross-noise-level attention significantly boosts continuity and visual quality. By integrating these key elements RAIN achieves substantial improvements in real-time video generation, delivering superior visual quality and consistency over prolonged animations. Figure 2 : Animation clips for crossdomain face morphing. Best viewed with Acrobat Reader. Click the images to play the animation clips. 2 Related Work Motion Transfer Motion transfer aims at generating image of a character with a set of driven poses. GAN-based methods , Tulyakov et al. [40] , Siarohin et al. [34] , Siarohin et al. [35] , Lee et al. [19] , usually tried to inject pose information into intermediate representation of image inside GAN and synthesized edited images. However, due to limitations of GAN itself (e.g. instability and mode collapse), the generated contents suffer from blur, inaccuracy and poor quality. Since the recent success of diffusion model in visual content generation, many works [ 43 , 47 , 13 , 4 ] have delved into the possibility of motion transfer on diffusion model. Since the diffusion model itself has a series of external peripherals, like ControlNet [ 51 ] , we can easily make use of different kinds of control signal, like depth, pose, normal and semantic maps. In Wang et al. [43] , the input is disentangle into character, motion and background, then the authors make exploits of a ControlNet-like structure and add an additional motion module to keep inter-frame continuity. In Hu et al. [13] , Xu et al. [47] , video diffusion model is leveraged for better temporal consistency and two-stage training ensure the disentanglement of motion control, character identity and temporal consistency. These method generally have a long video generation algorithm, that is, performing denoising on overlapped adjacent temporal batches and averaging the overlapped area after every denoising step, or performing generation in a autoregressive manner. However, it requires additional computation for the overlapped frames, while also this is not suitable for the first-in-first-out condition of live stream. Stream Video Processing For downstream tasks such as live broadcast and online conferences, it is important that the model can process streaming input with infinite length in real-time. StreamDiffusion Kodaira et al. [17]