On-device Sora: Enabling Training-Free Diffusion-based Text-to-Video Generation for Mobile Devices
Bosung Kim, Kyuhwan Lee, Isu Jeong, Jungmin Cheon, Yeojin Lee, Seulki Lee, Ulsan National Institute of Science, Technology · 2025
Read On-device Sora: Enabling Training-Free Diffusion-based Text-to-Video Generation for Mobile Devices by Bosung Kim, Kyuhwan Lee, Isu Jeong, Jungmin Cheon, Yeojin Lee, Seulki Lee, Ulsan National Institute of Science, Technology (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04363.txt.
Introduction
Bosung Kim and five co-authors bring text-to-video generation onto a phone in a 2025 paper. Video diffusion models are heavy: they need many cleanup steps and huge memory, far beyond a smartphone. Their system, which they call On-device Sora, adapts a pretrained model with no new training.
Three tricks make it fit. A leap method cuts the number of cleanup steps sharply. A merging method fuses similar frame tokens over time, shrinking the attention work. A loading method slices the model into blocks and streams them through memory in parallel, so the whole model never sits on the chip at once. Running on a recent high-end phone, the system produced videos judged comparable to those from big graphics cards. The authors frame it as a first step toward creative video tools that live on the device, not in a data center.
Catalog record
| CREATOR | Bosung Kim, Kyuhwan Lee, Isu Jeong, Jungmin Cheon, Yeojin Lee, Seulki Lee, Ulsan National Institute of Science, Technology |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-04363
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On-device Sora: Enabling Training-Free Diffusion-based Text-to-Video Generation for Mobile Devices Bosung Kim, Kyuhwan Lee, Isu Jeong, Jungmin Cheon, Yeojin Lee, Seulki Lee, Ulsan National Institute of Science, Technology Abstract We present On-device Sora, the first model training-free solution for diffusion-based on-device text-to-video generation that operates efficiently on smartphone-grade devices. To address the challenges of diffusion-based text-to-video generation on computation- and memory-limited mobile devices, the proposed On-device Sora applies three novel techniques to pre-trained video generative models. First, Linear Proportional Leap (LPL) reduces the excessive denoising steps required in video diffusion through an efficient leap-based approach. Second, Temporal Dimension Token Merging (TDTM) minimizes intensive token-processing computation in attention layers by merging consecutive tokens along the temporal dimension. Third, Concurrent Inference with Dynamic Loading (CI-DL) dynamically partitions large models into smaller blocks and loads them into memory for concurrent model inference, effectively addressing the challenges of limited device memory. We implement On-device Sora on the iPhone 15 Pro, and the experimental evaluations show that it is capable of generating high-quality videos on the device, comparable to those produced by high-end GPUs. These results show that On-device Sora enables efficient and high-quality video generation on resource-constrained mobile devices. We envision the proposed On-device Sora as a significant first step toward democratizing state-of-the-art generative technologies, enabling video generation on commodity mobile and embedded devices without resource-intensive re-training for model optimization (compression). The code implementation is available at a GitHub repository 1 1 1 https://github.com/eai-lab/On-device-Sora . 1 Introduction Recent advancements in generative models [ 60 ] have significantly expanded capabilities in data generation across various modalities, including text [ 27 ] , image [ 55 ] , and video [ 63 ] . In particular, diffusion-based models for image tasks [ 23 , 58 , 46 , 50 , 53 , 14 , 78 , 47 ] have emerged as foundational tools for a wide range of applications, such as image generation [ 25 ] , image editing [ 30 ] , and personalized content creation [ 79 ] . Further extending these technologies, diffusion-based generative models are now driving remarkable progress in video generation tasks [ 56 , 41 , 24 , 26 , 4 , 42 , 21 , 5 , 65 , 68 , 31 ] , including video synthesis [ 37 ] and real-time video analysis [ 44 ] . With the rapid expansion of mobile and embedded devices, there is an increasing demand for executing generative applications directly on-device. In response, extensive research has focused on developing on-device image generation methods [ 35 , 64 , 8 , 13 , 12 , 81 ] , with recent advancements extending to on-device video generation [ 71 , 70 ] . However, generating lightweight and compressed on-device video generation models necessitates additional model optimization (training), such as distillation [ 66 , 36 , 57 ] , quantization [ 62 , 80 ] , and pruning [ 69 ] , to reduce model size and computational complexity. Furthermore, mitigating performance degradation in the compressed models requires a time-intensive and iterative model re-training. This process usually demands substantial computational and memory resources, e.g . , SnapGen-V [ 70 ] takes over 150K training iterations on 256 NVIDIA A100 80GB GPUs to generate mobile-deployable compressed video generation models. We introduce On-Device Sora, the first training-free framework that enables diffusion-based text-to-video generation on mobile devices, which allows video generative models to run directly on-device, enabling the production of high-quality videos on smartphone-grade devices. While previous approaches [ 70 ] require additional training and substantial GPU resources to optimize video generation models for mobile environments, the proposed On-Device Sora eliminates the need for training and directly enables efficient on-device execution. Using pre-trained video generative models, e.g . , Open-Sora [ 83 ] , Pyramidal Flow [ 29 ] , On-device Sora significantly enhances their efficiency, enabling on-device video generation with limited resources. To achieve this, we address key challenges of enabling on-device text-to-video generation: 1) Linear Proportional Leap (LPL) reduces the iterative denoising steps in the diffusion process by leaping through a significant portion of steps using the Euler’s method [ 3 ] along an estimated direct trajectory, 2) Temporal Dimension Token Merging (TDTM) lowers computational complexity of STDiT (Spatial-Temporal Diffusion Transformer) [ 83 ] by merging consecutive tokens [ 6 , 7 ] in the attention layers, and 3) Conference Inference and Dynamic Loading (CI-DL) enables video generation with limited memory capacity of mobile devices by integrating concurrent model inference with the dynamic loading of models into memory. With these three proposed methods, high-quality video generation becomes feasible on smartphone-grade devices with limited computing resources, overcoming the requirements for substantial computational power, such as high-end GPUs. To the best of our knowledge, On-device Sora proposed in this work is the first training-free solution that enables the efficient generation of video directly on the device. The advantages of On-device Sora, i.e . , directly deploying well-trained video generative models onto resource-constrained devices without time-consuming model modification and/or re-training, are expected to become even more pronounced with the active development of compact DiT-based text-to-video generative models [ 46 ] . We implement On-device Sora on the iPhone 15 Pro [ 1 ] using Open-Sora [ 83 ] . The full implementation is available as open-source code in an anonymous GitHub 1 . The extensive experiments are conducted to evaluate the performance of On-device Sora using the state-of-the-art video benchmark tool, i.e . , VBench [ 28 ] , compared with NVIDIA A6000 GPUs. We also evaluate the proposed methods on additional text-to-video generative models, including Pyramidal Flow [ 29 ] . The experimental results demonstrate that On-device Sora maintains comparable video quality while accelerating generation speed with the proposed methods. While the iPhone 15 Pro has a GPU of 143 times less computational power [ 1 ] and 16 times smaller memory compared to the NVIDIA A6000 GPU, the evaluation results show that On-device Sora significantly improves the efficiency of video generation by effectively compensating for the limited computing resources of the device. 2 Background and Challenges We first provide a background of diffusion-based text-to-video generation ( e.g . Open-Sora [ 83 ] ), which is the backbone of the proposed On-device Sora, and key challenges in realizing on-device video generation for mobile devices. 2.1 Background: Diffusion-based Text-to-Video Generation In general, diffusion-based text-to-video generation models, such as Open-Sora [ 83 ] , generate videos from user prompts (texts) through the three stages: 1) prompt (text) embedding, 2) latent video generation, and 3) video decoding. 1) Prompt (Text) Embedding. The first stage of diffusion-based text-to-video generation is to map a user prompt, a textual description of the desired video, to an embedding vector, which is used as input for the subsequent video generation stage. For example, to produce prompt embeddings from user texts, Open-Sora employs T5 (Text-to-Text Transformer) [ 48 ] , a language model specifically fine-tuned to support video generation tasks. 2) Latent Video Generation. The next stage is to generate the latent video representation conditioned on the prompt embedding obtained from language models, e.g . , T5 (Text-to-Text Transformer) [ 48 ] . To this end, for instance, Open-Sora employs STDiT (Spatial-Temporal Diffusion Transformer) [ 83 ]