MoViE: Mobile Diffusion for Video Editing

Adil Karjauv, Noor Fathima, Ioannis Lelekas, Fatih Porikli, Amirhossein Habibian · 2024

Read MoViE: Mobile Diffusion for Video Editing by Adil Karjauv, Noor Fathima, Ioannis Lelekas, Fatih Porikli, Amirhossein Habibian (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06578.txt.

Introduction

Adil Karjauv and co-authors bring video editing to phones with a tuned diffusion model in a 2024 paper. Editing video with these models is powerful but heavy to run. Phones cannot carry the full cost of the desktop versions. Their work squeezes the model in three stacked moves.

A slimmer build and a light decoder came first in the chain. Guidance distillation across input types tripled on device speed next. A final trick cut the sampling to a single step while keeping control. The result edits twelve frames per second on a phone.

Catalog record

CREATORAdil Karjauv, Noor Fathima, Ioannis Lelekas, Fatih Porikli, Amirhossein Habibian
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06578

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MoViE: Mobile Diffusion for Video Editing Thanks: Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc Adil Karjauv, Noor Fathima, Ioannis Lelekas, Fatih Porikli, Amirhossein Habibian Abstract Recent progress in diffusion-based video editing has shown remarkable potential for practical applications. However, these methods remain prohibitively expensive and challenging to deploy on mobile devices. In this study, we introduce a series of optimizations that render mobile video editing feasible. Building upon the existing image editing model, we first optimize its architecture and incorporate a lightweight autoencoder. Subsequently, we extend classifier-free guidance distillation to multiple modalities, resulting in a threefold on-device speedup. Finally, we reduce the number of sampling steps to one by introducing a novel adversarial distillation scheme which preserves the controllability of the editing process. Collectively, these optimizations enable video editing at 12 12 frames per second on mobile devices, while maintaining high quality. Our results are available at https://qualcomm-ai-research.github.io/mobile-video-editing/ . Figure 1 : MoViE is a fast video editing model, capable of generating 12 frames per second on a mobile phone. It requires significantly fewer floating point operations (FLOPs) to edit a single video frame, making it computationally more efficient than competing methods. * * footnotetext: Equal Contribution † † footnotetext: Qualcomm AI Research is initiative of Qualcomm Technologies, Inc. Snapdragon and Qualcomm branded products are products of Qualcomm Technologies, Inc. and/or its subsidiaries. 1 Introduction Diffusion-based generative models have shown impressive quality in image editing hertz2022prompt ; tumanyan2023plug ; ip2p ; parmar2023zero . Building on these advancements, zero-shot video-based models extend these capabilities to the temporal dimension for video editing wu2023tune ; qi2023fatezero ; geyer2023tokenflow ; ceylan2023pix2video ; wu2024fairy ; liu2024video . This is typically achieved by developing techniques to propagate information between frames ensuring that changes made in one frame are coherently reflected in subsequent frames. While these techniques achieve remarkable results in style transfer and attribute editing, they are computationally expensive, making them prohibitive to run on constrained edge devices, e.g . mobile phones. While significant efforts have been made to reduce the computational cost of image diffusion models for on-device generation li2024snapfusion ; zhao2023mobilediffusion ; choi2023squeezing ; castells2024edgefusion , there has been little effort towards developing video editing models on edge, mainly due to the aforementioned high computational and memory costs. Inversion-based models qi2023fatezero ; geyer2023tokenflow ; kara2024rave that extract source video features using DDIM inversion incur high computational and memory costs during inference. On the other hand, methods that propagate features from one or more frames to the other, a process known as cross-frame attention wu2023tune , also remain expensive to run on-device, as each frame must attend to multiple frames. For example, editing a 120-frame video at a resolution of 512\times 384 with Fairy wu2024fairy , a fast video-to-video (V2V) model, takes 13.8 seconds on 8\times A100 GPUs. This is even higher for TokenFlow geyer2023tokenflow , an inversion-based V2V model, which takes 744 seconds wu2024fairy . In this work, we introduce the first on-device video editing model capable of generating 12 frames per second on Xiaomi-14 Pro. Xiaomi-14 Pro uses a Snapdragon ® 8 Gen. 3 Mobile Platform with a Hexagon TM processor. To achieve this, we begin with a conventional video editing pipeline. Broadly speaking, diffusion editing pipelines wu2023tune ; wu2024fairy first encode source frames into the latent space using a Variational AutoEncoder (VAE) pinheiro2021variational . A denoising UNet model then iteratively denoises latents to generate edited latents, which are then decoded by the VAE decoder to produce the edited frames. In addition, many diffusion models use the classifier-free guidance technique to balance quality and diversity. Specifically, editing models utilize two guidance scales — image and text — which can be adjusted to trade off how strongly the generated image correspond with the source image and the edit instructions, respectively. In this work, we identify the factors causing computational bottlenecks within an editing pipeline. First, we observed that the architectures of both the denoising UNet and the VAE encoder/decoder are major bottlenecks in the editing pipeline. Second, while classifier-free guidance technique enhances the model’s controllability, it increases the computational cost three times as we need 3 forward passes per diffusion step to obtain the output. Lastly, the intrinsic design of diffusion models necessitates iterative denoising to edit frames. Each of denoising steps involves complex computations, making the generation process expensive. To address the aforementioned challenges, we propose a series of contributions: • We introduce the first on-device video-to-video diffusion model, capable of editing a 120 -frame video at a resolution of 512\times 384 on a mobile device in 9.6 seconds, achieving a generation rate of 12 fps. • We propose a method to jointly distill multiple guidance scales in the editing models, reducing the Number of Forward Evaluations (NFE) per diffusion step by 3 times. • We present an adjusted adversarial distillation recipe for reducing number of diffusion steps, ensuring the model’s controllability is maintained post-distillation while reducing NFE of denoiser by 10 times. Additionally, unlike some methods wu2023tune that can generate only a limited number of frames due to memory constraints, our model can operate on long videos. This is because our model generate video frames independent of the video length, and its complexity increases linearly with the number of frames. 2 Related work Video generation. Recent methods can be categorised into two groups. The first group enhances image models with temporal components, i.e. 3D convolutions, spatio-temporal transformers ho2022imagen ; ho2022video ; hong2022cogvideo ; singer2022make ; zhou2022magicvideo , for modeling temporal dimension at the expense of the inflated computational footprint, along with the need for extensive training on large and diverse collections of data. In the effort to overcome these hurdles, zero-shot models kahatapitiya2025object , rely on training-free techniques for boosting temporal consistency geyer2023tokenflow ; khachatryan2023text2video ; liu2024video ; qi2023fatezero ; zhang2023controlvideo . Common methods are, utilization of structural conditioning from source video zhang2023controlvideo ; chen2023control , injection of diffusion inversion features into the generation process geyer2023tokenflow ; qi2023fatezero and employing cross-frame attention using a set of key frames qi2023fatezero ; zhang2023controlvideo ; wu2023tune ; esser2023structure . Our method aligns with this direction but is computationally much lighter, making it suitable for low-end edge devices. Optimized diffusion. Optimizing diffusion mainly entails the reduction of required steps for traversing the probability path from pure noise to the denoised sample, as well as reducing the cost of each step. In line with the former are the utilization of higher order solvers zhang2022fast ; lu2022dpm ; lu2022dpmpp , the straightening of underlying ODE trajectories or direct mapping to data via Rectified Flows liu2022flow ; liu2022rectified ; zhu2025slimflow and consistency models lu2024simplifying ; song2023consistency ; song2023improved , respectively and the employment of progressive-step distillation li2024snapfusion ; salimans2022progressive ; meng2023distillation or adversarial training wang2022diffusion ; sauer2024fast ; sauer2025adversarial ; zhang2024sf for fewer or even single step diffusion. On the other hand, quantization he2024ptqd ; pandey2023softmax ; shang2023post and pruning li2024snapfusion ; choi2023squeezing have been used, as well as extensive research conducted on simplifying the denoiser itself dockhorn2023distilling ; kim2023bk ; habibian2024clockwork . Our work incorporates both architectural optimization and a reduction in the number of steps. While extending li2024snapfusion with multimodal distillation, we manage to address a limitation of sauer2024fast ; zhang2024sf , by restoring

Read the complete text