Video Unlearning via Low-Rank Refusal Vector
Simone Facchiano, Stefano Saravalle, Matteo Migliarini, Edoardo De Matteis, Alessio Sampieri, Andrea Pilzer, Emanuele Rodolà, Indro Spinelli, Luca Franco, Fabio Galasso · 2025
Read Video Unlearning via Low-Rank Refusal Vector by Simone Facchiano, Stefano Saravalle, Matteo Migliarini, Edoardo De Matteis, Alessio Sampieri, Andrea Pilzer, Emanuele Rodolà, Indro Spinelli, Luca Franco, Fabio Galasso (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07891.txt.
Introduction
Simone Facchiano and colleagues removed unsafe concepts from video generators without retraining in a 2025 paper. Video models trained on web data can produce harmful content, and unlearning methods either filter outputs, which can be bypassed, or fine-tune weights at high cost. Their method needs only five paired prompts, safe and unsafe versions of one idea, to estimate a refusal vector in the model's internal space.
That vector is folded into the weights in closed form, and a low-rank contrastive step keeps the edit focused on the target concept so unrelated quality survives. On two open video models and two safety benchmarks, unsafe generations fell by large average margins while prompt fidelity held. Cheap, surgical edits suit fast-moving model releases.
Catalog record
| CREATOR | Simone Facchiano, Stefano Saravalle, Matteo Migliarini, Edoardo De Matteis, Alessio Sampieri, Andrea Pilzer, Emanuele Rodolà, Indro Spinelli, Luca Franco, Fabio Galasso |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-07891
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Video Unlearning via Low-Rank Refusal Vector Simone Facchiano, Stefano Saravalle, Matteo Migliarini, Edoardo De Matteis, Alessio Sampieri, Andrea Pilzer, Emanuele Rodolà, Indro Spinelli, Luca Franco, Fabio Galasso Abstract Video generative models democratize the creation of visual content through intuitive instruction following, but they also inherit the biases and harmful concepts embedded within their web-scale training data. This inheritance creates a significant risk, as users can readily generate undesirable and even illegal content. This work introduces the first unlearning technique tailored explicitly for video diffusion models to address this critical issue. Our method requires 5 multi-modal prompt pairs only. Each pair contains a “safe” and an “unsafe” example that differ only by the target concept. Averaging their per-layer latent differences produces a “refusal vector”, which, once subtracted from the model parameters, neutralizes the unsafe concept. We introduce a novel low-rank factorization approach on the covariance difference of embeddings that yields robust refusal vectors. This isolates the target concept while minimizing collateral unlearning of other semantics, thus preserving the visual quality of the generated video. Our method preserves the model’s generation quality while operating without retraining or access to the original training data. By embedding the refusal direction directly into the model’s weights, the suppression mechanism becomes inherently more robust against adversarial bypass attempts compared to surface-level input-output filters. In a thorough qualitative and quantitative evaluation, we show that we can neutralize a variety of harmful contents, including explicit nudity, graphic violence, copyrights, and trademarks. Project page: https://www.pinlab.org/video-unlearning . 1 Sapienza University of Rome, Italy 2 ItalAI 3 NVIDIA Warning : This paper contains data and model outputs which are offensive in nature. Figure 1: Using just 5 image-prompt pairs, our method computes a low-rank refusal vector from latent differences to suppress unwanted concepts (e.g., logos, nudity). It preserves video quality and unrelated content (e.g., skateboarding) without retraining or degrading model capabilities. 1 Introduction Generative video diffusion models have rapidly gained popularity as industrial solutions, enabling the creation of high-fidelity, text-conditioned clips for applications ranging from virtual cinematography to interactive simulation. These models are trained on massive, uncurated video corpora to learn rich motion and semantics, so they inevitably capture unsafe concepts such as explicit nudity, graphic violence, or recognizable copyrighted characters. This unwanted material may emerge unpredictably at inference time. Rather than filtering, harmful concepts should be amended directly from the model’s parameters, selectively neutralizing its ability to generate unwanted content at the source. Machine unlearning in generative models seeks to remove a target concept from a pretrained model while preserving its overall quality and broad semantic knowledge. Traditional approaches, such as data curation or retraining on filtered datasets, are effective but prohibitively costly and impractical at web scale. Fine-tuning approaches can mitigate unwanted behaviors but often induce catastrophic forgetting ( mukhoti2023fine ; french1999catastrophic ; sai2024machine ) of unrelated concepts or, under benign updates, allow the “resurgence” of erased concepts ( suriyakumar2024unstable ) . To avoid these pitfalls, recent work in text-to-image diffusion models has focused on training-free, weight-editing unlearning techniques. Notably, UCE gandikota2024unified applies closed-form edits to textual cross-attention layers to suppress unwanted activations, and RECE gong2024rece iteratively computes and replaces unsafe latent embeddings in cross-attention with harmless proxies. UCE and RECE were designed for image - based diffusion architectures and do not generalize to video generators, leaving video unlearning as an important but unexplored area in machine learning research. We present the first training-free framework for targeted concept removal in video diffusion models. Our method requires only 5 curated unsafe-safe input pairs differing exclusively in the presence of the unwanted concept. Inspired by the steering approach in nanda2023emergent , we adapt “refusal vectors” to unlearning task by averaging per-layer activation differences across few semantically relevant intermediate layers, as demonstrated in recent literature nanda2023emergent . We propose a novel method to refine the refusal vector with a contrastive PCA (cPCA) factorization abid2018cpca . We first identify the directions strongly correlated with the target concept and weakly correlated with everything else. Then we project to a lower-dimensional subspace that captures the unwanted concept in isolation. Concept removal is then performed strictly within this subspace, leaving the rest of the latent space, and thus the model’s broader semantics, unchanged. The refined refusal vector is utilized to suppress the concept without additional training by embedding the refusal direction directly into the weight matrices. Moreover, a single scalar hyperparameter \lambda provides control over the unlearning intensity to balance mild attenuation against complete removal. Our approach maintains the model’s original fidelity and generalization across diverse prompts, offering a practical, efficient, and controllable tool for enabling safe video generation. We conduct a thorough experimental evaluation to support our claims, providing both quantitative and qualitative results across multiple categories of unwanted concepts, which we quantitatively measure the efficacy of concept removal on the T2VSafetyBench benchmark miao2024tvsafetybench . The impact of our method on video fidelity and quality is assessed using the Fréchet Video Distance (FVD) ( unterthiner2019fvd ) . We analyze key technical aspects of our method, including an ablation on the dimension of the low-rank factorization of the “refusal vector”, showing that only 0.8\% of the original space is needed. A key contribution, the use of both text and image conditioning inputs for concept extraction (as opposed to text alone), is shown to preserve overall generation quality and prevent the forgetting of unrelated semantics. We also show the precise control offered by our method, showing that the model’s response to the \lambda coefficient is smooth and monotonic, enabling fine-grained suppression tuning. We summarize our core contributions in these points: • First Training-Free Framework for Video Concept Removal: We propose the first method capable of targeted concept removal specifically in video diffusion models without requiring expensive retraining or fine-tuning. • Novel Low-Rank Refusal Vector Method: We introduce a new technique to derive a "refusal vector" based on the difference of covariance matrices of safe and unsafe embeddings, using low-rank factorization to isolate the target concept and minimize collateral damage. • Leveraging Multimodal Data for Concept Extraction: We show that using both text and image prompts for concept extraction improves effectiveness, preserves overall generation quality, and prevents forgetting unrelated semantics compared to using text alone (as supported by experiments). 2 Related Works Video Generation. Transformer-based models (e.g., CogVideo ( hong2023cogvideo ) , CogVideoX ( yang2025cogvideox ) ) and diffusion-based pipelines (e.g., ImagenVideo ( ho2022imagenvideohighdefinition ) , Make - A - Video ( singer2022makeavideotexttovideogenerationtextvideo ) , ModelScopeT2V ( wang2023modelscope ) ) have each demonstrated remarkable progress in generating high - fidelity videos from text prompts. These architectures are trained at massive scale—often involving billions of parameters and extensive compute—and deliver realistic motion, sharp details, and strong prompt adherence. OpenAI’s Sora ( liu2024sorareviewbackgroundtechnology ) (released February 2024) pioneered high - quality video diffusion. Its open - source counterpart Open-Sora ( opensora ) matches Sora - level performance: human evaluations report parity with leading proprietary systems (e.g., RunwayGen - 3 ( runwaygen3 ) , HunyuanVideo ( kong2024hunyuanvideo ) ) despite an order - of - magnitude reduction in training cost. In summary, state - of - the - art text - to - video generators produce longer, more coherent, and visually compelling clips than ever before, but only by leveraging very