VAU-R1: Advancing Video Anomaly Understanding via Reinforcement Fine-Tuning
Liyun Zhu, Qixiang Chen, Xi Shen, Xiaodong Cun · 2025
Read VAU-R1: Advancing Video Anomaly Understanding via Reinforcement Fine-Tuning by Liyun Zhu, Qixiang Chen, Xi Shen, Xiaodong Cun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23504.txt.
Introduction
Reinforcement learning sharpens a language model's reasoning, but scoring a whole answer with one number wastes information about which parts helped. Liyun Zhu and co-authors introduce VAU-R1 in this 2025 paper, a framework that teaches models to reason about strange events in video. It fine-tunes a multimodal model with reinforcement learning on question answering about anomalies.
Alongside it, the team built VAU-Bench, a test set with multiple-choice questions, written rationales, time stamps, and captions for abnormal scenes. Their method improved answer accuracy, the timing of when events were located, and the coherence of the reasoning across varied settings. The pair aims to make video anomaly tools explain themselves, for uses from city safety to disaster alerts.
Catalog record
| CREATOR | Liyun Zhu, Qixiang Chen, Xi Shen, Xiaodong Cun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23504
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VAU-R1: Advancing Video Anomaly Understanding via Reinforcement Fine-Tuning Liyun Zhu, Qixiang Chen, Xi Shen, Xiaodong Cun Abstract Video Anomaly Understanding (VAU) is essential for applications such as smart cities, security surveillance, and disaster alert systems, yet remains challenging due to its demand for fine-grained spatio-temporal perception and robust reasoning under ambiguity. Despite advances in anomaly detection, existing methods often lack interpretability and struggle to capture the causal and contextual aspects of abnormal events. This limitation is further compounded by the absence of comprehensive benchmarks for evaluating reasoning ability in anomaly scenarios. To address both challenges, we introduce VAU-R1 , a data-efficient framework built upon Multimodal Large Language Models (MLLMs), which enhances anomaly reasoning through Reinforcement Fine-Tuning (RFT). Besides, we propose VAU-Bench , the first Chain-of-Thought benchmark tailored for video anomaly reasoning, featuring multiple-choice QA, detailed rationales, temporal annotations, and descriptive captions. Empirical results show that VAU-R1 significantly improves question answering accuracy, temporal grounding, and reasoning coherence across diverse contexts. Together, our method and benchmark establish a strong foundation for interpretable and reasoning-aware video anomaly understanding. Our code is available at https://github.com/GVCLab/VAU-R1 . Figure 1 : Effectiveness of Reinforcement Fine-Tuning. We compare QA accuracy and temporal anomaly grounding performance across different models. VAU-R1, trained via Reinforcement Fine-Tuning (RFT), consistently outperforms its Supervised Fine-Tuning (SFT) counterpart. This demonstrates that RFT enhances both reasoning and temporal localization capabilities in VAU tasks. 1 Introduction Figure 2 : Overview of VAU-R1. VAU-R1 leverages Reinforcement Fine-Tuning to enhance the reasoning ability of MLLMs for video anomaly understanding. Specifically, we adopt Group Relative Policy Optimization (GRPO) to optimize the model with task-specific rewards, such as answer format, accuracy, and temporal Intersection-over-Union (IoU). We decompose the VAU task into four complementary tasks to facilitate comprehensive reasoning: multiple-choice QA, temporal anomaly grounding, anomaly reasoning, and anomaly classification. Anomalies are events or behaviors that deviate from regular patterns or expected activities in a given context. In surveillance settings, these may include incidents such as fighting, theft, or traffic violations, etc . Video Anomaly Understanding (VAU) aims to detect and interpret such irregular events in unstructured, real-world video streams [ 22 ] . The task is challenging due to scene complexity, context dependence, varying camera viewpoints, and diverse anomaly types [ 44 , 27 , 56 ] . Early approaches only focuses on detecting anomalies, which typically framed the task as binary classification, assigning normal or abnormal labels to individual frames and identifying the temporal boundaries of anomalous events [ 11 , 21 , 36 , 47 , 16 , 55 , 33 , 5 , 6 ] . While effective for localization, these methods offer limited interpretability and provide little insight into the underlying causes of anomalies [ 49 , 7 , 8 ] . Recent advances in Multi-modal Large Language Models (MLLMs) have introduced the ability to generate textual descriptions of anomalous events [ 48 , 51 , 9 , 53 , 52 ] , improving model transparency to some extent. However, current methods still face three key limitations: (i) they lack the ability to generate coherent, multi-step reasoning chains; (ii) no comprehensive benchmark provides rich annotations to support detailed causal reasoning; and (iii) evaluation protocols for reasoning quality remain underdeveloped. To move beyond shallow classification and toward deeper understanding, we decompose VAU into four progressive stages: (i) Perception — identifying the scene and relevant objects, either through free-text descriptions or guided multiple-choice questions; (ii) Grounding — localizing the precise temporal segment where the anomaly occurs; (iii) Reasoning — explaining the event by analyzing causal factors, temporal dynamics, and contextual cues; and (iv) Conclusion — summarizing the event with a final decision, such as assigning it to a specific category (e.g., fighting vs. robbery ). This structured formulation enables models to progressively build semantic understanding and supports more interpretable and task-aligned evaluation. To implement this four-stage formulation, we introduce VAU-R1 , a Reinforcement Fine-Tuning (RFT) framework designed to improve the reasoning capabilities of MLLMs on the VAU task. Our method builds on Group Relative Policy Optimization (GRPO) [ 31 ] , incorporating task-specific reward signals based on answer format correctness, question-answer accuracy, and temporal grounding alignment. The framework is data-efficient and can be applied in low-resource settings, making it practical for real-world deployments. To support training and evaluation, we also construct VAU-Bench , a new benchmark that spans diverse scenarios and provides rich annotations across the four reasoning stages, including multiple-choice QA pairs, detailed event descriptions, temporal groundings, and step-by-step rationales. Finally, we propose a set of evaluation metrics—QA accuracy, temporal Intersection-over-Union (IoU), GPT-based reasoning score, and classification accuracy—to quantitatively assess model performance across perception, grounding, reasoning, and conclusion. Together, VAU-R1 and VAU-Bench offer a scalable and unified framework for advancing structured video anomaly understanding. Our contribution can be summarized as follows: • We propose VAU-R1 , a data-efficient Reinforcement Fine-Tuning framework that improves the reasoning ability of MLLMs for video anomaly understanding. It outperforms standard supervised fine-tuning on reasoning-intensive tasks. • We present VAU-Bench , the first large-scale benchmark with Chain-of-Thought annotations designed for video anomaly reasoning. It contains a diverse collection of videos, QA pairs, temporal labels, and detailed rationales spanning a wide range of real-world scenarios. • We design a unified evaluation protocol that measures model performance across four reasoning stages, jointly considering reasoning quality, answer correctness, and temporal localization to capture both interpretability and detection precision. 2 Related Works From Detection to Understanding. Early efforts in Video Anomaly Detection (VAD) can be broadly categorized into self-supervised and weakly-supervised paradigms. Self-supervised methods rely solely on normal video samples, learning the distribution of normal behavior and flagging deviations as anomalies [ 21 , 11 , 25 ] . In contrast, weakly-supervised methods are trained with both normal and anomalous videos using coarse video-level labels rather than fine-grained frame-level annotations [ 33 , 36 , 47 , 16 , 55 , 5 , 45 ] . These approaches typically adopt a top- k selection strategy to identify the most likely anomalous segments. While effective for localizing anomaly boundaries, they often rely heavily on motion cues [ 56 ] , operating under the assumption that rapid or irregular motion is indicative of anomalous behavior. However, this assumption does not hold for subtle or semantically complex anomalies, leading to poor interpretability. To address these limitations, recent work has turned to video anomaly understanding, leveraging MLLMs to provide more semantically grounded and interpretable reasoning [ 26 ] . Prompt-Based vs . Learning-Based Approaches for VAU. Building on the shift toward semantic understanding, recent approaches to VAU fall into two main categories: prompt-based and learning-based methods. Prompt-based methods typically use MLLMs as anomaly scorers [ 51 , 30 ] , or as reasoning agents via rule-based few-shot prompting [ 48 ] or learned question templates [ 49 ] . While these methods avoid computationally expensive training, their generalization ability is often limited due to the absence of task-specific adaptation. On the other hand, pretraining [ 8 ] and finetuning [ 53 , 52 ] approaches aim to learn anomaly-aware representations by incorporating video captions and causal reasoning signals ( e.g . , cause and effect). Despite this progress, existing methods remain constrained to improving anomaly description and fail to capture the full