A Modular Zero-Shot Pipeline for Accident Detection, Localization, and Classification in Traffic Surveillance Video

Amey Thakur, Sarvesh Talele · 2026

Read A Modular Zero-Shot Pipeline for Accident Detection, Localization, and Classification in Traffic Surveillance Video by Amey Thakur, Sarvesh Talele (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09685.txt.

Introduction

Thakur and Talele describe a zero-shot pipeline for the ACCIDENT at CVPR 2026 challenge. It predicts when, where, and what kind of accident appears in surveillance video, without labeled training data. Three modules split the task. Peak detection over z-score-normalized frame-difference signals locates the collision in time. The centroid of optical-flow maps finds the impact location. Cosine similarity between CLIP image embeddings and multi-prompt text descriptions classifies the collision type. The implementation is public.

Catalog record

CREATORAmey Thakur, Sarvesh Talele
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18530
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.09685

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Modular Zero-Shot Pipeline for Accident Detection, Localization, and Classification in Traffic Surveillance Video Amey Thakur, Sarvesh Talele Abstract We describe a zero-shot pipeline developed for the ACCIDENT @ CVPR 2026 challenge. The challenge requires predicting when, where, and what type of traffic accident occurs in surveillance video, without labeled real-world training data. Our method separates the problem into three independent modules. The first module localizes the collision in time by running peak detection on z-score normalized frame-difference signals. The second module finds the impact location by computing the weighted centroid of cumulative dense optical flow magnitude maps using the Farneback algorithm. The third module classifies collision type by measuring cosine similarity between CLIP image embeddings of frames near the detected peak and text embeddings built from multi-prompt natural language descriptions of each collision category. No domain-specific fine-tuning is involved; the pipeline processes each video using only pre-trained model weights. Our implementation is publicly available as a Kaggle notebook ( 8 ) . Keywords Zero-Shot Learning \cdot Accident Detection \cdot Traffic Surveillance \cdot CLIP \cdot Optical Flow 1 Introduction Road traffic crashes kill over one million people each year according to the World Health Organization ( 9 ) . Surveillance cameras installed at intersections and along highways record continuous footage that captures the circumstances of many of these incidents. If this footage could be analyzed automatically, emergency dispatchers would receive faster alerts and investigators would gain access to objective scene reconstructions. Yet most current detection methods require supervised training on annotated video from the same deployment environment ( 6 ; 10 ) , which makes them expensive to transfer across camera installations with different viewpoints, lighting, and traffic patterns. The ACCIDENT @ CVPR 2026 competition ( 4 ) tests whether accident analysis can work without any labeled real-world data. Participants receive only synthetic videos generated by the CARLA simulator ( 1 ) for development (see Figure 1 ); the test set is composed of real CCTV recordings and manual annotation of it is prohibited. Three predictions are required per video: (1) the accident time in seconds, (2) normalized spatial coordinates of the impact point, and (3) the collision type from a five-class taxonomy (head-on, rear-end, sideswipe, single-vehicle, t-bone). Scoring uses the harmonic mean of a Gaussian temporal similarity, a Gaussian spatial similarity, and top-1 classification accuracy, so a poor prediction in any one dimension pulls the overall score down. We address each target with a separate module. Temporal localization runs statistical anomaly detection on pixel-intensity differences between consecutive frames. Spatial localization accumulates dense optical flow magnitudes and extracts a weighted centroid. Collision classification passes frames near the detected peak through CLIP ( 5 ) , a contrastive vision-language model trained on 400 million image-text pairs, and selects the class whose text prompts produce the highest cosine similarity. Because the modules are independent, any one of them can be swapped or tuned without modifying the rest. The full pipeline is implemented in a public Kaggle notebook ( 8 ) . Figure 1: Chronological sampled frames from a synthetic CARLA traffic incident in the ACCIDENT @ CVPR 2026 dataset. Most clips record approximately 18 seconds of motion at 20 FPS resolution. 2 Method 2.1 Problem Formulation A test video V has N frames recorded at f frames per second. The task is to produce a tuple (t^{*},c_{x}^{*},c_{y}^{*},k^{*}) : t^{*} is the accident time in seconds, (c_{x}^{*},c_{y}^{*})\in[0,1]^{2} are normalized image coordinates of the point of impact, and k^{*}\in\mathcal{K} is the collision type. The label space is \mathcal{K}=\{\texttt{head-on},\texttt{rear-end},\texttt{sideswipe},\texttt{single},\texttt{t-bone}\} . We split the problem into three parts. Time and location predictions rely on classical signal processing; type classification relies on a pre-trained vision-language model. 2.2 Temporal Peak Detection Collisions produce sudden intensity changes between consecutive frames. We exploit this observation by building a one-dimensional signal from per-frame brightness differences and then looking for statistical outliers in that signal. Let I_{t}\in\mathbb{R}^{H\times W} be the grayscale frame at index t , resized to H=180 , W=320 for speed. We compute the mean absolute difference between adjacent frames: d_{t}=\frac{1}{HW}\sum_{u=1}^{H}\sum_{v=1}^{W}\big|I_{t+1}(u,v)-I_{t}(u,v)\big| (1) for t=1,\ldots,N{-}1 . This series contains both collision signals and background noise from camera shake and ordinary traffic. A centered rolling mean with window w=5 suppresses the short-lived noise: \bar{d}_{t}=\frac{1}{|W_{t}|}\sum_{s\in W_{t}}d_{s} (2) where W_{t}=\{s:|s-t|\leq\lfloor w/2\rfloor\} . We normalize the smoothed values into z-scores using the series-wide mean \mu and standard deviation \sigma : z_{t}=\frac{\bar{d}_{t}-\mu}{\sigma+\varepsilon} (3) with \varepsilon=10^{-8} to avoid division by zero. Any frame whose z_{t} exceeds a threshold \tau is treated as an anomaly candidate. Among all candidates, we select the one with the strongest anomaly score: t^{*}_{\text{frame}}=\begin{cases}\arg\max_{t:\,z_{t}>\tau}\,z_{t}&\exists\,t\text{ s.t. }z_{t}>\tau\\ \arg\max_{t}\,z_{t}&\text{otherwise}\end{cases} (4) with \tau=1.5 . Taking the highest-scoring candidate among threshold-crossers favors the most prominent motion event. When no frame crosses the threshold, we fall back to the global maximum. Time in seconds is t^{*}=t^{*}_{\text{frame}}/f . Figure 2 shows the raw frame-difference series and its z-score transform for a synthetic head-on collision. The raw signal (top panel) contains a mixture of gradual intensity drift from vehicle motion and a sharp transient near the collision moment. After smoothing and normalization (bottom panel), the collision event stands out as a distinct peak while background variation stays below the detection threshold. Figure 2: Temporal localization on a synthetic CARLA video. Top: mean absolute frame difference d_{t} across all frames. Bottom: z-score anomaly series z_{t} after rolling-mean smoothing. The dashed line marks the detection threshold. 2.3 Spatial Impact Localization A collision concentrates high-magnitude motion in a small region of the image. We identify that region by accumulating dense optical flow over a short window and then computing the weighted centroid of the resulting magnitude map. When the predicted accident time t^{*} is available from the temporal module, we center a 30-frame window on the corresponding frame index; otherwise, we fall back to starting at frame \lfloor N/3\rfloor . For each pair of consecutive frames within this window, we run the Farneback dense optical flow algorithm ( 2 ) at 320\times 180 resolution. Farneback estimates per-pixel displacement using quadratic polynomial expansions and a multi-scale Gaussian pyramid. We set pyramid scale to 0.5 , three levels, window size 15 , three iterations, and polynomial parameters n=5 , \sigma_{p}=1.2 . Let \mathbf{f}_{t}(u,v)=(f_{x},f_{y}) be the flow vector at pixel (u,v) between frames t and t{+}1 . We sum the displacement magnitudes: M(u,v)=\sum_{t}\sqrt{f_{x}^{\,2}(u,v,t)+f_{y}^{\,2}(u,v,t)} (5) Before extracting the centroid, we apply a percentile threshold at the 90th percentile of M and zero out all values below it. This suppresses diffuse background motion and retains only the high-magnitude cluster near the collision site. The impact location is the weighted centroid of the thresholded map, normalized to unit coordinates: c_{x}^{*}=\frac{1}{W}\,\frac{\sum_{u,v}v\cdot M(u,v)}{\sum_{u,v}M(u,v)},\quad c_{y}^{*}=\frac{1}{H}\,\frac{\sum_{u,v}u\cdot M(u,v)}{\sum_{u,v}M(u,v)} (6) If \sum M<10^{-6} , we return the frame center (0.5,0.5) . This centroid calculation is a special case of the image moment framework introduced by 3 , applied to flow magnitudes instead of pixel intensities. Figure 3 shows the accumulation result on a synthetic video. The collision region concentrates most of the optical flow energy into a compact spatial cluster. After percentile thresholding, the weighted centroid falls within that cluster and provides a localized impact coordinate. Figure 3: Cumulative Farneback optical flow magnitude M(u,v) on a synthetic CARLA video after 90th-percentile thresholding. The bright region

Read the complete text