Video-based Vehicle Surveillance in the Wild: License Plate, Make, and Model Recognition with Self Reflective Vision-Language Models

Pouya Parsa Department of Civil, Environmental, and Geo- Engineering University of Minnesota [email protected] Keya Li Department of Civil, Architectural and Environmental Engineering The University of Texas at Austin [email protected] Kara M. Kockelman, Ph.D., P.E. Department of Civil, Architectura · 2025

Read Video-based Vehicle Surveillance in the Wild: License Plate, Make, and Model Recognition with Self Reflective Vision-Language Models by Pouya Parsa Department of Civil, Environmental, and Geo- Engineering University of Minnesota [email protected] Keya Li Department of Civil, Architectural and Environmental Engineering The University of Texas at Austin [email protected] Kara M. Kockelman, Ph.D., P.E. Department of Civil, Architectura (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01387.txt.

Introduction

Pouya Parsa, Keya Li, Kara M Kockelman, and Seongjin Choi test vision-language models for reading vehicles in moving video. Their 2025 paper covers license plates plus vehicle make and model. Fixed cameras make this task easier, while phone video adds motion, odd angles, blocked views, and unknown road shapes.

Their pipeline first picks sharp frames, then sends prompts to a large vision model. A second step lets the model reflect by comparing the query image with a reference from a large class set. Older systems relied on special hardware and hand-built text reading. The authors ask whether a general model can handle the messy real-world case. The work matters for tolls, policing, and crash follow-up, where video often comes from bystanders rather than fixed posts.

Catalog record

CREATORPouya Parsa Department of Civil, Environmental, and Geo- Engineering University of Minnesota [email protected] Keya Li Department of Civil, Architectural and Environmental Engineering The University of Texas at Austin [email protected] Kara M. Kockelman, Ph.D., P.E. Department of Civil, Architectura
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-01387

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Video-based Vehicle Surveillance in the Wild: License Plate, Make, and Model Recognition with Self Reflective Vision-Language Models Pouya Parsa Department of Civil, Environmental, and Geo- Engineering University of Minnesota [email protected] Keya Li Department of Civil, Architectural and Environmental Engineering The University of Texas at Austin [email protected] Kara M. Kockelman, Ph.D., P.E. Department of Civil, Architectural and Environmental Engineering The University of Texas at Austin [email protected] Seongjin Choi, Ph.D., Corresponding Author Department of Civil, Environmental, and Geo- Engineering University of Minnesota [email protected] Abstract \AuthorHeaders Parsa, Li, Kockelman, and Choi \TotalWords 5390 1 Abstract Automatic license plate recognition (ALPR) and vehicle make and model recognition underpin intelligent transportation systems, supporting law enforcement, toll collection, and post-incident investigation. Applying these methods to videos captured by handheld smartphones or non-static vehicle-mounted cameras presents unique challenges compared to fixed installations, including frequent camera motion, varying viewpoints, occlusions, and unknown road geometry. Traditional ALPR solutions, dependent on specialized hardware and handcrafted OCR pipelines, often degrade under these conditions. Recent advances in large vision–language models (VLMs) enable direct recognition of textual and semantic attributes from arbitrary imagery. This study evaluates the potential of VLMs for ALPR and makes and models recognition using monocular videos captured with handheld smartphones and non-static mounted cameras. The proposed license plate recognition pipeline filters to sharp frames, then sends a multimodal prompt to a VLM using several prompt strategies. Make and model recognition pipeline runs the same VLM with a revised prompt and an optional self-reflection module. In the self-reflection module, the model contrasts the query image with a reference from a 134-class dataset, correcting mismatches. Experiments on a smartphone dataset collected on the campus of the University of Texas at Austin, achieve top-1 accuracies of 91.67% for ALPR and 66.67% for make and model recognition. On the public UFPR-ALPR dataset, the approach attains 83.05% and 61.07%, respectively. The self-reflection module further improves results by 5.72% on average for make and model recognition. These findings demonstrate that VLMs provide a cost-effective solution for scalable, in-motion traffic video analysis. Keywords : Automatic license plate recognition, vision–language models, CLIP image quality assessment, prompt engineering, traffic law enforcement, automated enforcement, self-reflection 2 Introduction Nearly 1.2 million people die each year globally in road traffic crashes [ 1 ] , with over 40,000 fatalities occurring in the United States alone in 2023 [ 2 ] . Speeding, reckless driving, and hit-and-run collisions are major contributors to crashes and fatalities, yet the lack of real-time enforcement tools makes preventing them especially difficult. Stationary cameras, while effective, are very expensive to install and maintain, averaging over $120,000 per location [ 3 ] . Many nations, states, and cities are reluctant to adopt them due to their limited coverage from fixed placements and concerns over privacy. Moreover, drivers often slow down near enforcement points (and speed up downstream, once out of camera view), a phenomenon known as the “kangaroo effect” [ 4 ] . A scalable alternative lies in enabling citizens to assist enforcement by capturing video footage of infractions using smartphones or dashcams. However, leveraging such crowd-sourced data requires robust tools for analyzing low-quality and personal videos, which is a significant technical challenge. Several nations have developed public reporting platforms to facilitate citizen-driven traffic enforcement. For example, in South Korea, citizens can use the official ‘‘Safety e-Report’’ service 1 1 1 https://www.safetyreport.go.kr/eng/ to report traffic violations by uploading video evidence directly to law enforcement. In return, some users may receive monetary rewards or recognition, effectively turning smartphones into distributed enforcement tools and encouraging community participation in road safety [ 5 , 6 ] . In New York City, citizens have been helping enforce diesel-truck idling laws; those submitting 3 minutes of video receive 25 percent of any fine obtained from heavy-truck owners, which is close to $87.50 [ 7 ] . Such low-cost programs cost-effectively extend enforcement reach while building social accountability among drivers. Despite their success, these systems still require users to manually enter important vehicle information, such as the license plate number, make, and model. In many cases, this information is added after the video upload is complete, or left out entirely when the plate is difficult to read. This extra step increases the burden on users and reduces the number of reports that can be successfully verified. Automating the extraction of this kind of information directly from the video would improve ease of use, minimize input errors, and increase the likelihood of valid reports for enforceable actions. To enable automatic traffic law enforcement from citizen-recorded videos, two core recognition tasks are essential: Automated license plate recognition (ALPR) and vehicle make (manufacturer) and model recognition . These tasks serve as the foundation for identifying and tracking offending vehicles across time and locations. ALPR allows for the extraction of unique vehicle identifiers, enabling citation issuance, database cross-referencing, and ownership tracing. Recognizing the make and model of a vehicle provides an additional layer of verification or confirmation, and is especially valuable in cases of partial plate visibility, occlusion, or modified plates. Together, these two tasks form the minimum viable input required for automating enforcement actions from unstructured video evidence. ALPR, also known as Automatic Number Plate Recognition (ANPR), is a basic intelligent transportation system (ITS) technology for automated enforcement, enabling the identification of vehicles involved in traffic violations, toll evasion, and criminal activities. It can play a critical role in issuing citations, locating stolen vehicles, and monitoring access points (to paid parking lots, tollways, and high-security events). However, traditional ALPR systems depend on high-resolution cameras and controlled environments, making them unreliable when applied to noisy footage from standard smartphones or legacy closed-circuit television (CCTV) camera systems. In such cases, license plates may occupy only a few pixels or appear blurred or occluded, reducing the effectiveness of conventional optical character recognition (OCR) systems. An illustrative comparison is provided in Figure 4.2.3 , contrasting an ideal high-resolution US license plate with a low-resolution, blurry plate often seen in real-world video footage. Figure 1 : Comparison of ideal and real-world low-quality license plates Beyond plate recognition, extracting vehicle make and model further enhances enforcement capabilities by supporting cross-verification, vehicle re-identification, and suspect profiling. But this task typically relies on separate classifiers that are sensitive to viewpoint variations and trained on limited datasets, making them brittle and hard to scale. Conventional approaches [ 8 , 9 , 10 ] , for ALPR typically rely on dedicated OCR engines applied to tightly cropped plate regions, while make and model recognition is handled by specialized CNN-based classifiers trained on curated vehicle datasets [ 11 ] , [ 12 ] . These pipelines often assume clean, high-resolution imagery and controlled viewing conditions, which rarely hold in citizen-sourced or dashcam videos. In such scenarios, low resolution, motion blur, obstructions, and unusual angles significantly degrade the performance of traditional classification models. Moreover, maintaining and deploying multiple task-specific models increases system complexity and makes real-time, on-device processing difficult. A practical system therefore needs to be task - agnostic. In other words, it should take an entire video sequence as input and, without relying on separate specialist modules, return the plate text together with make and model. Vision–Language Models (VLMs) emerge as a potential solution to this challenge, as

Read the complete text