WildRoadBench : A Wild Aerial Road-Damage Grounding Benchmark for Vision–Language Models and Autonomous Agents
Bingnan Liu, Chenhang Cui, Rui Huang, Jiani Luo, Zhirong Shen, [0.4em] Tinghao Wang, Xiande Huang, Lingbei Meng, Fei Shen, An Zhang, [0.9em] University of Electronic Science, Technology of China, [0.4em] Equal contribution. · 2026
Read WildRoadBench : A Wild Aerial Road-Damage Grounding Benchmark for Vision–Language Models and Autonomous Agents by Bingnan Liu, Chenhang Cui, Rui Huang, Jiani Luo, Zhirong Shen, [0.4em] Tinghao Wang, Xiande Huang, Lingbei Meng, Fei Shen, An Zhang, [0.9em] University of Electronic Science, Technology of China, [0.4em] Equal contribution. (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20306.txt.
Introduction
Bingnan Liu and nine co-workers set out WildRoadBench, road damage tests on one marked UAV set. Fixed vision-language models point to damage from one photo, short prompt, under one shared routine. Agents get a brief, a slice, and set budget, hunt the web, adapt parts, write code, and file guesses through a score oracle on hidden data. Closed models lead yet leave over half unclaimed; open models plateau lower, small marks failing for all. Agents trail the best model, some filing no valid guess in budget.
Catalog record
| CREATOR | Bingnan Liu, Chenhang Cui, Rui Huang, Jiani Luo, Zhirong Shen, [0.4em] Tinghao Wang, Xiande Huang, Lingbei Meng, Fei Shen, An Zhang, [0.9em] University of Electronic Science, Technology of China, [0.4em] Equal contribution. |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57069 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20306
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
WildRoadBench : A Wild Aerial Road-Damage Grounding Benchmark for Vision–Language Models and Autonomous Agents Bingnan Liu, Chenhang Cui, Rui Huang, Jiani Luo, Zhirong Shen, [0.4em] Tinghao Wang, Xiande Huang, Lingbei Meng, Fei Shen, An Zhang, [0.9em] University of Electronic Science, Technology of China, [0.4em] Equal contribution. Abstract We introduce WildRoadBench , a wild aerial road-damage grounding benchmark that couples direct visual grounding by vision–language models with autonomous research-and-engineering by LLM-driven agents on a single professionally annotated UAV corpus. The same image set and the same per-class AP 50 metric are evaluated under two protocols. The VLM Track measures whether a fixed VLM can localise domain-specific damage from one image and one short prompt under a unified prompting, decoding and parsing pipeline. The Agent Track measures whether an autonomous agent, given only a written task brief, a small exploratory slice and a fixed interaction budget, can search the public web, adapt pretrained components, write training and inference code, and submit predictions through a scalar-feedback oracle on a hidden holdout. We benchmark a broad pool of close-sourced frontier models and open-source VLMs together with several frontier LLM-driven agents. Both routes remain far from reliable performance in this wild setting: close-sourced frontier models lead the VLM leaderboard but still leave more than half of the metric on the table; open-source grounders plateau well below them, and newer generations or reasoning-style variants do not consistently improve grounding; small targets collapse for every open-source model; agents lag the strongest VLM despite richer affordances, and several fail to land a valid submission within the budget. We release the code and data in https://github.com/liangyunuancha/wildroadbench to support reproducible follow-up research. 1 Introduction Visual grounding is becoming a central test of whether multimodal systems can move from language-level understanding to physical-world action: an embodied agent, inspection platform or maintenance assistant must not only describe an image, but also localise the object, defect or region that requires intervention. Recent progress in large vision–language models (LVLMs) such as GPT-5 [ 21 ] , Claude [ 1 ] , Qwen3.5 [ 23 ] and InternVL3 [ 32 ] has made this goal appear increasingly attainable; in parallel, LLM-driven agents have begun to write code, use tools, search the web and assemble task-specific pipelines on software-engineering, web-navigation and data-analysis tasks. This raises two related questions about grounding. First, if a system is used as a fixed VLM, can it localise the relevant regions directly from an image and a text prompt? Second, if the same localisation problem is presented as an autonomous research-and-engineering task, can an LLM-driven agent use examples, tools, external resources and metric feedback to build a working training and inference pipeline under a fixed budget? In both cases the required output is the same kind of object-level prediction—bounding boxes with confidence scores—but the route to producing those predictions is different: one relies on the pretrained VLM’s existing grounding ability, while the other relies on the agent’s ability to construct a detector for the task. We instantiate these two questions in wild aerial road-damage grounding . The task is built from UAV imagery used for road-maintenance inspection [ 33 , 7 , 9 ] , with damage regions annotated by road-maintenance experts following highway-maintenance specifications. The setting contains small, visually subtle and domain-specific defects that may be confused with shadows, water stains, seams, lane markings or ordinary asphalt texture. It therefore tests whether current systems can localise professional inspection concepts in noisy real-world aerial imagery, either through a fixed VLM query or through an autonomous detector-building workflow. WildRoadBench packages this setting as a benchmark with two evaluation settings on the same 1,061-image corpus and the same AP 50 scoring [ 8 ] (Section 4 ). In the VLM Track , a VLM receives one image and one target-scene prompt, performs a single forward pass, and returns a list of bounding boxes with confidence scores. This isolates the localisation ability of a fixed multimodal model without examples, tools or task-specific training. In the Agent Track , an LLM-driven agent receives a written task document, the target label schema and a small set of annotated samples; inside an isolated sandbox it may search the public web, inspect external datasets, download pretrained detectors, install dependencies, complete training and inference code, and submit predictions. The grader returns only a top-line mAP 50 score, with limited time budget. Across twenty-five VLMs (sixteen open-source releases plus nine close-sourced frontier models) and 15 frontier LLM-driven agents, both evaluation settings reveal substantial gaps on this wild localisation problem. The strongest VLM is Gemini 3 Pro [ 10 ] at 42.1 % macro AP 50 ; the strongest open-source VLM, Qwen2.5-VL-32B-Instruct-AWQ [ 4 ] , reaches only 25.5 % macro AP 50 ; the strongest autonomous agent, Claude Opus 4.7 achieves the strongest agent-track result (0.1676) after five attempts in five hours. Model scaling and newer generations do not consistently translate into better localisation—some newer VLMs produce many more false positives, and reasoning-style variants can perform worse than their non-reasoning counterparts—and small targets remain especially challenging. For agents, the benchmark further exposes the difficulty of turning task descriptions, external resources and sparse metric feedback into a working aerial road-damage detector. This paper makes three contributions. First , we introduce WildRoadBench , a wild aerial road-damage grounding benchmark with professionally annotated ground truth, built from a 1,061-image UAV corpus. Second , we define a unified two-track evaluation protocol that studies grounding from two complementary perspectives: direct VLM grounding, and automated research-and-engineering by LLM-driven agents on the same perception task. Both tracks use the same image set and evaluation metric, while the agent track is implemented through a controlled harness that manages the workspace, tool use, interaction budget, submissions, and evaluation. Third , we benchmark 25 VLMs (16 open-source + 9 close-sourced models) and 15 frontier LLM-driven agents, and show that wild grounding remains challenging for both multimodal perception and autonomous agent workflows. Figure 1 : WildRoadBench at a glance. A single 1,061-image corpus of professionally-annotated UAV road-damage imagery is graded under two protocols: the VLM Track evaluates grounding VLMs in a single forward pass against a schema (released with images and labels), while the Agent Track evaluates autonomous LLM-driven agents against the underlying schema as a hidden holdout; agents may browse the web, train detectors and submit up to five times within a 5 h wall-clock. The grader returns COCO-style AP 50 . 2 Related Work Grounding and road-damage benchmarks. RefCOCO/+/g [ 13 , 30 ] , Visual Genome [ 15 ] and recent spatial probes such as DORI [ 24 ] , SSI-Bench [ 25 ] and MMSI-Bench [ 27 ] evaluate grounding on ground-level or internet imagery, where reported performance is often high but the setting differs substantially from aerial road inspection. Road-damage datasets such as RDD2022 [ 2 ] focus on ground-level smartphone imagery, while aerial datasets such as VisDrone [ 34 ] cover pedestrians and vehicles rather than road-surface defects. Together, these benchmarks provide strong coverage of grounding, road damage, and aerial perception, but they do not jointly evaluate grounding on aerial road-surface defects or compare VLMs and LLM-driven agents on the same localisation task. Autonomous LLM agents on perception tasks. Recent agent frameworks such as SWE-Agent