Confidence-Aware Tool Orchestration for Robust Video Understanding

Yangfan He, Yujin Choi, Jaehong Yoon · 2026

Read Confidence-Aware Tool Orchestration for Robust Video Understanding by Yangfan He, Yujin Choi, Jaehong Yoon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26904.txt.

Introduction

Yangfan He, Yujin Choi, and Jaehong Yoon present Robust-TO, a video understanding framework that distrusts damaged frames. Video reasoning models normally treat every frame as equally reliable. The authors name the resulting failure the Blind Trust Problem. Blur, glare, and occlusion can cut accuracy sharply on real-world tasks while the model stays unaware its evidence degraded.

Robust-TO routes questions to specialist perception tools through one evidence interface. A reliability-relevance score picks trustworthy frames for each tool. Every tool returns a prediction, its time span, and a calibrated reliability score. Reasoning then weights evidence in three trust tiers. A training reward balances correctness, evidence reliability, and cost together. Across two benchmarks spanning eight tasks, the framework held up far better under realistic damage than standard models.

Catalog record

CREATORYangfan He, Yujin Choi, Jaehong Yoon
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.26904

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Confidence-Aware Tool Orchestration for Robust Video Understanding Yangfan He, Yujin Choi, Jaehong Yoon Abstract Video reasoning language models implicitly assume that every input frame is equally reliable. This leads to what we term the Blind Trust Problem : under realistic perturbations such as motion blur, glare, or occlusion, frontier video reasoning models can suffer 15-30%p accuracy drops on real-world embodied benchmarks, while remaining unaware that their visual evidence has been degraded. To address this challenge, we propose Robust-TO, an agentic video understanding framework that explicitly integrates per-frame trustworthiness into every stage of reasoning. Robust-TO organizes heterogeneous visual perception tools under a unified evidence interface. Each tool receives a sub-query derived from the original question and a set of trustworthy frames selected by the reliability-relevance score. It returns evidence in a shared format: a concrete prediction (e.g., a bounding box, motion trajectory, recognized text, or action label), temporal grounding, and a calibrated reliability score. During reasoning, these calibrated scores guide evidence weighting in a three-tier synthesis process (high/medium/low) and define a confidence-cost GRPO reward that jointly optimizes correctness, evidence reliability, and efficiency. On two video reasoning benchmarks spanning eight tasks, Robust-TO achieves 56.4% average accuracy on clean inputs, surpassing the strongest open-source baseline by 10.6%p and outperforming Gemini-2.5-Pro (46.2%). Under five realistic corruption types, Robust-TO maintains 54.3% average accuracy, 5.8%p above the strongest open-source baseline, while exhibiting the smallest clean-to-corrupted accuracy drop among all compared methods. 1 Introduction When asked which car ran the red light in a video with a smudged windshield and motion blur, a careful observer does not respond immediately. They first decide which moments are worth trusting, lean on the clearest evidence, and re-examine ambiguous segments only if they remain uncertain. The decision proceeds in stages: first, a quick scan identifies interpretable segments; next, a focused examination inspects relevant details (e.g., faces, license plates, traffic-light state); and only if these are insufficient, the model deliberately revisits challenging frames with a clearer sense of what to seek. Contemporary video large language models (Video-LLMs) ( 10 ; 23 ; 3 ; 14 ) largely omit these processes. They typically rely on uniform frame sampling ( 3 ; 10 ; 23 ) , encode the sampled frames through a vision backbone, and produce answers without explicitly evaluating whether the underlying visual evidence is sufficiently reliable to justify the prediction. This design collapses three fundamentally distinct capabilities into a one-step feed-forward prediction. We formalize this implicit assumption as the Blind Trust Problem : every frame is treated as equally informative, every perception output as equally reliable, and the model’s confidence in its answer is decoupled from the visual conditions that produced it. Its cost is well documented and silent: recent benchmarks ( 11 ; 28 ; 1 ; 9 ) show frontier video reasoning models losing 15-30%p on UrbanVideo under common corruptions, while their self-reported confidence remains largely unchanged. Zhang et al. ( 29 ) highlight that scaling parameters and data alone do not necessarily lead to robustness. In safety-critical settings such as forensic video analysis, surveillance review, or post-hoc autonomous driving analysis, this silent failure is precisely the mode that must be eliminated. To address this problem, we introduce Robust-TO, an agentic video understanding framework that performs robust reasoning on real-world videos through adaptive visual tool use guided by frame-wise reliability estimates. The pipeline proceeds in three stages (see Figure 2 ): (1) Frame Selection via Quality Profiling : we design a parameter-free assess_quality tool (see Table 9 ) that characterizes each frame’s degradation in terms of blur, brightness deviation, and occlusion, producing a disturbance profile. This profile captures both the dominant corruption type and its severity . Frames are then jointly scored by reliability and query relevance, filtering out corrupted yet query-relevant distractors, and retaining the top- K trustworthy frames for downstream perception. (2) Confidence-Guided Tool Routing : To obtain fine-grained perceptual evidence that addresses each distinct aspect of the query, we decompose the input query into atomic sub-queries and route each sub-query to the perception tool best suited to the dominant corruption observed in the selected frames. Every tool call returns a (result, confidence) pair, where confidence is computed as the product of the tool’s intrinsic certainty and the estimated reliability of its input frames, thereby down-weighting tool outputs derived from degraded inputs during reasoning. (3) Video Reasoning with Reliability-Aware Evidence : (result, confidence) pairs collected across all sub-query tool calls are grouped into three reliability tiers (high/medium/low) to synthesize the final answer; high-tier evidence drives the conclusion, medium-tier one is retained only if consistent, and low-tier one is considered only when no stronger evidence is available, with residual uncertainty explicitly reported in the final answer. The host VLM is trained end-to-end with Group Relative Policy Optimization (GRPO) ( 22 ) . The training reward combines four signals: (i) correctness reward that scores answer accuracy; (ii) confidence-cost reward that encourages high-confidence outputs while penalizing expensive tool calls on degraded frames; (iii) sub-query efficiency reward that penalizes both over- and under-decomposition of the query, anchored to a VLM-estimated target count; and (iv) format reward . We demonstrate the effectiveness of the proposed Robust-TO on UrbanVideo-Bench ( 30 ) and VSI-Bench ( 26 ) , spanning eight tasks under both clean and corrupted conditions generated via RoVA ( 9 ) . On clean benchmarks, Robust-TO with Qwen3-VL-7B achieves 56.4% average accuracy, surpassing Gemini-2.5-Pro (46.2%) and the supervised fine-tuned Qwen2.5-VL-7B (45.8%). On corrupted UrbanVideo-Bench, Robust-TO achieves an average accuracy of 54.3%, outperforming the strongest open-source baseline, Video-R1 ( 6 ) , by 5.8%p and the best proprietary model, Gemini-2.5-Pro, by 16.2%p. Moreover, the adaptive key-frame selector reduces the average number of processed frames by 35% (from 32 to 20.7) and cuts per-sample inference time by over 35%, while simultaneously improving accuracy by 1.6%p ( Table 5 ). These results highlight that the adaptive visual tool use in Robust-TO, guided by per-frame reliability estimates, benefits both corrupted and clean videos. 2 Related Work Video Large Language Models. Recent Video-LLMs extend strong image-language pretraining to the video domain by explicitly modeling temporal information. LLaVA-OneVision ( 10 ) demonstrates that a single architecture can transfer image instruction tuning to video; InternVL ( 4 ) , Qwen2/2.5-VL ( 23 ; 3 ) introduce dynamic FPS sampling, window attention, and absolute time embeddings to support hour-long inputs; earlier work such as Video-LLaVA ( 14 ) unifies image and video representations using a shared projector. These models achieve top results on Video-MME ( 7 ) , MVBench ( 12 ) , EgoSchema ( 16 ) , and NExT-QA ( 25 ) . They all assume the visual signal is clean: frames are uniformly or densely sampled, and the pipeline never questions whether a given frame deserves attention. Robust-TO inherits this design as a limiting cas when frames are clean, it works exactly the same; when they are not, noise at the frame level alters the entire reasoning path. Agentic and Tool-Augmented Video Reasoning. Researchers have also explored video understanding as iterative information acquisition. ReAct ( 27 ) and Toolformer ( 20 ) establish the paradigm of interleaving reasoning traces with tool calls; ToolLLM ( 18 ) scales this idea to thousands of APIs. In the video domain,

Read the complete text