Adaptive Dense Evidence Refinement for Video Relational Reasoning for VRR-QA Challenge
Yuyang Sun, Yongliang Wu, Xingyu Zhu, Yuxia Chen, Zhenxiang Jiang, Yangguang Ji, Wenbo Zhu, Yanxi Shi, Jay Wu, Shuo Wang, Xu Yang · 2026
Read Adaptive Dense Evidence Refinement for Video Relational Reasoning for VRR-QA Challenge by Yuyang Sun, Yongliang Wu, Xingyu Zhu, Yuxia Chen, Zhenxiang Jiang, Yangguang Ji, Wenbo Zhu, Yanxi Shi, Jay Wu, Shuo Wang, Xu Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01104.txt.
Introduction
Sun and co-authors spend heavy compute only on the video questions that need it in this 2026 paper. The VRR-QA challenge asks systems to infer relations in video: space, time, viewpoint, depth, and who can see what. One frame rarely settles such questions. A flat, heavy pass over every item wastes effort and can still miss the proof.
Their system starts cheap. A direct video model answers each question in one pass. Light extra views then flag the unstable items, whose answers flip between looks. Only those hard cases go to a dense evidence module. That module builds timestamped frame notes, runs relation specific probes, and vets candidate answers. A conservative temporal merge guards the final call. The design splits two jobs often confused: finding rival answers, and deciding when a change is earned. On the test split, the system obtains 90.07 in average accuracy.
Catalog record
| CREATOR | Yuyang Sun, Yongliang Wu, Xingyu Zhu, Yuxia Chen, Zhenxiang Jiang, Yangguang Ji, Wenbo Zhu, Yanxi Shi, Jay Wu, Shuo Wang, Xu Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01104
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adaptive Dense Evidence Refinement for Video Relational Reasoning for VRR-QA Challenge Yuyang Sun, Yongliang Wu, Xingyu Zhu, Yuxia Chen, Zhenxiang Jiang, Yangguang Ji, Wenbo Zhu, Yanxi Shi, Jay Wu, Shuo Wang, Xu Yang Abstract VRR-QA evaluates whether video-language systems can infer spatial, temporal, viewpoint, depth, and visibility relations that are not always resolved by a single frame. We present an inference-only system built around adaptive test-time computation. The system first answers each question with a direct video-language model pass, then uses multiple lightweight views to find unstable questions. Only these difficult questions are routed to a high-budget dense evidence module that constructs timestamped frame observations, relation-specific probes, candidate verification, and conservative temporal aggregation. This design separates two problems that are often confused in video question answering: finding plausible alternative answers and deciding when a current answer should actually be changed. On the test split, the final system obtains 90.07 average accuracy and 87.81 macro average accuracy. The report focuses on the final test system and the implementation settings required to reproduce the adaptive dense verifier. 1 Introduction Video question answering benchmarks increasingly test whether multimodal models can reason over time rather than only recognize visible objects or actions 1 . VRR-QA focuses on visual relational reasoning beyond explicit cues: a question may require inferring which object is closer, where an entity is located relative to another entity, whether an object is visible from a viewpoint, or how a relation changes across the clip 2 . These questions are difficult because the decisive evidence is often distributed across frames and may depend on the correct reference frame. Our system uses no supervised training or fine-tuning. It is an inference-time pipeline that treats video relation reasoning as an evidence allocation problem. A single direct video-language model pass is efficient and often strong, but it compresses all visual evidence into one answer. This is risky for relation questions where the model must distinguish screen coordinates from object-centric coordinates, separate object motion from camera motion, or resolve an occlusion from only a few frames. We therefore allocate more computation only when the question appears unstable. Multiple lightweight views produce candidate answers and disagreement signals. Stable questions keep the direct answer. Difficult questions are sent to a dense evidence refinement module that makes the visual reasoning explicit through timestamped observations, relation probes, and pairwise verification between the current answer and a candidate alternative. The main contribution is a conservative refinement framework for video relational reasoning. The method does not treat every model disagreement as a correction. Instead, it uses disagreement to identify hard cases, then requires visible temporal evidence before changing the answer. This keeps the method reproducible and explains why additional model calls are spent on a small set of high-risk questions rather than uniformly across the full test set. 2 Method 2.1 Overview Figure 1 shows the full pipeline. The method is an adaptive test-time reasoning system rather than a single prompt. It separates answer production into four nested layers: a direct answer layer, a candidate bank layer, an evidence construction layer, and a risk-controlled decision layer. The direct layer is applied to all questions. The later layers are activated only when the answer is unstable or when the question asks for a relation that is likely to require explicit temporal evidence. Figure 1 : Adaptive Dense Evidence Refinement pipeline. A direct answer is first produced for all questions. Multi-view evidence builds a candidate bank and identifies unstable relation questions. A high-budget branch then schedules dense frame evidence, relation probes, pairwise verification, and risk-gated temporal aggregation before any answer is changed. 2.2 Direct Video Inference The base predictor receives the full video, question, and answer options, and returns a single answer choice. This direct path is the default inference path for all questions. It is simple, reproducible, and avoids unnecessary answer changes when the visual evidence is already clear. Direct inference is not discarded by later stages. It defines the current answer that every candidate must challenge. This replacement framing is important: a candidate answer is accepted only if it is better supported by visual evidence than the current answer, not merely because it is plausible. 2.3 Candidate Bank Construction Before verification, the system builds a compact candidate bank. Each question is evaluated under several lightweight views of the same video-question pair. The views differ in frame sampling, storyboard construction, crop or zoom emphasis, temporal alignment, answer-option ordering, and relation-focused prompting. Each view returns an answer and a short evidence statement. The bank stores the current answer, non-current answer candidates, supporting views, relation keywords, and evidence snippets. It is intentionally recall-oriented: a candidate may enter the bank even if it is not yet safe to apply. Candidate generation and answer replacement are treated as separate problems. 2.4 Disagreement and Hard-Case Routing The router decides whether a question should remain on the direct path or enter the expensive evidence path. It combines answer instability, candidate support, relation type, and answer-option structure. Questions receive higher priority when independent views disagree, when a non-current answer appears repeatedly, or when the question type is likely to require frame-level evidence: • lateral, vertical, and front/back spatial relations; • relative depth and proximity; • motion direction and trajectory changes; • viewpoint, visibility, facing direction, and occlusion. The routing signal is not used as a final answer predictor. It only decides the amount and type of computation assigned to the question. Stable questions keep the direct answer, mild disagreements receive pairwise verification, and hard relation questions receive dense evidence construction. 2.5 Adaptive Evidence Scheduler The high-budget branch uses a fixed sequence of modules, but the number of visual views depends on the routed difficulty. The scheduler starts with a global evidence scout, then expands into dense frame chunks only when the scout finds relevant entities and candidate timestamps. For relation questions, it also requests targeted crops, local zooms, and temporal windows around the suspected decisive moments. Table 1 : Adaptive evidence schedule for difficult questions. The high-budget path is deliberately more expensive than direct inference, but it is activated only for selected unstable cases. Stage Evidence produced Purpose Scout entities, rough timestamps localize relevant moments Storyboard views global frame sequence preserve temporal context Crop/zoom views local entity evidence resolve small objects and occlusion Temporal windows before/after relations check motion and state changes Probe matrix relation-specific facts test spatial, depth, motion, visibility Pairwise verifier current-vs-candidate decision test replacement safety Risk gate ambiguity and contradiction flags block unsafe corrections 2.6 Dense Frame Evidence Refinement For a selected difficult question, the video is converted into a sequence of timestamped evidence chunks. Instead of asking the model to immediately choose an answer again, the refinement stage asks for structured observations: which entities are visible, which reference object matters, what relation is visible, which timestamps support each hypothesis, and which ambiguity flags are present. This produces an explicit evidence table over time. The final decision can therefore be traced to visual observations rather than to a single global model judgment. 2.7 Relation Probe Matrix The verifier uses a probe matrix specialized to the predicted relation type. Spatial probes check left/right, front/behind, above/below, and reference-frame consistency. Depth probes check foreground/background, occlusion, and proximity. Motion probes compare