VideoSEAL: Mitigating Evidence Misalignment in Agentic Long Video Understanding by Decoupling Answer Authority

Chenhao Qiu, Yechao Zhang, Xin Luo, Shien Song, Xusheng Liu · 2026

Read VideoSEAL: Mitigating Evidence Misalignment in Agentic Long Video Understanding by Decoupling Answer Authority by Chenhao Qiu, Yechao Zhang, Xin Luo, Shien Song, Xusheng Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12571.txt.

Introduction

Qiu and four colleagues spot a subtle failure in long-video question answering. Agents can answer right without support from the evidence they found. They define two checks, temporal groundedness and semantic groundedness, and trace misalignment to two pushes. Prompt push comes from shared-context saturation at inference, while reward push comes from outcome-only optimisation in training. The root cause, they argue, is coupling long-horizon planning with answer authority. Their decoupled planner-inspector setup separates the two roles.

Catalog record

CREATORChenhao Qiu, Yechao Zhang, Xin Luo, Shien Song, Xusheng Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84819
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12571

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VideoSEAL: Mitigating Evidence Misalignment in Agentic Long Video Understanding by Decoupling Answer Authority Chenhao Qiu, Yechao Zhang, Xin Luo, Shien Song, Xusheng Liu Abstract Long video question answering requires locating sparse, time-scattered visual evidence within highly redundant content. Although current MLLMs perform well on short videos, long videos introduce long-horizon search and verification, which often necessitates multi-turn, agentic interaction. We show that existing LVU agents can exhibit evidence misalignment : they produce correct answers that are not supported by the retrieved or inspected evidence. To characterize this failure, we introduce two diagnostics ( temporal groundedness and semantic groundedness ) and use them to reveal two pressures that amplify misalignment: prompt pressure from shared-context saturation at inference time and reward pressure from outcome-only optimization during training. These findings point to a structural root cause: the coupled agent paradigm conflates long-horizon planning with answer authority. We therefore propose the decoupled planner–inspector framework, which separates planning from answer authority and gates final answering on pixel-level verification. Across four long-video benchmarks, our framework improves both answer accuracy and evidence alignment, achieving 55.1% on LVBench and 62.0% on LongVideoBench while producing interpretable search trajectories. Moreover, the decoupled architecture scales consistently with increased search budgets and supports plug-and-play upgrades of the MLLM backbone without retraining the planner. Code and models are available at https://github.com/Echochef/VideoSEAL . 1 Introduction Long-video understanding (LVU) is substantially more challenging than short-video reasoning because relevant evidence is sparse and temporally dispersed, while the video contains overwhelming irrelevant content ( 19 ; 7 ; 15 ) . As a result, LVU systems are typically agentic : they iteratively navigate the timeline, retrieve candidate spans, and inspect visual evidence over multiple rounds of tool use ( 38 ; 23 ; 36 ; 47 ) . The prevailing paradigm in existing LVU systems is a unified agentic orchestration, where a monolithic planner controls retrieval, inspection tools within a single shared context window, reasons primarily over accumulated textual traces (e.g., captions and frame descriptions), and delivers the final answer ( 46 ; 39 ; 48 ; 27 ) . However, we observe a recurring failure mode in this paradigm: agents exhibit premature commitment, outputting an answer even when their interaction trace cannot provide sufficient grounded support ( 41 ; 29 ) . We formalize this phenomenon as evidence misalignment by evaluating outcome correctness alongside trace groundedness, focusing on the pathological correct but ungrounded regime. To characterize this failure, we introduce two complementary diagnostics: temporal groundedness , which checks whether the agent accesses relevant temporal regions during interaction, and semantic groundedness , which uses an LLM judge to verify whether the final answer is logically supported by the tool outputs in the trace. Together, these diagnostics test whether an agent truly earns its answer from accessed evidence and help identify the root cause of misalignment. Our diagnostics first reveal a systematic driver of evidence misalignment at inference time. During inference, as the agent’s tool traces grow longer and noisier, the planner is increasingly prompted to be decisive without reliably locating and reconciling the relevant evidence ( 20 ; 3 ) , drifting from verification toward plausible completion ( 24 ) . We term this pathology prompt pressure . Prompt pressure shifts behavior from evidence seeking to evidence fitting : when support is missing, ambiguous, or hard to locate in a long trace, the planner resolves uncertainty by falling back to parametric priors and generic plausibility templates, producing answers that are weakly supported by the trace. On the other hand, during training, while reinforcement learning (RL) offers a promising route to learning effective multi-turn interaction policies ( 42 ; 43 ) , agentic LVU is often equipped with outcome-only rewards because fine-grained temporal grounding labels are scarce ( 13 ; 10 ; 40 ) . This constitutes another systematic driver of evidence misalignment, since the reward primarily reflects answer correctness rather than evidence grounding, encouraging shortcut behaviors that improve outcomes without improving alignment. We refer to this incentive-driven pathology as reward pressure . Under reward pressure, agents can maximize return by relying on speculative completion instead of verifying evidence, leading to a widening hallucination gap ( 33 ; 8 ) . We argue these limitations share a common structural root: the coupled agent paradigm, where evidence seeking, inspection, and answer generation are conflated within a single model operating over the same context. To address this, we propose a decoupled planner–inspector framework, which strategically separates these distinct roles. This design gives the LLM planner a sparse, structured state for long-horizon navigation. An MLLM inspector then verifies candidate evidence and holds exclusive answer authority through an inspector gate. By architecturally decoupling these responsibilities and leveraging GRPO ( 31 ) , we mitigate prompt and reward pressures, effectively preventing unsupported answers driven by unverified tool outputs. Our decoupled framework achieves 55.1% answer accuracy and 62.0% accuracy on LongVideoBench ( 40 ) under a fixed budget, while exhibiting consistent gains with larger search budgets and stronger inspector backbones. Our contributions are threefold: 1. Evidence Misalignment Diagnostics: We formalize evidence misalignment in agentic VideoQA and introduce two diagnostics, temporal groundedness and semantic groundedness , which reveal two structural drivers: reward pressure and prompt pressure. 2. Decoupled Planner–Inspector Framework: We propose the decoupled planner–inspector framework that separates long-horizon planning from answer authority, with an inspector gate that produces an answer only when the inspected visual evidence is sufficient. 3. Empirical Gains and Scaling: Across four long-video benchmarks, our framework improves both answer accuracy and evidence alignment, and scales monotonically with larger search budgets and stronger inspector models, without retraining the planner. 2 Related Work MLLMs for LVU. Recent Multimodal LLMs ( 1 ; 21 ; 37 ) have demonstrated remarkable proficiency in short-horizon video understanding by scaling model capacity and instruction tuning. However, extending these architectures to long video encounters a structural bottleneck: visual token counts grow with video duration, whereas decisive evidence remains sparse. Standard approaches resort to aggressive downsampling or token compression, inevitably sacrificing critical evidence ( 27 ; 34 ; 53 ) . This limitation ( 27 ; 38 ; 28 ) has necessitated a shift from single-pass inference to iterative, agentic reasoning paradigms that actively navigate the video timeline. Agentic Evidence Seeking and Reasoning. To overcome the computational bottleneck of single-pass inference, recent work adopts iterative, evidence seeking agents that retrieve candidate temporal spans and reason over intermediate textual summaries across multiple turns ( 38 ; 17 ; 30 ; 51 ) . However, these agents often rely on a strong monolithic planner to manage long-horizon search, which incurs substantial inference cost. More fundamentally, they typically reason over lossy intermediate textual summaries rather than visual evidence, so inaccurate summaries can propagate into evidence misalignment when key spans are not retrieved. Distinct from inference-time methods, complementary approaches aim to learn planning and tool-use behaviors directly via RL ( 11 ) . Frameworks such as Video-MTR and LongVT ( 42 ; 43 ) train a single model to iteratively acquire evidence with intrinsic visual grounding ( 45 ; 32 ) . However, these methods typically place search, accumulated tool outputs, and final answering in the same shared context. This design can encourage speculative completion, where the model compulsively reasons answers from superficial

Read the complete text