TimeLogic Challenge @ CVPR 2026: Strong MLLMs Meet Evidence-Seeking Agents for Temporal-Logic Video Question Answering

Zhaoyang Xu, Xusheng He, Wei Liu, Zhenyang Li, Jianlong Wu · 2026

Read TimeLogic Challenge @ CVPR 2026: Strong MLLMs Meet Evidence-Seeking Agents for Temporal-Logic Video Question Answering by Zhaoyang Xu, Xusheng He, Wei Liu, Zhenyang Li, Jianlong Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01631.txt.

Introduction

Xu and co-authors build an evidence-seeking agent for temporal video questions. Such questions ask when actions happen relative to each other. Before, after, until, and since are typical relations in them. Standard vision-language models answer in one pass over fixed frames. Evidence often sits at narrow action boundaries, though. It can also spread across several distant events. Their agent treats the task as active exploration. A Think-Act-Observe loop drives its search.

Each observation carries its absolute timestamp on a shared time axis. Temporal relations then reduce to numerical comparisons. A light classifier routes each question to a temporal category. Tailored policies, depths, and prompts follow that routing. The training-free system scores 77 point 13 average accuracy.

Catalog record

CREATORZhaoyang Xu, Xusheng He, Wei Liu, Zhenyang Li, Jianlong Wu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01631

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TimeLogic Challenge @ CVPR 2026: Strong MLLMs Meet Evidence-Seeking Agents for Temporal-Logic Video Question Answering Zhaoyang Xu, Xusheng He, Wei Liu, Zhenyang Li, Jianlong Wu Abstract Temporal-logic video question answering requires a model to reason about when actions occur relative to one another—before, after, until, since, overlap, and multi-event chains—rather than merely what is present in a video. Standard vision–language models (VLMs) typically answer such questions in a single pass over a fixed, uniformly sampled set of frames, which is poorly matched to evidence that is often localized to narrow action boundaries or dispersed across several distant events. We present an evidence-seeking agent that treats temporal-logic VideoQA as active exploration. The agent follows a Think–Act–Observe loop driven by a multi-granular sampling toolkit (global overview, structured temporal scan, segment skim, and dense focus), where every observation is interleaved with its absolute timestamp so that temporal relations reduce to numerical comparisons on a shared time axis. Its behavior is shaped by benchmark structure: a lightweight classifier routes each question to a temporal category, each with a tailored policy, iteration depth, and prompt, while sampling budgets adapt to corpus characteristics and clip length. The resulting training-free system couples Gemini 3.1 Pro with a temporal-reasoning policy and achieves 77.13 AvgAcc on the official TimeLogic test set. 1 Introduction Temporal-logic video question answering (VideoQA) asks models to reason about when events occur—before, after, overlap, persistence, and multi-event chains—rather than only what appears in a video. TimeLogic 5 makes this capability explicit through natural-language questions grounded in formal temporal operators. However, standard VLM inference is poorly matched to this setting. Most systems uniformly sample a fixed frame set and answer in one pass, but the decisive evidence for a temporal relation is often localized to a short window, such as the boundary between adjacent actions. Once that evidence is missed, the model has no mechanism to recover it or verify a tentative event ordering. This failure becomes more pronounced for multi-event chains, where the answer depends on localizing several actions on a shared timeline rather than recognizing a single visual state. The central challenge is thus to allocate a limited visual budget to the moments most likely to decide the temporal relation, especially when relevant events are sparse or visually subtle. We therefore frame TimeLogic inference as an active evidence-seeking problem, in line with recent tool-guided video-agent systems 4 . Our system uses a ReAct-style 7 Think–Act–Observe loop with multi-granular, timestamp-interleaved sampling tools. Instead of committing after one global sample, the agent repeatedly decides where to inspect next, accumulating timestamped observations and reasoning over the full history. The available tools cover coarse overview sampling, structured temporal scans, segment-level skimming, and dense focus around a candidate boundary. Because each returned frame is paired with an absolute timestamp, temporal relations can be checked as comparisons on a common time axis. The policy is guided by the structure of the benchmark. The released split contains heterogeneous source corpora and many compound three-event categories. We therefore classify each question into a temporal category and adapt the initial tool, iteration depth, frame budget, and reasoning prompt accordingly. Long, action-dense clips receive deeper exploration, while shorter clips can often be handled with denser one-pass coverage. These choices keep the system simple while encouraging targeted verification before answering. Our contributions are: • We formulate TimeLogic inference as Think–Act–Observe evidence seeking with multi-granular, timestamp-interleaved sampling. • We design a training-free, question-type-aware policy that adapts tools, prompts, and budgets to benchmark structure. • With Gemini 3.1 Pro 1 , the final system achieves 77.13 AvgAcc on the official test set. 2 Benchmark Analysis We analyze the released TimeLogic benchmark structure 5 , focusing on source-corpus characteristics and question templates, to identify where temporal-logic VideoQA is hardest and to motivate the design choices in Section 3 . The split contains 3{,}000 questions over 1{,}850 unique videos, drawn from four source corpora and posed in two answer formats: multiple-choice (MC, 1{,}878 questions) and Boolean ( bool , 1{,}122 questions). Our analysis surfaces two structural properties— heterogeneous corpora and a long tail of compound temporal categories —each of which directly shapes a component of the method. 2.1 Corpus Composition The questions are sourced from four datasets with markedly different visual and audio characteristics (Table 1 ). Each video’s source is recoverable from its identifier prefix. The corpora span a wide range of clip length and action density. Among them, only CrossTask provides effective instructional narration; STAR, AGQA, and Breakfast are treated as having no reliable audio signal. Corpus Type #Vid. #Q Audio / length CrossTask 8 instructional 472 729 effective / long STAR 6 situated activity 466 777 none / short AGQA 2 activity QA 473 760 none / short Breakfast 3 kitchen webcam 439 734 none / multi-min Table 1 : Test-split composition by source corpus. Only CrossTask provides effective audio; the remaining corpora are treated as visual-only, with different clip lengths and action densities. This heterogeneity is the primary motivation for the corpus-adaptive budgeting in Section 3 : a single global frame/step budget is mismatched to inputs that range from narrated CrossTask clips to silent, action-dense recordings. The kitchen-webcam corpus in particular pairs high action density with no effective audio, so it benefits from a larger frame budget, finer temporal segmentation, and visual-only reasoning. 2.2 Temporal-Category Distribution We infer each question’s temporal-logic category from its surface template, mapping 2{,}821 of the 3{,}000 questions onto the benchmark’s operator set; the remaining 179 are spatial/object-relation MC questions that do not correspond to a temporal operator. Table 2 shows the resulting distribution. Category Total bool MC Until 185 71 114 Since 112 58 54 Disjoint 204 108 96 Implies 142 47 95 Before 161 55 106 Next / After 155 77 78 Co-Occur 161 88 73 Immediate Next 157 75 82 Always Before 132 69 63 Always Next 136 71 65 Always Co-Occur 34 10 24 Strict A,B,C 166 41 125 Loose A,B,C 549 225 324 A always before B,C 527 127 400 Other / non-temporal 179 0 179 Table 2 : Inferred temporal-category distribution of the test split. Compound three-event categories (Strict/Loose A,B,C and A-always-before-B,C) dominate the mass, especially in MC format. The distribution is heavily long-tailed toward compound, three-event categories : Loose A,B,C and A always before B,C alone account for over a third of the split, and together with Strict A,B,C they are strongly concentrated in MC format ( 849 MC questions). These categories require reasoning over a full event chain A\!\to\!B\!\to\!C rather than a single pairwise relation, and are the hardest to answer without an explicit timeline. This observation motivates two method components: the timeline-first protocol for chain categories and the larger iteration/thinking budgets allocated to them. 2.3 Summary of Design Implications The analysis yields a direct mapping from dataset structure to method: (i) corpus heterogeneity \Rightarrow corpus-adaptive sampling budgets and audio gating; and (ii) the long tail of compound chain categories \Rightarrow timeline-first reasoning with deeper iteration. Section 3 develops each of these components. 3 Method We cast temporal-logic VideoQA as an evidence-seeking problem rather than a fixed-budget classification problem, similar in spirit to long-horizon video agents that seek evidence through tools 4 . Instead of feeding a vision–language model (VLM) a single uniformly

Read the complete text