CRAFT: C ritic- R efined A daptive Key- F rame T argeting for Multimodal Video Question Answering

Mahesh Bhosale, Abdul Wasi, Vishvesh Trivedi, Pengyu Yan, Akhil Gorugantu, David Doermann · 2026

Read CRAFT: C ritic- R efined A daptive Key- F rame T argeting for Multimodal Video Question Answering by Mahesh Bhosale, Abdul Wasi, Vishvesh Trivedi, Pengyu Yan, Akhil Gorugantu, David Doermann (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19075.txt.

Introduction

Mahesh Bhosale and five colleagues present CRAFT, a system for answering questions across many news videos while grounding every claim in its sources. It selects keyframes per query. It transcribes speech with multilingual fallback. A critic loop checks and repairs claims before merging them, so each fact appears once with all its supporting sources. On the MAGMaR 2026 news benchmark it leads on average, reference recall, and citation quality. It also performs strongly on a converted WikiVideo set of 52 event queries, suggesting the approach generalises.

Catalog record

CREATORMahesh Bhosale, Abdul Wasi, Vishvesh Trivedi, Pengyu Yan, Akhil Gorugantu, David Doermann
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43516
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19075

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CRAFT: C ritic- R efined A daptive Key- F rame T argeting for Multimodal Video Question Answering Mahesh Bhosale, Abdul Wasi, Vishvesh Trivedi, Pengyu Yan, Akhil Gorugantu, David Doermann Abstract Grounded multi-video question answering over real-world news events requires systems to surface query-relevant evidence across heterogeneous video archives while attributing every claim to its supporting source. We introduce CRAFT (Critic-Refined Adaptive Key-Frame Targeting), a query-conditioned pipeline that combines dynamic keyframe selection, per-video ASR with multilingual fallback, and a hybrid critic loop to iteratively verify and repair claims before consolidation. The pipeline integrates UNLI temporal entailment, DeBERTa-v3 cross-claim screening, and a Llama-3.2-3B adjudicator, with a final citation-merging stage that emits each fact once with all supporting source identifiers. On MAGMaR 2026, CRAFT achieves the best overall average (0.739), reference recall (0.810), and citation F1 (0.635). We further evaluate on a MAGMaR-style conversion of WikiVideo with 52 non-overlapping event queries, where CRAFT also performs strongly (0.823 Avg), showing that its claim-centric evidence aggregation generalizes beyond MAGMaR. Ablations show that atomic claims, ASR, and the critic loop drive the main gains over the vanilla query-conditioned baseline. Code and implementation details are publicly available at https://github.com/bhosalems/CRAFT . 1 Introduction Multi-video question answering over real-world news events underlies tasks from event understanding to fact-checking and crisis reporting. Recent benchmarks such as MultiVENT 2.0 ( 14 ) and the WikiVideo article-generation task ( 21 ) formalize a strict variant of this problem: given a query and a collection of relevant videos, a system must produce a report whose every statement is grounded in identifiable visual, textual, or spoken evidence from the source videos. The MAGMaR 2026 oracle task adds two further constraints. Each query is paired with a persona and a background paragraph, and the resulting report is scored on six axes that separately measure content precision and recall ( Ref-P , Ref-R ) against a reference answer and citation precision and recall ( Cite-P , Cite-R ) against gold source videos. A high-scoring system must both surface the right facts and attach the right videos to them. Three properties of long news video make this hard. First, vision-language models face a hard token-budget bottleneck on hour-scale input: even at 1 FPS, a long video exceeds practical context windows ( 36 ; 9 ) , and uniform sampling silently truncates whatever falls outside the budget. Second, even when relevant frames are presented, recent hallucination benchmarks ( 40 ; 17 ; 48 ) show that VLMs routinely emit claims unsupported by the visual content, with errors concentrated at long-tail entities, numerical details, and event timing-precisely the content most likely to be cited in a news report. Third, much of the answer-relevant content in news video is spoken rather than shown: visual-only extraction misses interview answers, on-the-ground reporting, and official statements, especially in non-English coverage. Prior work addresses these challenges in isolation. Adaptive keyframe selectors ( 36 ; 9 ) trim the visual input to query-relevant frames but treat the result as terminal evidence, with no check that downstream claims are actually supported. Critic-driven video QA systems ( 20 ; 6 ) add verification, but typically at the final answer-aggregation stage and at the granularity of a single role rather than per claim. Modular video-RAG pipelines ( 12 ; 28 ; 45 ) compose retrieval and reasoning over long context but rely on a single visual stream and ignore speech leading to citation faithfulness diverging from citation correctness 38 . We present CRAFT (Critic-Refined Adaptive Key-Frame Targeting), a query-conditioned pipeline that integrates these threads for the MAGMaR 2026 oracle task (Figure 1 ). Our contributions are: (i) a multimodal evidence stream (§ 3.1 ) combining 120-second video chunking, per-video ASR (Qwen3-ASR-1.7B with a Whisper-large-v3 fallback for low-resource languages), automatic English translation, and dynamic query-conditioned keyframe selection, so the VLM receives a clip and transcript both targeted at the current query; (ii) a critic-guided extraction loop (§ 3.3 ) that runs a UNLI video-claim entailment model for temporal grounding, a DeBERTa-v3 MNLI cross-encoder for cross-claim contradiction screening, and a Llama-3.2-3B adjudicator that confirms contradictions and emits repair feedback, returning the critic report to the VLM for up to four re-extraction rounds; and (iii) atomic claim formatting (§ 3.2 ) with citation-merging consolidation (§ 3.6 ), which emits each fact once with all supporting source identifiers attached, preserving citation recall while suppressing the redundancy that inflates reference-precision loss. On MAGMaR 2026 (§ 4 ), CRAFT outperforms strong baselines with the highest overall average (0.739), reference recall (0.810), and citation F1 (0.635) of all evaluated configurations. Ablations (§ 4.5 ) show that the gains from the critic loop, atomic claims, and ASR-augmented extraction are partly orthogonal to the choice of base VLM, transferring across Qwen3.5-9B ( 25 ) and Qwen3-VL-30B ( 2 ) backbones, and outperforming strong VLMs such as Molmo2-8B ( 7 ) and Gemma-4-31B 1 1 1 https://huggingface.co/google/gemma-4-31B-it . 2 Related Work Long-video understanding with vision-language models. Open-source video-language models have improved rapidly along two axes: backbone capacity and temporal modeling. The Qwen-VL family progressed from dynamic-resolution and time-aligned M-RoPE in Qwen2.5-VL ( 3 ) to interleaved M-RoPE, DeepStack cross-layer fusion, and explicit timestamp tokens in Qwen3-VL ( 2 ) , while InternVL3 ( 55 ) introduced Variable Visual Position Encoding and native multimodal pre-training. LLaVA-Video ( 53 ) and LLaVA-OneVision ( 16 ) consolidated the LLaVA recipe for video instruction tuning. Despite these gains, all such models face a hard token-budget bottleneck on hour-scale input: even at 1 FPS, a long video produces token counts that exceed practical context windows ( 36 ; 9 ) . Specialized long-context architectures, including LongVU ( 30 ) , Video-XL ( 32 ) , MovieChat ( 33 ) , and MA-LMM ( 10 ) , mitigate this through spatiotemporal compression, sparse memory, or hierarchical attention, but typically at the cost of fine-grained temporal evidence that is essential for citation-grounded answering. Adaptive keyframe selection. Because uniform sampling is the dominant performance bottleneck on long videos, a substantial body of recent work has focused on query-conditioned frame selection. AKS ( 36 ) formulates selection as a joint optimization over prompt-frame relevance and temporal coverage, solved by a recursive split-and-judge algorithm; APVR ( 9 ) extends this idea to a two-granularity hierarchy in which Pivot Frame Retrieval expands the query into semantic facets and Pivot Token Retrieval performs query-aware token selection within retained frames. VideoTree ( 41 ) replaces flat selection with a query-adaptive tree of clustered keyframes captioned coarse-to-fine. Other recent variants include MDP3 ( 35 ) , which casts selection as a Markov decision process; Q-Frame ( 49 ) , which ranks frames into multiple resolution tiers; AdaRD-Key ( 51 ) , which encourages diversity through determinantal point processes; F2C ( 34 ) , which extends keyframes to short clips to preserve motion continuity; and VidF4 ( 18 ) , which proposes differentiable frame scoring for end-to-end VideoQA. A.I.R. ( 56 ) and T* ( 43 ) replace lightweight CLIP-based scorers with iterative VLM-based reasoning over candidate frames, trading cost for accuracy. A common property of these selectors is that their output is treated as the terminal evidence representation, with no mechanism to detect whether claims subsequently extracted from the chosen frames are

Read the complete text