ViTexQA: A Multi-Frame Temporal Perception Dataset for Video Text Question Answering
Zhentao Guo ∗ *, Chen Duan ∗ *, Tongkun Guan ∗ *, Zining Wang, Kai Zhou, Pengfei Yan · 2026
Read ViTexQA: A Multi-Frame Temporal Perception Dataset for Video Text Question Answering by Zhentao Guo ∗ *, Chen Duan ∗ *, Tongkun Guan ∗ *, Zining Wang, Kai Zhou, Pengfei Yan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24602.txt.
Introduction
Guo and colleagues build a video question answering dataset where reading one frame is never enough. Multimodal models still stumble over text in videos: signs, captions, labels. Meaning accumulates from textual cues scattered across frames. Existing datasets fail the point. Most of their questions can be answered from a single frame, so they test static reading wearing a video costume.
Every question in ViTexQA demands cross frame text fusion. It is enforced by a quality controlled chain of thought annotation pipeline, boosted with temporal constraints. The companion model, FrameThinker, trains in two stages. Supervised fine tuning on frame aware reasoning chains comes first. Reinforcement learning grounded temporally, with multi frame coherence rewards, follows. Evaluations show the method outperforming state of the art baselines on the new dataset. The reported text overlap score lifts by a clear margin. Guo's 2026 contribution is both yardstick and demonstration.
Catalog record
| CREATOR | Zhentao Guo ∗ *, Chen Duan ∗ *, Tongkun Guan ∗ *, Zining Wang, Kai Zhou, Pengfei Yan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24602
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ViTexQA: A Multi-Frame Temporal Perception Dataset for Video Text Question Answering Zhentao Guo ∗ *, Chen Duan ∗ *, Tongkun Guan ∗ *, Zining Wang, Kai Zhou, Pengfei Yan Abstract Despite remarkable progress in multimodal understanding, current MLLMs still exhibit limitations in video text understanding, particularly when semantics emerge through the integration of temporally distributed textual cues across multiple frames. This perception challenge fundamentally differs from static image text understanding, yet existing datasets fail to capture: the vast majority of questions remain answerable from single frames, inadequately reflecting real-world video text comprehension demands. To address this, we present ViTexQA, a large-scale video-text QA dataset, and FrameThinker for robust multi-frame temporal reasoning. We build ViTexQA via a quality-controlled Chain-of-Thought (CoT) annotation pipeline boosted with temporal constraints; all its QA pairs demand cross-frame text fusion to solve, enforcing true temporal reliance. FrameThinker adopts two-stage training for explicit temporal modeling: CoT-Guided Supervised Fine-Tuning (SFT) generates frame-aware reasoning chains, followed by Temporally-grounded Reinforcement Learning (RL) optimized with multi-frame coherence rewards. Evaluations show our method outperforms SOTA baselines on ViTexQA, lifting ROUGE-L by 6.3%. 0 0 footnotetext: ∗ Equal contribution. ✉ Corresponding author. 1 Introduction Figure 1 : Single-frame and multi-frame perception examples. (a) Single-frame perception, where the answer is derived from textual content within a single frame (such as reading “Waterloo 26” from the red bus.). All compared models answer correctly on such single-frame perception questions. (b) Multi-frame perception, which requires the model to integrate textual, spatial, and temporal cues across multiple frames (such as finding the answer to the question based on the progress of the competition, ranking changes, and time spent.). In contrast, while other models frequently produce incorrect or incomplete answers, our FrameThinker consistently arrives at the correct answer. The rapid development of Multimodal Large Language Models (MLLMs) [ 2 , 5 , 25 , 56 , 58 , 16 , 50 , 18 ] has enabled unified approaches to diverse vision-language tasks, from static image understanding [ 15 , 12 , 14 ] to dynamic video comprehension [ 52 , 49 , 27 , 24 , 43 , 31 ] . Among these capabilities, the ability to understand text within visual content has proven essential for real-world applications. While substantial progress has been made in image-based scene text understanding through benchmarks like DocVQA [ 34 ] , TextVQA [ 45 ] , and ST-VQA [ 3 ] , the challenge of comprehending text in videos remains fundamentally different. Videos introduce a critical temporal dimension that distinguishes them from static images. Textual information in videos often carries semantics that emerge only through temporal integration rather than appearing complete in isolated moments. Consider sports broadcasts where understanding the game progression requires tracking score changes across frames, news programs where complete stories unfold through sequential caption updates, or instructional videos where procedural knowledge accumulates through temporally distributed textual cues [ 22 , 8 ] . In these scenarios, comprehension requires synthesizing information dispersed across multiple timestamps to construct complete semantic interpretations. As shown in Fig. 1 , while single-frame questions can be answered by examining one snapshot, multi-frame perception demands integrating evidence distributed throughout the video timeline to derive meaningful answers. Despite the importance of temporal perception over textual content, existing video text QA datasets exhibit a critical limitation. Our statistical analysis reveals that only approximately 13% of questions in M4-ViteVQA [ 61 ] and 18% in EgoTextVQA [ 62 ] genuinely require multi-frame perception. The overwhelming majority can be answered from single frames, failing to capture the temporal complexity inherent in real-world video understanding. This gap reflects a broader issue in current benchmark design: datasets inadvertently allow models to bypass temporal perception by relying on static, frame-level perception. Consequently, MLLMs trained and evaluated on these benchmarks develop limited capabilities for cross-frame integration, hindering their effectiveness in scenarios demanding true temporal comprehension. To address this fundamental gap, we introduce ViTexQA, a large-scale dataset explicitly designed to require multi-frame temporal perception over textual content in videos. Unlike existing benchmarks where temporal perception remains optional, ViTexQA ensures that every question is deliberately unanswerable from any single frame alone. Each question necessitates integrating textual cues distributed across multiple timestamps, forcing models to engage in genuine temporal perception rather than frame-level pattern matching. This design philosophy reflects authentic video understanding demands where critical information emerges only through temporal synthesis. Specifically, ViTexQA is constructed through two complementary pipelines that together address both evaluation and training needs for multi-frame temporal perception. The Video Dataset Construction pipeline ensures genuine multi-frame temporal dependencies through rigorous data collection and quality controlled human annotation, producing 5,147 high-quality videos with 6,864 QA pairs where each question is verified to require cross-frame integration. The Multi-frame perception CoT Construction pipeline further enriches these QA pairs with temporally-grounded CoT annotations that explicitly preserve spatiotemporal textual information across video frames. Together, these pipelines yield a comprehensive dataset spanning approximately 363 hours across diverse domains including sports, news, tutorials, and everyday scenarios. Critically, we adopt a human-centric approach throughout: all annotations are created exclusively by human annotators without automated assistance from GPT or similar systems, ensuring alignment with natural human perception patterns and genuine challenge for current MLLMs [ 27 , 46 , 31 , 43 , 24 , 13 ] . Building upon ViTexQA, we further propose FrameThinker, a systematic method to enhance multi-frame temporal perception capabilities in MLLMs. FrameThinker addresses the temporal perception challenge through a two-stage training paradigm. First, CoT-Guided SFT trains models to generate temporally-grounded perception chains that explicitly enumerate textual evidence across frames with spatial and temporal grounding. Second, Temporally-grounded RL refines these capabilities through composite reward mechanisms that encourage temporal coherence, multi-frame dependency, and accurate evidence aggregation. This approach enables models to develop explicit temporal perception strategies rather than relying on implicit frame-level correlations. Finally, we conduct extensive experiments evaluating state-of-the-art MLLMs on ViTexQA. Results reveal substantial performance gaps compared to human accuracy, highlighting fundamental limitations in current models’ temporal perception abilities. Furthermore, FrameThinker achieves significant improvements over strong baselines, demonstrating the effectiveness of explicit temporal perception training. These findings underscore both the challenge posed by genuine multi-frame temporal perception and the potential for systematic capability enhancement through appropriate training methodologies. Our contributions are summarized as follows: • We present ViTexQA, the first large-scale dataset tailored for multi-frame temporal awareness, constructed through dual complementary pipelines producing both rich question-answer pairs and corresponding temporally-grounded CoT annotations via rigorous human-centric annotation. • We propose FrameThinker, a training methodology that systematically enhances multi-frame temporal perception through CoT-guided SFT and Temp- orally-grounded RL. • Comprehensive benchmarking on ViTexQA reveals critical gaps in current MLLMs’ temporal perception capabilities. In contrast, extensive experiments demonstrate that FrameThinker effectively mitigates these limitations, significantly outperforming strong baselines and validating the effectiveness of explicit temporal perception training for video text understanding. 2 Related Work Image-based text VQA datasets OCR-related datasets are among the most important benchmarks for evaluating MLLMs in visual text understanding. In the OCR domain, a variety of datasets have been proposed to evaluate different aspects of visual text understanding. Natural scene text VQA datasets, such as TextVQA [ 45 ] , ST-VQA [ 3 ] , EST-VQA [ 48 ] , and OCR-VQA [ 35 ] , focus on question