DynFrame: Adaptive Reasoning-Driven Multimodal Framework with Dynamic Frame Augmentation for Complex Video Understanding
Peng Zhang Guanghao Zhang Wanggui He Longxiang Zhang Mushui Liu Yan Xia, Zhenhao Peng Weilong Dai Jinlong Liu Haobing Tang Le Zhang Hao Jiang Pipei Huang, [0.5em] Zhejiang University Alibaba Group · 2026
Read DynFrame: Adaptive Reasoning-Driven Multimodal Framework with Dynamic Frame Augmentation for Complex Video Understanding by Peng Zhang Guanghao Zhang Wanggui He Longxiang Zhang Mushui Liu Yan Xia, Zhenhao Peng Weilong Dai Jinlong Liu Haobing Tang Le Zhang Hao Jiang Pipei Huang, [0.5em] Zhejiang University Alibaba Group (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26680.txt.
Introduction
Peng Zhang and twelve others present DynFrame, a video-understanding frame that learns both where to look and how densely to sample. Existing thinking-with-video systems fix frame rates per window and credit looking and answering with one shared score, forcing repeated retrievals. DynFrame emits window and density as tokens in one pass. It then splits credit at the retrieval boundary so sampling and answering earn separate rewards. Its 4B model rivals 7B to 8B rivals on six benches, and its 8B model sets new bests on most metrics.
Catalog record
| CREATOR | Peng Zhang Guanghao Zhang Wanggui He Longxiang Zhang Mushui Liu Yan Xia, Zhenhao Peng Weilong Dai Jinlong Liu Haobing Tang Le Zhang Hao Jiang Pipei Huang, [0.5em] Zhejiang University Alibaba Group |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48361 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26680
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DynFrame: Adaptive Reasoning-Driven Multimodal Framework with Dynamic Frame Augmentation for Complex Video Understanding Peng Zhang Guanghao Zhang Wanggui He Longxiang Zhang Mushui Liu Yan Xia, Zhenhao Peng Weilong Dai Jinlong Liu Haobing Tang Le Zhang Hao Jiang Pipei Huang, [0.5em] Zhejiang University Alibaba Group Abstract Recent video multimodal large language models (MLLMs) increasingly couple step-by-step reasoning with on-demand visual evidence retrieval, allowing models to revisit relevant video segments during inference. However, two structural gaps remain in existing thinking-with-video systems. (i) Sampling density is not a learnable decision: existing methods may let the model decide where to look, but the per-window frame rate is largely fixed. As a result, fine-grained evidence is often recovered through repeated retrieval calls which increases inference context length and training difficulty. (ii) Retrieval and answer generation are usually optimized with a single trajectory-level advantage, so the “where to look” tokens and the “how to answer” tokens receive the same credit even when one is correct and the other is not. To address these gaps, we present DynFrame , a framework that emits the temporal window and the sampling density as native tokens within a single autoregressive pass, such learnable span–density retrieval enables acquiring multi-granularity evidence with a single retrieval step. Based on the above tokenized retrieval interface, we further introduce Segment-Decoupled GRPO (SD-GRPO) , which splits each rollout at the retrieval boundary and assigns role-specific token-level advantages, separately crediting the sampling decision and the answer. Trained on the curated DM-CoT-74k and DM-RL-45k , DynFrame-4B is competitive with strong 7B–8B baselines across six benchmarks (NExT-GQA, Charades-STA, ActivityNet-MR, Video-MME, MLVU, LVBench), and DynFrame-8B sets new state-of-the-art on most metrics. Code is available at https://github.com/zhangguanghao523/DynFrame . † † * Equal contribution. § Project lead. † Corresponding author. 1 Introduction Multimodal large language models (MLLMs) [ 2 , 46 , 18 , 14 , 4 , 16 , 1 ] have substantially advanced video question answering, temporal grounding, and long-form video understanding [ 34 , 9 , 27 , 17 , 7 , 45 , 31 ] . Building on chain-of-thought (CoT) reasoning [ 33 , 25 , 32 ] , recent supervised and reinforcement-learning post-training methods further improve multi-step video inference [ 6 , 20 , 30 ] . However, most video reasoners still operate after a fixed visual pass: a sparse set of frames is selected before generation, and all subsequent intermediate steps are purely textual [ 23 , 40 , 15 , 43 , 1 ] . This design is fragile for long videos, where the answer may hinge on a short action, a brief object state change, or multiple clues scattered across a redundant temporal context. As reasoning chains grow, the model can drift away from the actual visual evidence and hallucinate missing fine-grained events. Figure 1 : Textual CoT vs. DynFrame. Textual CoT (left) reasons over a fixed sparse frame set and misses the airborne segment, yielding a wrong rotation count. DynFrame (right) emits <span> and <fps> tokens within its reasoning to retrieve a denser, temporally focused frame set, and then continues reasoning over the augmented visual context to reach the correct answer. A growing thinking-with-video line of work addresses this limitation by allowing models to actively revisit video evidence during inference. Existing systems instantiate visual revisiting through several retrieval interfaces: tool-based clip retrieval that calls an external module to crop or resample candidate video segments [ 41 , 36 ] , zoom-in or temporal-focusing mechanisms that inspect local regions at higher resolution [ 8 , 5 ] , multi-turn frame spotlighting or iterative perception that progressively refines clue-focused temporal regions [ 12 , 35 ] , and native interleaved tool invocation that couples evidence seeking with reasoning in a shared context [ 39 , 24 , 21 ] . These systems demonstrate the importance of revisiting visual evidence, but fine-grained evidence acquisition is often achieved by issuing repeated retrieval calls, refining temporal clues across turns, or appending additional high-resolution clips into an expanding context. Such multi-turn retrieval increases inference context length and also complicates training. This motivates a complementary question: can a video MLLM make each retrieval action more expressive, so that task-adaptive, multi-granularity evidence can be acquired with fewer retrieval steps? To address these challenges, we introduce DynFrame , a novel framework that emits the temporal window and the sampling density as native tokens within a single autoregressive pass, turning frame-rate adaptation from a system hyperparameter into a learnable per-step decision. This learnable span–density interface enables task-adaptive frame acquisition: the model can retrieve dense frames for short, motion-sensitive events and sparse frames for long-range semantic understanding, acquiring multi-granularity evidence with a single retrieval step. This reduces reliance on repeated multi-round retrieval calls, which often introduce long inference contexts and complex tool-call training designs. Furthermore, based on the explicit retrieval boundary created by this tokenized interface, we propose Segment-Decoupled GRPO (SD-GRPO) , which splits each rollout at the retrieval boundary and assigns role-specific token-level advantages, separately crediting the sampling decision and the answer reasoning. This provides targeted credit for temporal selection and sampling density while preserving the end-to-end answer signal. To train this behavior, we curate task-balanced DM-CoT-74k for supervised fine-tuning and DM-RL-45k for reinforcement learning, explicitly designed to cultivate robust native tokenized adaptive retrieval capabilities. Extensive experiments across six benchmarks spanning temporal grounding, grounded VideoQA, and long-form video understanding show that DynFrame-4B is competitive with strong 7B–8B baselines, while DynFrame-8B achieves state-of-the-art results on most metrics. In summary, our contributions are three-fold: • We introduce DynFrame , a multimodal reasoning framework that emits temporal span and sampling density as native tokens, turning adaptive temporal evidence acquisition from an externally scheduled tool operation into a model-native reasoning capability. • We curate DM-CoT-74k for cold-start SFT and DM-RL-45k for reinforcement learning, and propose SD-GRPO , which uses the explicit retrieval boundary to assign role-specific token-level advantages to the sampling and reasoning segments. • Across six benchmarks spanning temporal grounding, grounded VideoQA, and long-form video understanding, DynFrame-4B is competitive with strong 7B–8B baselines, while DynFrame-8B achieves state-of-the-art results on most metrics. 2 Related Work Multimodal Chain-of-Thought for video reasoning. Multimodal chain-of-thought extends textual CoT [ 33 ] by allowing intermediate reasoning to interact with visual evidence rather than relying on a fixed visual pass. Early video reasoning and post-training methods, such as Video-R1 [ 6 ] , VideoChat-R1 [ 20 ] , VideoRFT [ 30 ] , Temporal-RLT [ 19 ] , improve multi-step video inference through supervised fine-tuning or reinforcement learning, but they mostly reason over the visual tokens supplied at the beginning of generation. A growing thinking-with-video line instead lets the model actively revisit visual evidence during inference. VITAL [ 41 ] and LongVT [ 36 ] formulate evidence acquisition as tool-based clip retrieval; LOVE-R1 [ 8 ] and VideoZoomer [ 5 ] use zoom-in or temporal focusing; VideoChat-R1.5 [ 35 ] performs iterative perception; Video-o3 [ 39 ] , Open-o3 Video [ 24 ] , and VideoTemp-o3 [ 21 ] explore native interleaved tool invocation. These methods show that active evidence acquisition is important for long-video reasoning. However, their retrieval granularity is still largely governed by external tools, preset zoom/spotlight modes, or system-defined visual-token budgets. As a result, obtaining the right amount of