How Should Video LLMs Output Time? An Analysis of Efficient Temporal Grounding Paradigms
Shengji Jin, Yuanhao Zou, Victor Zhu, Zhengping Ji, Chen Chen · 2026
Read How Should Video LLMs Output Time? An Analysis of Efficient Temporal Grounding Paradigms by Shengji Jin, Yuanhao Zou, Victor Zhu, Zhengping Ji, Chen Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08966.txt.
Introduction
Jin and four colleagues study how video language models should output time when locating a moment inside a video, a task known as temporal grounding. Past methods tangle the output format with different backbones, datasets, and training recipes, so the format's own effect stays unclear. Edge deployment adds a cost question. Comparing paradigms on equal footing, they find the output choice shifts both accuracy and cost regardless of model size. A continuous distribution format gives the best balance of precise localization and low latency.
Catalog record
| CREATOR | Shengji Jin, Yuanhao Zou, Victor Zhu, Zhengping Ji, Chen Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53916 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.08966
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Should Video LLMs Output Time? An Analysis of Efficient Temporal Grounding Paradigms Shengji Jin, Yuanhao Zou, Victor Zhu, Zhengping Ji, Chen Chen Abstract While Multimodal Large Language Models (MLLMs) have advanced Video Temporal Grounding (VTG), existing methods often couple output paradigms with different backbones, datasets, and training protocols. This makes it challenging to isolate the specific impact of the output design. Additionally, as VTG systems are increasingly considered for resource-constrained edge deployment, the trade-off between output formulation and system-level efficiency requires systematic investigation. In this paper, we present a controlled empirical study comparing three dominant VTG output paradigms: Text Numeral Generation, Temporal Token Generation, and Continuous Temporal Decoding. We evaluate these paradigms across identical compact VLMs (SmolVLM2, FastVLM, and Molmo2) using consistent datasets and LoRA fine-tuning protocols. Evaluations on Charades-STA, QVHighlights, and YouCook2 measure both localization accuracy and system efficiency, including inference latency, training throughput, and parameter overhead. Our results demonstrate that the choice of output formulation significantly affects both grounding accuracy and computational cost, independent of model scale. Specifically, the continuous distribution paradigm consistently achieves the most favorable efficiency-accuracy trade-off on the Pareto frontier, delivering robust localization with minimal latency overhead. These findings provide objective empirical guidelines for designing efficient, deployment-ready VTG systems. 1 1 1 Our code and models are publicly available: https://tg-paradigms.github.io/ 1 Introduction Video temporal grounding (VTG), the task of localizing temporal segments in a video matching a natural language query, has attracted growing research interest. Core tasks including moment retrieval 4 ; 1 , dense video captioning 11 ; 34 , highlight detection 12 , and grounded video question answering 33 ; 32 all require models to produce precise temporal boundaries. As multimodal large language models (MLLMs) increasingly serve as the backbone for these tasks, diverse temporal output formulations have been proposed 30 . Based on the different temporal output formulation , existing approaches diverge into three main paradigms (Figure 1 ): (a) text numeral generation, expressing timestamps as text numerals within the language modality 8 ; 14 ; (b) temporal token generation, introducing dedicated temporal tokens into the vocabulary 22 ; 6 ; 5 ; and (c) continuous temporal decoding, encompassing both regression modules that predict scalar timestamps from hidden states 19 ; 28 and distribution-based methods that convert temporal embeddings into probability distributions over time bins 37 , both sharing the principle of mapping hidden representations to continuous temporal targets while differing in whether a distributional prior is imposed. Figure 1 : Overview of three output formulation paradigms for temporal grounding in Video LLMs. After extracting features through shared M-LLM Layers, the architecture diverges into three distinct designs: (a) Text Numeral Generation utilizes the standard M-LLM head to directly generate timestamps as sequences of regular text tokens. (b) Temporal Token Generation introduces specific temporal tokens and a dedicated temporal head, utilizing a restricted vocabulary library for controlled, sequential boundary generation. (c) Continuous Temporal Decoding maps hidden states through a lightweight MLP head to predict continuous temporal distributions, directly decoding boundary coordinates without auto-regressive generation. Each paradigm is exemplified by recent work: the text numeral approach generates temporal boundaries as plain-text numerals within a chat-style dialogue framework 8 ; the temporal token generation paradigm employs causal event modeling, decoding events auto-regressively through task-specific encoder-decoder pairs 6 ; and continuous temporal decoding approaches, such as distribution-based methods, introduce a learnable temporal token to model boundary ambiguity via a probabilistic decoder 37 . We omit standalone regression heads from our study as they can be viewed as a degenerate case of distribution-based decoding without the distributional prior. However, as summarized in Table 1 , each system uses its own backbone, data, and protocol, making it impossible to attribute performance differences to the output formulation alone. This raises a critical yet underexplored question: how should temporal predictions be formulated in the model’s output space? The survey 30 explicitly identifies this lack of controlled cross-paradigm comparison as an open challenge. Moreover, these representative systems are all built on 7B-class or larger backbones. As the field moves toward practical deployment on resource-constrained platforms such as edge devices, mobile applications, and real-time video analytics, it becomes essential to understand how these output paradigms perform when paired with compact models, where computational budget is limited and every architectural choice carries amplified impact. This paper addresses both gaps through a controlled empirical study. We deliberately choose compact backbones ranging from 0.5B to 8B parameters (SmolVLM2 0.5B/2.2B, FastVLM 1.5B, and Molmo2 4B/8B), covering a spectrum of resource-constrained deployment scenarios. We implement three representative temporal output paradigms under a unified framework, ensuring that backbone, raw data source, optimizer, epoch count, and fine-tuning protocol are identical across all conditions. Each paradigm naturally requires its own data formatting (e.g., structured event sequences for the generative paradigm, special token annotations for the distribution-based paradigm, and plain-text timestamp strings for the text numeral baseline), but the underlying video-query-annotation triples remain the same. The sole variable is thus the output formulation and its associated training objective. Our contributions are: • Unified fine-tuning comparison. We implement generative structured output (TRACE 6 -style), distribution-based temporal modeling (DisTime 37 -style), and text numeral baseline (VTimeLLM 8 -style) paradigms on the same set of compact backbones, using identical training configurations to isolate the output formulation as the sole experimental variable. • Efficiency-aware analysis. Beyond localization accuracy, we analyze parameter overhead, training stability, inference determinism, and computational cost, dimensions critical for practical deployment yet underexplored in the VTG-MLLM literature 30 . • Empirical insights at compact scale. We report how localization accuracy, training convergence, and inference determinism vary across paradigms when the backbone is scaled from 0.5B to 8B, and identify which output formulation offers the best accuracy-efficiency trade-off for deployment on resource-constrained platforms. Table 1 : Detailed comparison of existing VTG-MLLM systems. To highlight the extreme heterogeneity, we categorize the setups by LLM backbone, visual encoder, data scale, and optimization pipeline. 1-S , 2-S , and 3-S denote 1-stage, 2-stage, and 3-stage training protocols, respectively. Paradigm Method LLM Backbone Visual Encoder Data Size Stages Architectural Additions / Key Designs Text Numeral HawkEye 29 Vicuna v1.5 (7B) CLIP ViT-L/14 1.1M 3-S Recursive search space narrowing SlowFocus 20 Vicuna v1.5 (7B) CLIP ViT-L/14 1.0M 3-S Mixed-frequency sampling mechanism VTimeLLM 8 Vicuna v1.5 (7B) CLIP ViT-L/14 170K 3-S None (Textual boundary format) TimeChat 23 LLaMA-2 (7B) EVA-CLIP ViT-G/14 125K 1-S Sliding video Q-Former GroundingGPT 14 Vicuna v1.5 (7B) CLIP ViT-L/14 1.4M 3-S Modality adapters Chrono 18 BLIP-2 (OPT-2.7B) BLIP-2 ViT-g/14 \sim 12K 1-S None (Input sequence blueprint) Temporal Token Momentor 22 LLaMA (7B) CLIP ViT-L/14 10M 3-S TPM (Anchor interpolation) VTG-LLM 5 LLaMA-2 (7B) EVA-CLIP ViT-G/14 217K 1-S Absolute time tokens & slot compression LITA 9 Vicuna (7B) CLIP ViT-L/14 500K 1-S Time tokens; SlowFast visual tokens TRACE 6 Mistral (7B) CLIP ViT-L/14 2.8M 2-S Enc-Dec pairs (time/score/text) SeViLA 35 Flan-T5 XL (3B) ViT-G/14 (1B) 125K+ 2-S Keyframe selection via Localizer-Answerer Grounded-VideoLLM 25 Phi-3.5 / Vicuna CLIP + InternVideo2 2.3M 3-S Two-stream encoding with discrete tokens Continuous Temporal InternVideo2.5 28 InternLM2.5 (7B) InternViT-6B \sim 10M 3-S CG-DETR temporal regression head DisTime 37 InternVL2.5 (8B) InternViT-6B 1.25M 1-S Distribution time Enc & Dec (3-layer MLP) VideoMind 16 Qwen2-VL (7B) InternViT-6B 480K 3-S Timestamp decoder with Chain-of-LoRA