One Trajectory, One Token: Grounded Video Tokenization via Panoptic Sub-object Trajectory

Chenhao Zheng, Jieyu Zhang, Mohammadreza Salehi, Ziqi Gao, Vishnu Iyengar, Norimasa Kobori, Quan Kong, Ranjay Krishna · 2025

Read One Trajectory, One Token: Grounded Video Tokenization via Panoptic Sub-object Trajectory by Chenhao Zheng, Jieyu Zhang, Mohammadreza Salehi, Ziqi Gao, Vishnu Iyengar, Norimasa Kobori, Quan Kong, Ranjay Krishna (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23617.txt.

Introduction

Chenhao Zheng and co-authors rethink how video is chopped into tokens for AI models in a 2025 paper. Current systems cut video into fixed space-time patches, which floods the model with tokens and still blurs what matters. Camera motion makes it worse. Their idea follows perception instead: a video's content is objects and parts moving along paths.

Their encoder, TrajViT, extracts the paths of sub-object parts under a panoptic view of the scene and turns each trajectory into one token. Token count then tracks scene complexity, not raw duration, and the tokens keep their meaning over time. Contrastive training teaches the encoder to match paths with descriptions. The payoff aimed for is long-video models that run cheaper without losing the thread of who did what.

Catalog record

CREATORChenhao Zheng, Jieyu Zhang, Mohammadreza Salehi, Ziqi Gao, Vishnu Iyengar, Norimasa Kobori, Quan Kong, Ranjay Krishna
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23617

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

One Trajectory, One Token: Grounded Video Tokenization via Panoptic Sub-object Trajectory Chenhao Zheng, Jieyu Zhang, Mohammadreza Salehi, Ziqi Gao, Vishnu Iyengar, Norimasa Kobori, Quan Kong, Ranjay Krishna Abstract Effective video tokenization is critical for scaling transformer models for long videos. Current approaches tokenize videos using space-time patches, leading to excessive tokens and computational inefficiencies. The best token reduction strategies degrade performance and barely reduce the number of tokens when the camera moves. We introduce grounded video tokenization , a paradigm that organizes tokens based on panoptic sub-object trajectories rather than fixed patches. Our method aligns with fundamental perceptual principles, ensuring that tokenization reflects scene complexity rather than video duration. We propose TrajViT , a video encoder that extracts object trajectories and converts them into semantically meaningful tokens, significantly reducing redundancy while maintaining temporal coherence. Trained with contrastive learning, TrajViT significantly outperforms space-time ViT (ViT3D) across multiple video understanding benchmarks, e.g., TrajViT outperforms ViT3D by a large margin of 6% top-5 recall in average at video-text retrieval task with 10x token deduction. We also show TrajViT as a stronger model than ViT3D for being the video encoder for modern VideoLLM, obtaining an average of 5.2% performance improvement across 6 VideoQA benchmarks while having 4x faster training time and 18x less inference FLOPs. TrajViT is the first efficient encoder to consistently outperform ViT3D across diverse video analysis tasks, making it a robust and scalable solution. 1 Introduction Figure 1 : (a) Traditional video tokenization divides a video into space-time patches, leading to large number of redundant tokens. (b) We propose to represent a video via panoptic sub-object trajectory, where each token represents a trajectory. With video comprising over 80% of global internet traffic [ 58 ] , designing efficient video encoders is essential. Since the transformer architecture [ 60 ] is now the standard for encoding video data [ 44 , 36 ] , efficient design hinges on the tokenization algorithm. Today’s de-facto tokenization algorithm splits the video tensor into space-time patches [ 44 ] as shown in Fig. 1 (a). In other words, the longer the video, the more tokens are extracted. This tokenization paradigm introduces significant challenges: First, it leads to memory bottlenecks, as a one-minute video sampled at just one frame per second could produce over 6000 tokens [ 2 ] . Second, it does not respect the scene complexity, treating all videos equally regardless of their visual content. For instance, a one-minute video of a static office desk without any motion produces the same number of tokens as a one-minute video of a football match with high-speed athleticism. Third, it places a heavy burden on the transformer encoder to connect disjoint tokens across frames as a singular concept ( e.g . , the same office desk). Researchers have proposed various token merging strategies to mitigate these challenges, but they are either not content-aware [ 50 , 3 , 2 ] or perform poorly when the camera taking the video is moving [ 11 ] . We depart from patch-based tokenization to propose a new paradigm of grounded video tokenization . Instead of naively splitting videos into patches to be tokenized, we align tokenization with the fundamental perceptual principles governing object perception and motion [ 61 , 56 , 46 ] by organizing tokens to correspond to panoptic sub-object trajectories. Rooted in Spelke’s core cognitive principles [ 56 ] and the Gestalt Principle of common fate , we treat object parts as coherent entities that persist over time (Fig. 1 b). By correlating token count with the number of distinct panoptic sub-object trajectories—rather than frame count—our method directly reflects the complexity of a scene in terms of its constituent objects and their interactions, allowing efficient processing of long sequences. Moreover, because our tokenization divides a video into objects rather than raw pixel divisions, it remains robust to variations in lighting conditions, occlusions, and camera movements—a challenge for patch-based tokenizers [ 4 , 27 , 28 ] . We operationalize this paradigm by designing TrajViT , a video encoder with grounded video tokenization and a standard transformer encoder (Fig. 2 ). Given a video, TrajViT extracts panoptic trajectories for all object parts and converts each into a fixed-size token. All tokens are then processed by a transformer encoder. In addition to its content-aware nature, TrajViT offers several advantages: (1) it significantly reduces the number of tokens—by an average of 10\times in the Panda dataset [ 8 ] —compared to space-time patch tokenization. (2) the tokens are organized into semantically meaningful units with no temporal redundancy, both of which facilitate high-level reasoning about object interactions for the encoder; (3) it makes the tokenization invariant to low-level visual variations in lighting and camera motion, leading to improved performance in video understanding tasks despite using fewer tokens. Figure 2 : Overview of TrajViT. Given a video, we first panoptically extract the trajectories for all objects. Our trajectory encoder converts these dynamic object trajectories into fixed sized embeddings, which serve as the input to the transformer encoder. In designing TrajViT, our core technical challenge lies in developing an efficient trajectory generation pipeline that can extract trajectories panoptically with sub-object granularity, and a module that can encode distinct trajectories into fixed-size tokens. We first develop a trajectory generation pipeline that integrates off-the-shelf segmenter [ 6 ] and tracker [ 49 ] through a parallelizable algorithm, which generates panoptic sub-object trajectories while minimizing latency caused by the tracking model. We also design a lightweight trajectory encoder that maps each distinct trajectories with variable number of pixels and appearing frames into fixed-size tokens, while maintaining the information about their appearance and temporal position variations. TrajViT can also process image data by treating each image segment as a trajectory of length one, allowing seamingless joint training with both videos and images. Finally, we train TrajViT with a contrastive learning objective on a large-scale video/image-text dataset. We compare TrajViT with standard ViT with space-time patch tokens (ViT3D) and state-of-the-art token merging methods on a wide range of video understanding tasks, including video-text retrieval, spatial-temporal detection, action classification, and temporal clip retrieval. It outperforms VIT3D in all tasks, while all token merging baselines underperform ViT3D on average. At the largest training scale, TrajViT surpasses ViT3D by a significant margin of 6% in top-5 recall with 10x fewer tokens. TrajViT is particularly effective because most video tasks are object-centric or require fine-grained recognition. For efficiency, despite the additional overhead of trajectory generation, TrajViT trains faster, consumes less GPU memory, and performs faster inference for videos with more than 64 frames. Lastly, we train two VideoLLMs by connecting Llama3 with TrajViT and ViT3D as video encoders. On six VideoQA benchmarks, the average accuracy of the TrajViT-LLM surpasses ViT3D-LLM by 5.24%, while being trained 4x faster and running at 18x fewer inference FLOPs. On the MovieChat benchmark [ 54 ] for long video evaluation, TrajViT-LLM displays better accuracy and efficiency than ViT3D-LLM when we scale up the number of input frames. 2 Related Work Video tokenization. In the field of video understanding, ViViT [ 2 ] introduced spatio-temporal patches to tokenize and process videos with transformers more efficiently than tokenizing video frames independently [ 47 , 40 ] . Subsequent research has exploited

Read the complete text