CoPE-VideoLM: Codec Primitives For Efficient Video Language Models
Sayan Deb Sarkar, Rémi Pautrat, Ondrej Miksik, Marc Pollefeys, Iro Armeni, Mahdi Rad, Mihai Dusmanu · 2026
Read CoPE-VideoLM: Codec Primitives For Efficient Video Language Models by Sayan Deb Sarkar, Rémi Pautrat, Ondrej Miksik, Marc Pollefeys, Iro Armeni, Mahdi Rad, Mihai Dusmanu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-13191.txt.
Description
Video language models must fit long videos into limited context windows, and keyframe sampling misses both broad events and fine details. Sayan Deb Sarkar and six co-authors exploit video codec primitives, motion vectors and residuals, which natively encode redundancy and sparsity without full-image encoding for most frames. Lightweight transformer encoders aggregate those primitives and align them with image encoder embeddings through pretraining that speeds end-to-end fine-tuning convergence.
Their system, CoPE-VideoLM, cuts time-to-first-token by as much as 86% and token usage by as much as 93% versus standard video language models. Varying keyframe and codec primitive densities maintains or exceeds performance on 14 diverse video understanding benchmarks, spanning spatial scene understanding, long-form understanding, temporal and motion reasoning, and general question answering. Codec primitives thus offer an efficient, information-preserving alternative to sparse keyframes.
Catalog record
| CREATOR | Sayan Deb Sarkar, Rémi Pautrat, Ondrej Miksik, Marc Pollefeys, Iro Armeni, Mahdi Rad, Mihai Dusmanu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-13191