PEEK : P icking E ssential frames via E fficient K nowledge distillation

Killian Steunou, Anas Filali Razzouki, Khalil Guetari, Mounîm A. El-Yacoubi, Yannis Tevissen · 2026

Read PEEK : P icking E ssential frames via E fficient K nowledge distillation by Killian Steunou, Anas Filali Razzouki, Khalil Guetari, Mounîm A. El-Yacoubi, Yannis Tevissen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31029.txt.

Introduction

Steunou and colleagues tackle frame selection for video models, which can process only a limited number of frames. Uniform sampling is cheap but ignores visual content, while adaptive sampling costs more. Their PEEK distills frame relevance rankings from a teacher into a lightweight model that uses only visual content. On two captioning benchmarks, PEEK beats state-of-the-art methods, winning most configurations on ActivityNet Captions. Compared with baselines, PEEK is more accurate at low budgets and more efficient, adding about five percent to captioning time.

Catalog record

CREATORKillian Steunou, Anas Filali Razzouki, Khalil Guetari, Mounîm A. El-Yacoubi, Yannis Tevissen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46879
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.31029

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PEEK : P icking E ssential frames via E fficient K nowledge distillation Killian Steunou, Anas Filali Razzouki, Khalil Guetari, Mounîm A. El-Yacoubi, Yannis Tevissen Abstract Video-language models can process only a limited number of frames, making frame selection a key bottleneck for efficient video captioning. Most captioning pipelines still rely on uniform sampling, which is computationally cheap but agnostic to visual content. Adaptive frame sampling has recently emerged as a promising approach for selecting the most informative frames from a video; however, existing methods remain computationally expensive. We introduce PEEK, an efficient dynamic frame sampling method that distills caption-conditioned frame relevance rankings from a stronger teacher model into a lightweight temporal model that operates only on visual content. We find that, overall, on ActivityNet Captions and MSR-VTT, our method outperforms state-of-the-art methods across all evaluated downstream vision language models, especially when only one or two frames are selected for captioning, obtaining the best CIDEr for most frame budgets. On ActivityNet Captions, PEEK is particularly strong, winning 14 out of 16 configurations. Zero-shot evaluation on MSR-VTT shows that our model transfers best at low frame budgets, while results at four and eight frames are more mixed as temporal coverage and visual diversity become increasingly competitive. Compared with recent adaptive baselines, PEEK is both more accurate in the low-budget regime and more efficient: it adds only 5.2 % 5.2\% to the captioning time, compared with 65.4 % 65.4\% for CSTA and 211.9 % 211.9\% for MaxInfo. We release our code and pre-trained checkpoint at https://github.com/momentslab/peek . PEEK † † email: [email protected] † † email: [email protected] † † email: [email protected] † † email: [email protected] † † email: [email protected] † † affiliation: Télécom SudParis — SAMOVAR Institut Polytechnique de Paris Palaiseau, France † † affiliation: Moments Lab 69 Avenue Pierre Grenier Boulogne-Billancourt, France 1 Introduction {subfigure} Figure 1: Stage 1 — Oracle teacher scoring {subfigure} Figure 2: Stage 2 — query-free temporal scorer {subfigure} Figure 3: Stratified argmax inference with k=4 Figure 4: Overview of PEEK. (a) A frozen SigLIP 2 dual encoder acts as an Oracle teacher, producing per-frame relevance targets from ground-truth captions. (b) A small Transformer distills the teacher’s ranking into a query-free selector operating on MobileCLIP2 visual embeddings alone. (c) At inference, the segment is split into k equal temporal windows and the highest-scoring frame within each (blue dot) is kept. Modern vision–language models (VLMs) have made strong progress on image-language tasks, but video understanding remains expensive because videos are long, redundant, and often require sparse relevant cues to be extracted from a frame sequence [ 26 , 30 ] . To process a video, a VLM usually receives only a limited number of frames: enough to give a glimpse of the video, but not enough to guarantee that the decisive visual cue is present. In practice, even for state-of-the-art models, the default strategy is uniform sampling: partition the video into equal temporal segments and keep one frame from each [ 36 , 2 , 17 , 1 ] . Uniform sampling is deterministic, model-free, and often produces good results [ 10 , 33 , 11 ] , outperforming adaptive strategies on some benchmarks [ 4 ] . This makes uniform sampling a good baseline rather than a naive approach. However, it is still fundamentally content-blind: a short clip where the key event happens in a single instant and a clip where useful evidence is spread across the whole duration are treated identically [ 10 , 11 ] . Other works use a strong text-conditioned retriever, such as CLIP [ 21 ] or SigLIP [ 34 ] to score every frame by image–text similarity and keep the most relevant frames [ 23 ] . This scheme is query-dependent, however, which makes it impossible to consider at inference time for video captioning. Nonetheless, such a selector may be used to select visually relevant frames and diagnose how much they can improve captioning. To address this, we propose a caption-conditioned Oracle as a diagnostic for frame relevance: it scores candidate frames against the ground-truth caption and ranks them according to their semantic alignment with the target description. This Oracle cannot be used at inference time, since the target caption is unknown. However, its rankings provide a useful supervisory signal, indicating which frames are visually salient or semantically aligned with the caption, and may indirectly reveal temporally distinctive moments within the segment. Our hypothesis is that part of this caption-conditioned relevance can be distilled into a lightweight visual model that never sees text at inference time. Concretely, we propose PEEK, a two-stage distillation framework for efficient frame selection in video captioning. In the first stage, a frozen vision-language teacher scores candidate frames against the ground-truth caption, producing dense caption-conditioned relevance rankings used only for supervision. In the second stage, a lightweight temporal Transformer learns to predict these rankings from visual embeddings alone. At inference time, PEEK requires neither captions nor a text encoder: it scores the video visually and selects frames from the predicted relevance scores. Unlike query-aware or captioner-coupled frame selectors, PEEK uses text-conditioned relevance only as an offline supervision signal. The deployed selector is query-free, caption-agnostic, and independent of the downstream captioning model. We make the following contributions: • We introduce PEEK, a query-free frame selector that distills SigLIP 2 [ 28 ] caption-conditioned rankings into a lightweight temporal scorer operating only on visual features. • We propose caption-conditioned frame scoring as an Oracle diagnostic to quantify the value of semantic frame relevance for video captioning. • We evaluate PEEK on ActivityNet Captions [ 12 ] and MSR-VTT [ 32 ] with four downstream VLMs, showing consistent gains in the low-frame regime and a much lower selection cost than recent content-aware baselines. The paper is organized as follows. Section 2 reviews prior work on frame selection. Section 3 presents PEEK, Section 4 describes the datasets, experimental protocol, and results. Finally, we discuss the limitations of our approach and conclude the paper. 2 Related Work VLMs usually operate under a fixed visual-token or frame budget, which makes temporal sampling a central design choice rather than a neutral preprocessing step. Uniform sampling remains widely used because it is deterministic, model-free, and cheap. It is also a strong baseline: Brkic et al [ 4 ] have recently shown, in a controlled benchmark for small VLMs, that frame-sampling choices can substantially affect video question-answering results, and that uniform sampling is the strongest strategy on VideoMME [ 9 ] across the evaluated models. This reinforces the need to compare adaptive selectors against uniform sampling carefully, rather than treating it as a weak baseline. Nevertheless, uniform sampling ignores the large variation in information density across videos. Training-free frame selection. Recent work replaces uniform sampling with adaptive keyframe selection. Training-free methods often optimize a combination of informativeness, diversity, and temporal coverage. MaxInfo selects representative frames by maximizing the geometric volume spanned by frame embeddings, reducing redundancy while preserving visual diversity [ 13 ] . Other adaptive samplers combine text relevance with visual coverage [ 25 , 11 , 35 , 24 ] , which makes them query-dependent. These methods show that the choice of frames matters, but they are computationally expensive to run, as they require running

Read the complete text