GridProbe: Posterior-Probing for Adaptive Test-Time Compute in Long-Video VLMs

Mohamed Eltahir 1 Lama Ayash 1 Ali Habibullah 1 Tanveer Hussain 2 Naeemullah Khan 1 1 King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia 2 Department of Computer Science, Edge Hill University, Ormskirk, England {mohamed.hamid, lama.ayash, ali.habibullah}@kaust.edu.sa [email protected], [email protected] · 2026

Read GridProbe: Posterior-Probing for Adaptive Test-Time Compute in Long-Video VLMs by Mohamed Eltahir 1 Lama Ayash 1 Ali Habibullah 1 Tanveer Hussain 2 Naeemullah Khan 1 1 King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia 2 Department of Computer Science, Edge Hill University, Ormskirk, England {mohamed.hamid, lama.ayash, ali.habibullah}@kaust.edu.sa [email protected], [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10762.txt.

Introduction

Eltahir and four colleagues present GridProbe, a training-free way to spend compute adaptively when video-language models answer questions about long videos. Frames are arranged on a grid, row and column probes read each probe's peak answer confidence, and their outer product forms an importance map whose shape sets a per-question frame budget. On Video-MME-v2 it nearly matches a full pass at 3.36 times fewer floating-point operations.

Catalog record

CREATORMohamed Eltahir 1 Lama Ayash 1 Ali Habibullah 1 Tanveer Hussain 2 Naeemullah Khan 1 1 King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia 2 Department of Computer Science, Edge Hill University, Ormskirk, England {mohamed.hamid, lama.ayash, ali.habibullah}@kaust.edu.sa [email protected], [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40310
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10762

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GridProbe: Posterior-Probing for Adaptive Test-Time Compute in Long-Video VLMs Mohamed Eltahir 1 Lama Ayash 1 Ali Habibullah 1 Tanveer Hussain 2 Naeemullah Khan 1 1 King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia 2 Department of Computer Science, Edge Hill University, Ormskirk, England {mohamed.hamid, lama.ayash, ali.habibullah}@kaust.edu.sa [email protected], [email protected] Abstract Long-video understanding in VLMs is bottlenecked by a single monolithic forward pass over thousands of frames at quadratic attention cost. A common mitigation is to first select a small subset of informative frames before the forward pass; common for training-free selectors via auxiliary encoder-space similarities. Such signals are capped by contrastive pretraining, which usually fails on reasoning-heavy queries (negation, cross-frame counting, holistic summarization). We propose GridProbe , an efficient training-free posterior-probing inference paradigm that scores evidence in answer space using a frozen VLM’s own reasoning and then selects question-relevant frames adaptively, resulting in sub-quadratic attention cost with little to no accuracy loss. We arrange frames on a K × K K{\times}K grid and run lightweight row R and column C probes, where each probe reads its peak posterior as a query-conditioned confidence. The outer product of R and C yields an interpretable importance map whose skewness and kurtosis drive Shape-Adaptive Selection , a closed-form rule that reliably replaces the fixed frame budget M M with a per-question M eff M_{\mathrm{eff}} . We show empirically that M eff M_{\mathrm{eff}} , surprisingly, tracks intrinsic question difficulty without ever seeing the answer, a sign of test-time adaptive compute. On Video-MME-v2, GridProbe matches the monolithic baseline within 1.6 1.6 pp Avg Acc at 3.36 × 3.36\times TFLOPs reduction, while on LongVideoBench it Pareto-dominates the baseline ( + 0.9 +0.9 pp at 0.35 × 0.35\times compute). Because the selector and QA models can be decoupled, pairing a small 2B selector with a stronger 4B or 8B QA is strictly Pareto-dominant over the 2B monolithic baseline (up to + 4.0 +4.0 pp at 0.52 × 0.52\times compute, on average), with no retraining. Finally, the interpretability of the importance maps opens future avenues for behavioral diagnostics, grounding, and frame-selection distillation. Figure 1: (a) VMME-V2 Pareto across QA model sizes. GridProbe variants in the green region Pareto-dominate the 2B baseline. (b) Compute reduction across K at fixed 2B QA. † † footnotetext: Code: https://www.github.com/mohammad2012191/GridProbe 1 Introduction Modern video VLMs process long videos by compressing many frames into one forward pass. Qwen3-VL-2B 1 , for example, uses an adaptive per-frame resolution that crushes individual frames to \approx 240 visual tokens when 2048 frames are passed, an order of magnitude below the \approx 2960 tokens per frame at the 64-frame setting. This trade-off exchanges per-frame fidelity for temporal coverage and reflects a structural limit: per-token cost is dominated by the (linear-in-tokens) FFN at current scales while attention adds asymptotically quadratic-in-sequence-length growth on top, so reducing the number of input tokens delivers the strongest compute savings, and even models trained with 256K-token contexts cannot afford dense sampling and dense attention at scale. An orthogonal response is frame selection : pick the M\ll N most informative frames and run the VLM on only those. Recent training-free selectors (MDP3 9 , CLIP-matching, SigLIP-based scoring) and learned variants (Frame-Voyager 14 , Focus 16 , HFS 13 ) share a common structure: frames and the query are embedded by separate vision and text encoders, and a similarity function in that shared space scores each frame. We call this paradigm encoder-space selection . Its weakness is documented: MDP3’s own qualitative analysis shows SigLIP-matching failing on negation, cross-frame counting, and summarization queries, because these queries typically require reasoning outside the encoder’s representational capacity. We argue for a stronger move than swapping in a better selector. The VLM already knows which frames matter, it just needs to be asked. If we feed the VLM a subset of frames with the query, its posterior over the answer space encodes how confidently it can answer given that subset (Figure 3 ). High confidence on a small subset implies those frames carry the answer. This observation motivates a different inference paradigm rather than a different selector. To this end, we introduce GridProbe (Figure 2 ), a training-free posterior-probing inference paradigm that replaces the standard one-shot forward pass with a self-probing recipe. We factorize the candidate frame pool into a K\times K grid and run lightweight, axis-aligned probe passes over the rows and columns through a frozen VLM. The outer product of the row and column peak-posterior confidences yields a question-conditioned importance map. By default, the same frozen VLM serves as both the selector and the answerer. We further show that the two roles can be decoupled for a strict Pareto improvement. This single design shift has three structural consequences. First, the selection signal is reasoning-grounded , it inherits the VLM’s full reasoning capacity, so negation, cross-frame counting, and compositional queries are handled natively rather than being lost in contrastive embedding. Second, the signal scales with backbone capability without retraining, a stronger VLM automatically yields a sharper importance map. Third, the maps are mechanically interpretable, rendering the model’s evidence-gathering legible at the frame level. Notably, the current formulation reads a peak posterior over a finite answer space. Once frames are scored, selectors must determine how many to pass to the final model. Existing methods enforce a static budget M , creating an unavoidable trade-off: they waste compute on highly localized questions and bottleneck accuracy on holistic ones by discarding necessary context. Crucially, the GridProbe importance map resolves this natively. We demonstrate empirically that the shape of this importance distribution strongly correlates with question difficulty (Figure 5 , right). Rather than using a static frame budget, we utilize this insight to introduce shape-driven adaptive test-time compute , which sets the per-question size M_{\mathrm{eff}} via a closed-form rule on the map’s skewness and kurtosis. Coupling answer-space probing with shape-driven adaptive selection yields GridProbe , a training-free posterior-probing inference paradigm for long-video VLMs. Three findings anchor our empirical claims: (a) Pareto-dominant cross-model composition without retraining, (b) Pareto-efficient single-model operation, and (c) Adaptive test-time compute mirrors intrinsic difficulty. Contributions: Posterior-probing inference paradigm. We formalize GridProbe , a sub-quadratic training-free inference method for long-video VLMs that operates in answer space rather than encoder space, replacing the standard one-shot forward pass. Question-conditioned importance map. A per-question, frame-level importance map exposes the VLM’s evidence-gathering for each query, making long-video understanding interpretable. Shape-driven adaptive test-time compute. A closed-form statistic on the importance map distribution replaces the fixed frame budget M with a per-question M_{\mathrm{eff}} that adapts to the question difficulty. The Redundancy Principle. Positive-skew (sparse peaks) and negative-skew (redundant high-importance) maps are different distribution shapes that share the same selection answer. Figure 2: GridProbe pipeline. Stage 1: 2K row/column probes on K^{2} candidate frames yield an importance map. Stage 2: one focused pass on the top- M_{\mathrm{eff}} cells, sized adaptively from the map’s distribution shape. 2 Related Work Long-video VLMs and the cost of monolithic inference. Recent video VLMs such as Qwen3-VL 1 , InternVL3.5 10 , and LLaVA-Video 7 scale to thousands of frames via extended context windows combined with adaptive per-frame visual-token budgets. Despite design differences, all share a structural commitment to a single monolithic forward pass with quadratic attention in input length O(N^{2}) . Even

Read the complete text