Evict3R: Training-Free Token Eviction for Memory-Bounded Streaming Visual Geometry Transformers
Soroush Mahdi, Fardin Ayar, Ehsan Javanmardi, Manabu Tsukada Mahdi Javanmardi, [0.5em] Amirkabir University of Technology (AUT), Tehran, Iran, [0.5em] Corresponding author: [email protected] · 2025
Read Evict3R: Training-Free Token Eviction for Memory-Bounded Streaming Visual Geometry Transformers by Soroush Mahdi, Fardin Ayar, Ehsan Javanmardi, Manabu Tsukada Mahdi Javanmardi, [0.5em] Amirkabir University of Technology (AUT), Tehran, Iran, [0.5em] Corresponding author: [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17650.txt.
Introduction
Soroush Mahdi, Fardin Ayar, Ehsan Javanmardi, Manabu Tsukada, and Mahdi Javanmardi bound the memory of streaming 3D vision models. Their 2025 arXiv paper starts from StreamVGGT, a transformer that reads video and builds three-dimensional understanding as frames arrive. Its key-value memory grows without limit, which caps how long a stream it can handle.
Their policy, Evict3R, discards redundant tokens at inference time and keeps the most informative ones, with no retraining. On long sequences in the 7-Scenes data set, peak memory fell by about half while accuracy barely moved. Under tight budgets, eviction frees room to sample frames more densely, which can even raise reconstruction quality. Tests span video depth, 3D reconstruction, and camera pose tasks. The authors offer the policy as a practical route to long-horizon streaming perception.
Catalog record
| CREATOR | Soroush Mahdi, Fardin Ayar, Ehsan Javanmardi, Manabu Tsukada Mahdi Javanmardi, [0.5em] Amirkabir University of Technology (AUT), Tehran, Iran, [0.5em] Corresponding author: [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-17650
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evict3R: Training-Free Token Eviction for Memory-Bounded Streaming Visual Geometry Transformers Soroush Mahdi, Fardin Ayar, Ehsan Javanmardi, Manabu Tsukada Mahdi Javanmardi, [0.5em] Amirkabir University of Technology (AUT), Tehran, Iran, [0.5em] Corresponding author: [email protected] Abstract Streaming visual transformers like StreamVGGT achieve strong 3D perception but suffer from unbounded growth of key–value (KV) memory, which limits scalability. We propose a training-free, inference-time token eviction policy that bounds memory by discarding redundant tokens while keeping the most informative ones. Our method uses significantly less memory with little to no drop in accuracy: on 7-Scenes with long sequences it reduces peak memory from 18.63 GB to 9.39 GB while accuracy and completeness drop by only 0.003. Under strict memory budgets, eviction enables denser frame sampling, which improves reconstruction accuracy compared to the baseline. Experiments across video depth estimation (Sintel, KITTI), 3D reconstruction (7-Scenes, NRGBD), and camera pose estimation (Sintel, TUM-dynamics) show that our approach closely matches StreamVGGT at a fraction of the memory and makes long-horizon streaming inference more practical. I Introduction Robotic platforms rely on dense, view-consistent 3D perception for navigation, manipulation, and long-horizon autonomy. Classical Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipelines deliver accurate reconstructions via global optimization [ 1 , 2 ] , but their iterative bundle adjustment and repeated data association hinder real-time operation and incremental updates on resource-constrained hardware. Recent learning-based approaches replace hand-crafted geometry with end-to-end predictors that infer depth, pose, and point maps directly from images [ 3 , 4 , 5 ] . Among these, large feed-forward transformers achieve strong accuracy by reasoning jointly across many views, but they are typically offline: each new frame requires re-encoding the entire sequence, making them ill-suited to streaming robotics workloads. Causal, streaming variants have emerged to close this gap. StreamVGGT adopts the autoregressive paradigm of language models: each incoming frame attends to a cached key–value (KV) memory of all prior frames, avoiding redundant recomputation and enabling low-latency updates [ 6 ] . However, the same KV cache that boosts throughput also causes a fundamental scalability bottleneck: its size increases with sequence length. Even with fewer than 10 frames, the KV cache size can exceed the model size! Other streaming designs impose fixed external memories or explicit spatial pointer stores [ 7 , 8 , 9 ] , but capacity limits introduce information loss and drift, and the required architectural changes complicate adoption. We ask: Can a streaming visual transformer preserve the benefits of long-range context while bounding its internal KV memory, without retraining or modifying model weights? I- 1 Approach We introduce a training-free, inference-time token budgeting and eviction mechanism for StreamVGGT that (i) enforces per-layer KV-cache budgets and (ii) selects tokens for retention using an attention-based importance score. We further allocate the total budget across global-attention layers using an attention-sparsity prior, reflecting the empirical observation that early/late layers attend more densely than middle layers, in line with long-context LLM findings [ 10 , 11 , 12 , 13 ] . Our experiments show that on video depth, multi-view reconstruction, and camera pose, our method matches StreamVGGT accuracy at moderate budgets while substantially reducing peak memory and per-frame latency; critically, it enables much longer sequences (e.g., 2{\times} – 10{\times} ) on a single GPU. Also, qualitative reconstructions remain complete and consistent under tight budgets (Fig. 1 ). (a) StreamVGGT (b) Ours ( B{=}0.2 ) Fig. 1 : Qualitative 3D pointmap comparison. Reconstructions from StreamVGGT and our method with B{=}0.2 . Despite using less than half the memory, our method shows no visible drop in quality. II Related Work II- 1 From Conventional to Learning-Based Reconstruction Classical 3D reconstruction pipelines, including Structure-from-Motion (SfM) and Multi-View Stereo (MVS), achieve high accuracy through sophisticated global optimization procedures [ 1 , 2 ] . These methods, however, suffer from computational limitations that restrict their applicability to scenarios requiring real-time processing or incremental updates. In dynamic environments or scenes with limited texture, feature matching and photometric consistency deteriorate, further compromising system robustness. These fundamental limitations have motivated the development of end-to-end learning-based approaches that directly infer 3D geometry from visual inputs. Pairwise reconstruction methods, exemplified by DUSt3R [ 3 ] demonstrate the capability to predict view-consistent point maps from image pairs. While effective for two-view scenarios, pairwise approaches inherently lack the global context necessary for handling complex multi-view sequences with occlusions and varying viewpoints. Large-scale feed-forward transformers, such as Fast3R [ 4 ] and VGGT [ 5 ] , address this limitation by enabling comprehensive multi-view reasoning through single forward passes that consider all views simultaneously. However, these transformer-based models operate exclusively in offline mode, requiring complete sequence re-encoding whenever new frames arrive, which fundamentally prevents real-time streaming applications. II- 2 Streaming 4D Reconstruction Real-time 4D reconstruction has motivated streaming methods that process continuous video with a persistent state while remaining efficient. Memory-augmented approaches such as CUT3R [ 7 ] maintain explicit spatial state to predict pointmaps in a common frame. More recent works— Spann3R [ 8 ] and Point3R [ 9 ] —introduce stronger memory management at the cost of architectural complexity. Spann3R employs a two-tier memory (dense working + sparse long-term) and evites its long-term store by retaining top- k items according to accumulated attention , keeping the long-term memory bounded. Point3R maintains an explicit spatial pointer memory and fuses nearby pointers under a distance schedule to suppress redundancy as the scene grows; this curbs growth but does not guarantee a fixed bound. StreamVGGT [ 6 ] takes another path: temporal causal attention with a cached KV memory enables efficient autoregressive updates, but the cache grows linearly with sequence. We manage KV memory of StreamVGGT with a training-free, inference-time policy that enforces per-layer token budgets , ranks keys by attention with exposure/length normalization, and preserves a fixed set of important tokens. Compared to v ’s attention-based eviction, ours adds per-layer control and the mentioned normalizations, remaining plug-and-play. In contrast to Point3R ’s explicit pointer memory, our method is architecture-agnostic and enforces a strict, per-layer memory budget. By operating directly on the key and value (KV) cache within StreamVGGT, our approach avoids the need to reproject features into keys and values at each step, resulting in lower computational overhead and seamless integration with existing transformer-based architectures. Fig. 2 : Overview of our token eviction framework. (A) Tokens in each layer’s KV cache are ranked by attention importance and evicted under a per-layer budget. (B) Importance scores are derived from query–key attention statistics. (C) Scores are normalized to guide consistent token selection across layers. II- 3 Token Management and Eviction Token reduction is essential for managing the computational demands of transformer architectures. In computer vision, early methods such as token merging (e.g., ToMe [ 14 ] ) and token pruning (e.g., DynamicViT [ 15 ] ) were developed to reduce redundancy and focus computation on informative tokens. More recent approaches combine these strategies, as in PuMer [ 16 ] , to further improve efficiency. Key-value (KV) caching is a core technique in transformer models with causal attention, such as those used in large language models (LLMs). In causal attention, each new token attends only to previous tokens, enabling efficient autoregressive inference by storing and reusing the keys and values (the ”KV cache”) from earlier steps.