CanViT: Toward Active-Vision Foundation Models
Yohaï-Eliel Berreby, Sabrina Du, Audrey Durand, B. Suresh Krishna · 2026
Read CanViT: Toward Active-Vision Foundation Models by Yohaï-Eliel Berreby, Sabrina Du, Audrey Durand, B. Suresh Krishna (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22570.txt.
Introduction
Berreby and co-authors introduce CanViT, a foundation model for active vision. Instead of one full look, active systems gather a scene through a chain of small glimpses. CanViT pairs a vision transformer with a scene-wide memory it calls the canvas. A label-free training scheme teaches it to rebuild rich scene features from random low-resolution glimpses. It was trained on millions of scenes and a billion glimpses. From one glimpse it already beats prior active models on segmentation, and more glimpses lift it further.
Catalog record
| CREATOR | Yohaï-Eliel Berreby, Sabrina Du, Audrey Durand, B. Suresh Krishna |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84549 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.22570
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CanViT: Toward Active-Vision Foundation Models Yohaï-Eliel Berreby, Sabrina Du, Audrey Durand, B. Suresh Krishna Abstract Active computer vision promises efficient, biologically plausible perception through sequential, localized glimpses, but lacks scalable general-purpose architectures and pretraining pipelines, leaving Active-Vision Foundation Models (AVFMs) underexplored. We introduce CanViT , the first task- and policy-agnostic AVFM. CanViT uses scene-relative RoPE to bind a retinotopic Vision Transformer backbone and a spatiotopic scene-wide latent workspace, the canvas . Efficient interaction with this high-capacity working memory is supported by Canvas Attention, a novel asymmetric cross-attention mechanism. We decouple thinking (backbone-level) and memory (canvas-level), eliminating canvas-side self-attention and fully-connected layers to achieve fast sequential inference and scalability to high output resolutions. We propose a label-free active vision pretraining scheme, policy-agnostic passive-to-active dense latent distillation : reconstructing scene-wide DINOv3 embeddings from sequences of low-resolution glimpses with randomized locations, zoom levels, and lengths. We pretrain CanViT-B from a random initialization on 13.2 million ImageNet-21k scenes—an order of magnitude more than previous active models—and 1 billion random glimpses, in 166 hours on a single H100. On ADE20K segmentation, a frozen CanViT-B achieves 38.5% mIoU in a single low-resolution glimpse, outperforming the best active model’s 27.6% with 20x fewer inference FLOPs as well as its FLOP- or input-matched DINOv3 teacher. Given additional glimpses, CanViT-B reaches 45.9% ADE20K mIoU. On ImageNet-1k classification, CanViT-B also sets a new active-vision state of the art, with 84.5% top-1 accuracy after fine-tuning. CanViT generalizes to longer rollouts, larger scenes, and new policies. Our work narrows the wide gap between passive and active computer vision, demonstrating the potential of task- and policy-agnostic AVFM pretraining. 1 Introduction Deep Artificial Neural Networks have achieved outstanding performance in a variety of computer vision tasks, and proven valuable to life sciences research as computational models of biological visual processing 64 , 63 , 51 , 71 , 7 , 36 , 46 . This practical and scientific success has largely hinged on vision encoders which process individual frames independently and passively, without the ability to reuse previous computation to guide further processing in a recurrent, active, human-like manner. Unlike most Artificial Neural Networks, humans sample their visual environment by actively and frequently orienting their sensory apparatus towards regions of interest, through gaze shifts. This process is inherently sequential, and involves strategic planning 66 , 24 , integration of evidence across time in visual working memory 6 , 37 , and top-down recurrent feedback—rich conditioning of early visual pathways by signals from higher brain areas, allowing each fixation’s visual information to be processed in a contextually informed manner, based on what was seen earlier 45 , 17 , 30 , rather than tabula rasa . Figure 1: A CanViT rollout. We consider a high-resolution scene ( A ). At each timestep t , CanViT ingests a 128^{2}\,\text{px} glimpse ( B , 1st row), a crop extracted at a viewpoint with center (x_{t},y_{t})\in[-1,+1]^{2} and scale (zoom level) s_{t}\in(0,1] . This updates a scene-wide latent representation, the canvas , with which CanViT integrates broad context and fine detail from variable-scale glimpses, extrapolates to unobserved regions, and conditions visual processing on a working understanding of the scene. We visualize the canvas via Principal Component Analysis across tokens ( B , 2nd row), and canvas updates (\Delta) as cosine dissimilarity heatmaps across consecutive timesteps ( B , 3rd row). Drawing inspiration from human vision, various Active Computer Vision models, which process visual scenes through sequential, localized glimpses, have been proposed 38 , 5 , 35 , 1 , 15 , 61 , 41 , 53 , 34 , 42 , 43 , 62 , 44 , 47 , 33 , 39 . These models often seek biological relevance and computational efficiency. Yet, despite their theoretical advantages, they have struggled to match the accuracy, efficiency, flexibility and representational richness of their passive counterparts. This disconnect is particularly striking on dense prediction, which is unsupported by most existing active vision models; the few exceptions lag dramatically behind passive models on standard benchmarks like ADE20K segmentation 70 when evaluated on it 42 , 43 . An active vision model can be considered along three axes: making sense of what it sees at a given moment (instantaneous vision); updating a persistent, evolving understanding of the scene, which may in turn inform further processing (memory); and deciding where and at what zoom level to look next (action selection). The first two axes define an observer’s ability to understand, persist, and recall visual inputs, while the last defines its input-selection strategy, or sensory policy. They do not play comparable roles: given perfect instantaneous vision and memory, naive and strategic policies should reach the same accuracy after enough glimpses of a static scene, but no policy can make up for a poor observer. Yet, prior Active Computer Vision work has often focused on action selection 38 , 5 , 35 , 1 , 15 , 61 , 41 , 53 , 34 , 42 , 43 , 62 , 44 . Concurrently, passive-vision foundation models like DINOv3 54 excel at general-purpose instantaneous vision but lack essential active-vision components, as they have no concept of localized glimpses or memory. Focusing on the intersection of instantaneous vision and memory, we sought to build a task- and policy-agnostic Active-Vision Foundation Model: a vision model capable of understanding the spatial and semantic structure of scenes across arbitrary sequences of glimpses, with rich representations that transfer across tasks and viewing policies. This policy-agnostic approach disentangles “how to see in an active-vision setting” from “where to look,” thus freeing active-vision pretraining from the complexity of Reinforcement Learning. Policy agnosticism may also be desirable for real-world deployment, where physical limitations (e.g., a motorized camera’s range of motion and optical zoom levels) may preclude the use of a general-purpose policy. Contributions. We introduce the Canvas Vision Transformer (CanViT, Figure 1 ), a recurrent architecture built around a latent scene-wide representation called the canvas (§ 4 ) and pretrained with a novel passive-to-active distillation scheme (§ 5 ). We evaluate CanViT-B on ADE20K semantic segmentation and ImageNet-1k 49 classification (§ 6 ). CanViT-B sets a new state of the art among active vision models on both ADE20K and ImageNet-1k while transferring across policies, temporal horizons, and canvas resolutions without retraining. 2 Related work Deep active vision models typically process sequences of glimpses —fixed- or variable-scale crops extracted from a larger image or video. This line of research traces back to the Recurrent Attention Model 38 , and remained largely confined to simple tasks such as digit recognition 5 , 1 until 2019, when Saccader 15 achieved 75% ImageNet-1k 49 top-1 accuracy by introducing an intermediate pretraining step to stabilize learning. GFNet 61 and AdaptiveNN 62 later showed that active vision could deliver computational efficiency gains on real-world tasks, although both remained structurally limited to classification tasks and fixed zoom levels. Dense prediction in active vision has remained underexplored, as most Active Computer Vision architectures cannot produce the scene-wide, spatially dense outputs required in tasks like semantic segmentation or depth estimation. Among the few exceptions, AME 42 and AdaGlimpse 43 achieve dense prediction through post-hoc expansion of encoder outputs: at each timestep, a MAE-style 20 Transformer decoder receives all encoded glimpse tokens with a full