MambaEye: A Size-Agnostic Visual Encoder with Causal Sequential Processing

Changho Choi, Minho Kim, Jinkyu Kim · 2025

Read MambaEye: A Size-Agnostic Visual Encoder with Causal Sequential Processing by Changho Choi, Minho Kim, Jinkyu Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19963.txt.

Introduction

Changho Choi, Minho Kim, and Jinkyu Kim build MambaEye, an image encoder that accepts any input size. Human vision handles changing sizes naturally, but most encoders cannot, a gap the authors aim to close. Their design uses a pure Mamba2 backbone that processes patches in strict causal order, unlike earlier Mamba vision models that look both ways.

A relative move embedding records the shift between consecutive patches, giving translation invariance and freedom in resolution and scan pattern. A diffusion-inspired loss gives step-by-step supervision, training the model to grow confident as more of the image arrives. MambaEye stays robust across resolutions, above all at large sizes on ImageNet-1K classification, while time and memory grow only linearly with patch count.

Catalog record

CREATORChangho Choi, Minho Kim, Jinkyu Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-19963

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MambaEye: A Size-Agnostic Visual Encoder with Causal Sequential Processing Changho Choi, Minho Kim, Jinkyu Kim Abstract Despite decades of progress, a truly input-size agnostic visual encoder—a fundamental characteristic of human vision—has remained elusive. We address this limitation by proposing MambaEye , a novel, causal sequential encoder that leverages the low complexity and causal-process based pure Mamba2 backbone. Unlike previous Mamba-based vision encoders that often employ bidirectional processing, our strictly unidirectional approach preserves the inherent causality of State Space Models, enabling the model to generate a prediction at any point in its input sequence. A core innovation is our use of relative move embedding, which encodes the spatial shift between consecutive patches, providing a strong inductive bias for translation invariance and making the model inherently adaptable to arbitrary image resolutions and scanning patterns. To achieve this, we introduce a novel diffusion-inspired loss function that provides dense, step-wise supervision, training the model to build confidence as it gathers more visual evidence. We demonstrate that MambaEye exhibits robust performance across a wide range of image resolutions, especially at higher resolutions such as 1536 2 1536^{2} on the ImageNet-1K classification task. This feat is achieved while maintaining linear time and memory complexity relative to the number of patches. Figure 1 : MambaEye Resolution Scaling Our MambaEye-S (11M params) models are benchmarked against Mamba-based models of similar size. While deterministic scanning methods like FractalMamba++ 15 achieve higher peak accuracy at medium resolutions, our model demonstrates superior scaling at extreme resolutions. Notably, MambaEye outperforms FractalMamba++ at resolutions over 1280^{2} , despite using only a naive random sampling policy. This highlights our architecture’s inherent robustness and size-agnostic capabilities. 1 Introduction A central goal in computer vision is a truly input-size agnostic encoder. These encoders will ideally operate on arbitrary resolutions and allocates resources based on availability, without committing to a fixed input size. While high resolution media is becoming more common, most vision encoders are limited on their ability to work on various input dimensions. Instead, these encoders typically resize or crop during the data piping process, incurring unavoidable information loss which could be detrimental to the task. The disadvantages from this lack of versatility is already well-known in many models but is difficult to solve. Convolutional Neural Networks (CNNs) 14 ; 8 , while endowed with strong inductive biases, rely on fixed-size kernels whose effective receptive field grows slowly with depth, making global reasoning at high resolution costly. Vision Transformers (ViTs) 5 provide global receptive fields but incur quadratic complexity in the number of patches, which rapidly becomes prohibitive for native-resolution inputs 24 . Previous remedies such as linear/sparse attention 34 ; 16 ; 38 reduce the cost at the expense of expressivity. To complicate matters, standard ViT pipelines require resizing and often overfits to the image size of the training data by learning the absolute positional embeddings. Even with sophisticated training strategies such as FixRes 32 or relative encodings (e.g., RoPE) 31 ; 10 , the resolution brittleness persists. State Space Models (SSMs) such as Mamba 6 , offer a better chance of realizing an input-size agnostic encoder due to their linear-time scaling and constant-memory recurrent inference, making them attractive for long sequences. However, when adapted to vision, most approaches serialize images with fixed, hand-crafted scanning paths 13 (e.g., raster scanning), disrupting 2D locality. Many Mamba based encoders also rely on bidirectional processing to compensate 19 ; 39 , which forfeits the memory advantages of causal recurrence. Recent work on fractal/space-filling traversals improves locality 15 ; 33 but still enforces a predefined, holistic scan and does not eliminate the need to pick a fixed resolution up front. To have a truly size-agnostic model, we can once again turn to human vision perception. A human gathers information by dynamically scanning a scene, not by processing with a certain deterministic path. From this, unlike other architectures, we emphasize the importance of causal and sequential nature of vision perception. An ideal model should see the same part of the image multiple times, without a dependence in the order they are seen. Human vision rarely concludes the scene in one shot; instead, it cumulates information through time to build a confident understanding. With these motivations in mind, we propose MambaEye , a causal and sequential visual encoder built on from pure Mamba2 backbone 3 that directly addresses these issues. Instead of committing to a global, fixed traversal or training resolution, MambaEye processes an arbitrary sequence of patches and can emit a prediction at any step. Two design choices enable this: (1) a relative move embedding that encodes the spatial displacement between consecutive patches, providing a strong inductive bias for translation invariance and making the model independent of image resolution and scan order; and (2) a diffusion-inspired loss 11 ; 2 , conceptually related to flow matching 17 , that provides dense, step-wise supervision by scheduling targets from a uniform prior to the ground truth according to observed information. This design preserves the causal, recurrent nature of SSMs at inference, enabling constant memory per step and predictions at any point in the sequence. Interestingly, this can be achieved without the need of deterministic scanning patterns. For MambaEye, random sampling is much more successful compared to more sophisticated scanning protocols such as raster scan (Table 5 ). From this new paradigm, an additional point of improvement is introducing a new diffusion-inspired loss. This loss function guides the model to calibrate its confidence at any arbitrary timestep, which leads to improved generalization on longer sequences. This also incorporates the human vision perception where we cumulate information through time. Indeed, compared to the standard cross entropy loss (Table 4 ). Empirically, MambaEye achieves a competitive ImageNet-1K 28 results while exhibiting robust scaling across resolutions and outperforming previous Mamba based encoders, such as FractalMamba++ 15 at larger image sizes (Figure 1 ). Our primary contributions can be summarized as the following: 1. We propose MambaEye , a novel and efficient visual encoder that is size-agnostic and causal. 2. We introduce a relative move embedding technique that provides translation invariance and flexibility for arbitrary scanning paths. 3. We design a new diffusion-inspired loss function that provides dense, step-wise supervision for sequential encoders. 2 Related Work 2.1 Sequential and Active Vision Paradigms Sequential processing of visual information has long been studied as a biology-inspired alternative to holistic image encoding. Early work in recurrent attention models demonstrated that neural networks could learn to selectively attend to image regions through reinforcement learning, processing high-resolution images via sequences of focused glimpses 21 ; 1 . This paradigm is inspired by the saccadic movements of human eyes, in that rapid shift of attention across a visual scene rather than processing all information simultaneously is how humans perceive information. Recent advances have extended these ideas in several directions. The Continuous Thought Machine 4 explores how neural dynamics and synchronization can enable emergent reasoning capabilities, using sparse supervision at selected timesteps rather than dense feedback. Other approaches have investigated information-theoretic frameworks for modeling eye movements 26 and generative models that learn task-driven sequential processing strategies 22 . These works demonstrate that sequential visual processing can be both efficient and effective when properly supervised. 2.2 Network Architectures in Vision Encoders Vision encoders have

Read the complete text