Atlas : Multi-Scale Attention Improves Long Context Image Modeling

Kumar Krishna Agrawal, Long Lian, Longchao Liu, Natalia Harguindeguy, Boyi Li, Alexander Bick, Maggie Chung, Trevor Darrell, Adam Yala · 2025

Read Atlas : Multi-Scale Attention Improves Long Context Image Modeling by Kumar Krishna Agrawal, Long Lian, Longchao Liu, Natalia Harguindeguy, Boyi Li, Alexander Bick, Maggie Chung, Trevor Darrell, Adam Yala (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12355.txt.

Introduction

Kumar Krishna Agrawal and eight co-authors wrote a 2025 paper on Atlas, a neural architecture that reads very large images efficiently. Modeling massive images at full resolution strains standard attention, whose cost balloons with pixel count. Their method, multi-scale attention, attacks the cost structurally.

The image is represented at a ladder of progressively coarser scales, only logarithmically many. Cross-attention lets information flow both ways between scales, so fine detail and broad context inform each other. On a high-resolution ImageNet variant, Atlas nearly matched a leading convolutional model's accuracy at 1024 pixels while running 4.3 times faster. Against other long-context rivals it was both faster and more accurate. At 4096 pixels it beat a state-space competitor by 32 accuracy points at similar speed. Code and pretrained models are public.

Catalog record

CREATORKumar Krishna Agrawal, Long Lian, Longchao Liu, Natalia Harguindeguy, Boyi Li, Alexander Bick, Maggie Chung, Trevor Darrell, Adam Yala
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-12355

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Atlas : Multi-Scale Attention Improves Long Context Image Modeling Kumar Krishna Agrawal, Long Lian, Longchao Liu, Natalia Harguindeguy, Boyi Li, Alexander Bick, Maggie Chung, Trevor Darrell, Adam Yala Abstract Efficiently modeling massive images is a long-standing challenge in machine learning. To this end, we introduce Multi-Scale Attention (MSA). MSA relies on two key ideas, (i) multi-scale representations (ii) bi-directional cross-scale communication. MSA creates 𝒪 ⁡ ( l ​ o ​ g ​ N ) \mathcal{O}(logN) scales to represent the image across progressively coarser features and leverages cross-attention to propagate information across scales. We then introduce Atlas , a novel neural network architecture based on MSA. We demonstrate that Atlas significantly improves the compute-performance tradeoff of long-context image modeling in a high-resolution variant of ImageNet 100. At 1024px resolution, Atlas-B achieves 91.04% accuracy, comparable to ConvNext-B (91.92%) while being 4.3x faster. Atlas is 2.95x faster and 7.38% better than FasterViT, 2.25x faster and 4.96% better than LongViT. In comparisons against MambaVision-S, we find Atlas-S achieves 5%, 16% and 32% higher accuracy at 1024px, 2048px and 4096px respectively, while obtaining similar runtimes. Code for reproducing our experiments and pretrained models is available at https://github.com/yalalab/atlas . † † affiliationnotice: Equal contribution 1 Introduction Figure 1: (a) Training efficiency comparison of different vision architectures on HR-IN100 across increasing input resolutions (1024-4096px). (b) Atlas exhibits similar runtime scaling as MambaVision while obtaining significantly better accuracy. Long-context image modeling remains a fundamental challenge in computer vision with broad applications to biomedicine ( Xu et al., 2024 ) , satellite imagery ( Rad, 2024 ) , and vision-language modeling ( Gemini-Team et al., 2023 ; Wang et al., 2024 ; Qwen-Team, 2025 ; Chen et al., 2024 ) . At the core of this challenge is a compute expressivity trade-off; we aim to develop models that efficiently scale to massive input sequences while capturing arbitrary pair-wise dependencies between input tokens. As shown in Figure 1 (a), self-attention, as used in Vision Transformers, is highly expressive, but its computational cost scales poorly (i.e., quadratically) with sequence length. It remains infeasible to train end-to-end Vision Transformers on massive imaging modalities such as mammograms or whole-slide pathology images. At another end of the spectrum, state space models (SSMs) and recurrent architectures are highly efficient, achieving linear computational complexity; however, SSM-based models perform poorly in long-context imaging modeling ( Figure 1 b). Long-context image modeling requires novel neural primitives and new benchmarks to guide their development. Recent work in efficient architecture design, such as FasterViT ( Hatamizadeh et al., 2023 ) and MambaVision ( Hatamizadeh & Kautz, 2024 ) , has primarily focused on improving the throughput vs accuracy trade-offs in the context of standard resolution ImageNet experiments ( 224\times 224 ). While valuable, this setting yields little insight into how methods scale to larger input resolutions. To this end, we propose a new high-resolution benchmark based on ImageNet-100 (HR-IN 100). We evaluate the speed vs accuracy trade-off of different neural networks at progressively larger resolutions, ranging from 1024\times 1024 to 4096\times 4096 images. As input resolution increases, long-range communication across distant parts of the image becomes more essential for image classification, and asymptotic computational complexity begins to dominate model runtime. In designing a novel neural primitive, we aim to enable arbitrary cross-token interaction with minimal intermediate steps (i.e., communication complexity) while minimizing computational complexity as a function of input sequence length. To this end, we propose Multiscale Attention (MSA), a novel primitive for high-resolution image modeling. MSA is built on two key ideas: multiscale representations and cross-scale communication. In each MSA block, we leverage a simple S -token max-pooling kernel to summarize small spatial regions (e.g., 4x4 input region), into progressively coarser summary representations across O(\log_{S}N) spatial scales, where N is the total sequence length. We then leverage a windowed cross-attention mechanism to enable information-sharing between tokens at different scales. At each scale, tokens attend to nearby tokens of the same scale and tokens from all coarser scales. This “top-down” communication enables MSA to integrate information across the entire sequence. Each scale’s tokens also cross-attend to its “parent” finer-grain scale tokens, allowing each coarse token to refine its representation through “bottom-up” communication. Altogether, this bi-directional communication pattern enables information mixing between all input tokens through O(\log N) intermediate tokens (i.e. coarse scale representations) and within O(N\log N) runtime. In controlled block-level experiments (see Table 3 ), we find that MSA outperforms alternative neural network primitives in long-context modeling, including LongNet’s dilated attention ( Ding et al., 2023 ) , MambaVision Mixer ( Hatamizadeh & Kautz, 2024 ) , and FasterViT’s Hierarchical Attention ( Hatamizadeh et al., 2023 ) . We propose Atlas , a novel architecture designed around the unique advantages of MSA. Given a sequence length N, which defines \log N scales within MSA, Atlas leverages \log N macro-stages to progressively down-sample the input until MSA recovers only a single scale. We leverage the rich scale-2 representations of our MSA block as a down-sampling mechanism, enabling both faster and more performant down-sampling than traditional approaches. We demonstrate that Atlas significantly improves the Pareto frontier in long-context modeling. In 1024\times 1024 experiments, as shown in Table 1 , Atlas obtains comparable runtime to MambaVision (23.1hr vs 22.6hr) on the same hardware, while obtaining 6.1% higher accuracy (91.04 vs 84.86). Compared to FasterViT and LongViT, Atlas is 2.95\times and 2.25\times faster, obtaining 7.38% (91.04 vs 83.66) and 4.96% (91.04 vs 86.08) higher accuracy, respectively. Moreover, the performance advantage of Atlas is especially pronounced as we scale input resolution to 4096px, achieving a 34% accuracy improvement over MambaVision at similar runtime. We summarize our contributions as follows: • We propose a High-Res ImageNet-100 (HR-IN 100), an efficient benchmark with input resolutions ranging from 1024\times 1024 to 4096\times 4096 for evaluating the frontier of long-context image modeling. • We introduce Multi-Scale Attention (MSA), a novel neural network primitive that maintains representations across O(\log N) spatial scales and enables bi-directional information mixing across all scales within O(N\log N) runtime. Building on MSA, we introduce Atlas , a novel neural network architecture. • With extensive experiments on High-Res ImageNet-100, we demonstrate that Atlas improves the Pareto frontier in long-context image modeling. Atlas outperforms representative efficient architectures in long-context image modeling, including FasterViT, MambaVision, and LongViT. Figure 2: The Atlas architecture consists of a convolutional stem for initial feature extraction, followed by a series of Multi-Scale Attention (MSA) blocks that progressively downsample the feature maps while preserving global context. This hierarchical design facilitates the effective processing of high-resolution images with efficient communication between features. 2 Related Work Vision Transformers (ViTs). ViTs ( Dosovitskiy, 2020 ) directly apply Transformers ( Vaswani, 2017 ) architecture to image patches, demonstrating the effectiveness of self-attention in visual data processing. Building on this, DeiT ( Touvron et al., 2021 ) improves training data efficiency. However, the self-attention primitive in ViT, which scales quadratically with input sequence length, limits its application toward high-resolution imaging. Our study focuses on developing efficient alternatives to standard self-attention to enable expressive and computationally efficient long-context image modeling. Efficient Long Sequence Modeling in Language. To address the challenges of long-sequence modeling, LongNet ( Ding et al., 2023 ) introduces a dilated attention mechanism, allowing transformers to process

Read the complete text