Attention and Compression is all you need for Controllably Efficient Language Models

Jatin Prakash, Aahlad Puli, Rajesh Ranganath · 2025

Read Attention and Compression is all you need for Controllably Efficient Language Models by Jatin Prakash, Aahlad Puli, Rajesh Ranganath (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05313.txt.

Introduction

Jatin Prakash and co-authors propose language models whose cost and recall can be tuned at run time, not fixed in training. Cheap mixers like sparse attention and convolutions cut inference cost, but often lose in-context recall, the skill of using long prompts well.

Fixing that trade-off in advance means some uses get too little memory and others pay for speed they do not need. Their controllable design lets one model shift its balance per task. That flex could let a single model serve both quick chat and deep document work.

Catalog record

CREATORJatin Prakash, Aahlad Puli, Rajesh Ranganath
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-05313

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Attention and Compression is all you need for Controllably Efficient Language Models Jatin Prakash, Aahlad Puli, Rajesh Ranganath Abstract The quadratic cost of attention in transformers motivated the development of efficient approaches: namely sparse and sliding window attention, convolutions and linear attention. Although these approaches result in impressive reductions in compute and memory, they often trade-off with quality, specifically in-context recall performance. Moreover, apriori fixing this quality-compute tradeoff in an architecture means being suboptimal from the get-go: some downstream applications require more memory for in-context recall, while others require lower latency and memory. Further, these approaches rely on heuristic choices that artificially restrict attention, or require handcrafted and complex recurrent state update rules, or they must be carefully composed with attention at specific layers to form a hybrid architecture that complicates the design process, especially at scale. To address above issues, we propose C ompress & A ttend T ransformer ( cat ), a conceptually simple architecture employing two simple ingredients only: dense attention and compression. cat decodes chunks of tokens by attending to compressed chunks of the sequence so far. Compression results in decoding from a reduced sequence length that yields compute and memory savings, while choosing a particular chunk size trades-off quality for efficiency. Moreover, cat can be trained with multiple chunk sizes at once, unlocking control of quality-compute trade-offs directly at test-time without any retraining, all in a single adaptive architecture. In exhaustive evaluations on language modeling and common-sense reasoning, in-context recall, and long-context understanding, a single adaptive cat model outperforms many existing efficient baselines, including hybrid architectures, across different compute-memory budgets. Further, a single cat matches dense transformer in language modeling across different model scales while being 1.4 − 3 × 1.4-3\times faster and requiring 2 − 9 × 2-9\times lower total memory usage. Play with cat s at: github.com/rajesh-lab/cat-transformer 1 Introduction Figure 1: cat unlocks test-time control of quality-efficiency trade-offs , where a single adaptive cat model (all red dots come from a single model) outperforms nearly every popular efficient architecture on real-world in-context recall tasks across varying compute-memory budgets. Transformers ( Vaswani et al. 2017 ) are the default architectures for large language models (LLMs) and rely on the powerful self-attention mechanism ( Bahdanau et al. 2014 ) . However, the compute required for decoding with self-attention grows quadratically with the sequence length, with memory costs growing linearly, making transformers expensive to deploy. Given the cost of self-attention, there has been interest in designing efficient alternatives: while approaches like sparse and sliding window attention ( Child et al. 2019 ; Zaheer et al. 2020 ; Jiang et al. 2023 ) heuristically restricts the tokens being attended to; approaches like linear attention ( Katharopoulos et al. 2020 ; Arora et al. 2024a ; Dao & Gu 2024 ; Yang et al. 2025b ) rely on fixed-size recurrent states with complex state update rules, to enable constant compute and memory costs. However, restricting attention to tokens apriori or using fixed-size recurrent states, that have problems managing information over long sequences, hurt in-context recall performance ( Arora et al. 2024a ; Jelassi et al. 2024 ; Wen et al. 2024 ) . To make these approaches performant, they require careful composition with dense attention at specific layers, making the design process cumbersome especially at scale ( Waleffe et al. 2024 ; Wang et al. 2025 ) . Enabling efficiency by recursively compressing the sequence can avoid fixed-memory bottlenecks and heuristic restrictions ( Rae et al. 2020 ; Chevalier et al. 2023 ) , but sequential computations in these approaches makes the training slow and optimization difficult ( Geiping et al. 2025 ) . Figure 2: The Compress and Attend Transformer ( cat ) architecture. cat chunks up a sequence of length N into N/C chunks of C tokens (illustrated for C=3 ). Each chunk is parallelly compressed into a chunk representation. cat then decodes each chunk by attending to past chunk representations. Compression results in a reduced sequence length enabling compute and memory savings during decoding. Chunk size in cat acts as knob, offering test-time control of quality-efficiency trade-offs, where higher chunk sizes result in increased efficiency. Additionally, existing approaches do not account for the differences in compute and memory requirements across diverse downstream tasks. For example, writing short email replies does not require strong in-context recall performance and usage of linear attention can be sufficient; but code auto-completion demands accurate recall of function names from the entire code repository in the context, where more memory and compute of dense attention may be preferred. The existing approaches for efficiency fix the compute-memory usage before training. This means if at test time a problem demands a higher budget for better performance, a whole new model needs to be trained. Training multiple models with different tradeoffs is one way to tackle this problem but repeating this for every downstream task can become quickly prohibitive. Even if such models were available, routing between these models based on the context requires holding all of them in memory. To address the issues raised above, we propose a conceptually simple architecture: C ompress & A ttend T ransformer ( cat ) that employs two simple well-known ingredients, namely dense attention and compression. cat compresses chunks of tokens in parallel into a shorter sequence using a compressor , which a decoder then attends to while autoregressively modeling the tokens in the latest chunk; both compressor and decoder are simple dense transformers themselves (see Figure 2 ). With the compression and decoding being parallel over tokens during training, there is no recurrence along the sequence dimension, which enables end-to-end scalable training . Decoding from the reduced sequence length due to compression enables compute and total memory savings . This reduction allows the use of more parameters in cat , thereby improving model quality. 1 1 1 similar in vein to flagship models ( Yang et al. 2025a ; Agarwal et al. 2025 ) that are all parameterized as Mixture-of-Experts (MoEs) ( Shazeer et al. 2017 ) , where additional model parameters does not mean more compute, and brings improvements in quality. Choosing a particular chunk size in cat trades-off quality for compute and memory (see Figure 1 ). At the same time, the memory grows gracefully , linearly with sequence length but at a significantly slower rate, to enable in-context recall performance at long sequence lengths. Importantly, training cat s across multiple chunk sizes at once unlocks control of quality-compute trade-offs directly at test-time without any retraining, all in a single adaptive architecture. To summarize, this paper: • Introduces the cat architecture to efficiently model sequences by decoding each chunk of tokens given parallelly compressed representations of the past chunks. Adjusting a single knob (chunk size) at test-time controls quality-efficiency trade-offs, allowing a single cat model to interpolate between the dense transformer and efficient alternatives without any retraining. • Provides a parallel and scalable implementation for training cat s (we scale from 90M to 1B parameters) and an efficient pure PyTorch implementation for generation that does not require any custom CUDA or Triton kernels, unlike most efficient baselines. We release code at: github.com/rajesh-lab/cat-transformer • Demonstrates that a single adaptive cat model – outperforms many popular

Read the complete text