Compute Where it Counts: Self Optimizing Language Models

Yash Akhauri, Mohamed S. Abdelfattah · 2026

Read Compute Where it Counts: Self Optimizing Language Models by Yash Akhauri, Mohamed S. Abdelfattah (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10875.txt.

Introduction

Yash Akhauri and Mohamed S. Abdelfattah argue that language models waste compute by spending the same effort on every token, easy or hard. Their Self-Optimizing Language Models pair a frozen base model with a small policy network that reads hidden states and picks an efficiency move at each decoding step.

Each move can thin attention, prune activations in the feed-forward layers, or lower quantization precision, all without touching base weights. The policy trains by comparing counterfactual compute schedules for the same token path, balancing quality against a target budget. Across models and budgets, the method beats static compression. Inference engineers get per-token thrift that follows difficulty instead of fighting it.

Catalog record

CREATORYash Akhauri, Mohamed S. Abdelfattah
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56631
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10875

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Compute Where it Counts: Self Optimizing Language Models Yash Akhauri, Mohamed S. Abdelfattah Abstract Efficient LLM inference research has largely focused on reducing the cost of each decoding step (e.g., using quantization, pruning, or sparse attention), typically applying a uniform computation budget to every generated token. In practice, token difficulty varies widely, so static compression can over-compute on easy steps and under-compute on hard ones. We study dynamic budget allocation for autoregressive decoding: learning how much computation to spend per token from within a single model. Self-Optimizing Language Models (SOL) pair a frozen LLM with a lightweight policy network that reads the LLM hidden state and selects a discrete efficiency action at each decode step. Actions can jointly control (i) token-level attention sparsity, (ii) structured activation pruning in the MLP, and (iii) activation quantization bit-width, while leaving the base model weights unchanged. We train the policy with group-relative policy optimization on teacher-forced episodes: the token sequence is fixed, while we sample multiple compute schedules (i.e., “counterfactual” schedules that vary only the efficiency actions for the same token path) and compare their likelihoods under the same supervision. Our reward trades off language‑model quality against soft penalties that encourage episode‑average budget usage to match a requested target. Across model variants and compute regimes, SOL improves quality at matched budget over static allocation and strong random schedule search, offering a complementary axis for inference‑efficiency optimization. SOL discovers a better quality-efficiency pareto-front across all our experiments and improves MMLU accuracy by up to 7.3% over uniform budget allocation strategies. [Code] Figure 1 : (Top Left) The policy is a small autoregressive transformer that uses the LLM’s own hidden state to decide how much compute to allocate per token . (Bottom Left) At training time, the policy samples multiple counterfactual compute schedules (trajectories) and learns to optimize quality under a requested budget target. (Right) At inference, the controller incurs a minimal, constant overhead (proportional to the control horizon, e.g., 16 decode steps), enabling fine-grained control over per-step efficiency. 1 Introduction Deploying LLMs at scale has driven efficiency research to make each decoding step cheaper via quantization ( 7 ; 18 ; 27 ; 43 ) , pruning ( 26 ; 1 ; 8 ) , low-rank compression ( 5 ; 6 ; 41 ) , sparse attention patterns ( 9 ; 40 ; 2 ; 42 ; 23 ) , speculative decoding ( 21 ) , and related techniques. Most of these methods, however, apply essentially the same compute budget to every generated token. In practice, token difficulty varies widely: some steps are locally predictable, while others depend on long‑range context or precise intermediate computations. A fixed per‑token budget therefore tends to over‑compute on easy steps and under‑compute on harder steps. A fixed per-token budget drives the model to often either over, or under-compute. For example, with token sparsity; some tokens require longer-range dependencies whereas others do not, providing an opportunity to dynamically determine computation budget through the number of tokens attended to with each generated token. This also applies to other LLM compression methods such as quantization, weight sparsity, and low-rank factorization. Controlling the compression rate based on token generation difficulty has the potential to unleash a new axis of efficiency in LLMs. One main challenge is that per‑token efficiency decisions interact across time. Each generated token can be attended to by future tokens, therefore, aggressively compressing or dropping information that seems safe at step t can cause degradation in quality several steps later. This motivates treating compute allocation as a sequential decision problem rather than a per‑step or step-agnostic heuristic. To address this, we propose Self‑Optimizing Language Models (SOL). The base LLM is kept frozen and we add a lightweight policy model that reads the LLM hidden state and simple progress/budget features, then selects a discrete efficiency action at each decoding step. Each action instantiates a compute regime inside the same model by controlling (i) token‑level attention sparsity, (ii) structured MLP activation pruning, and/or (iii) activation quantization bit‑width—without changing the base weights. A key design goal is deployment-time controllability. Rather than hard-coding a single operating point, we condition the policy on a requested compute budget. During training, we sample per-sequence budget targets and provide them as part of the policy observation, so a single policy can learn to operate across a range of compute budgets. We train the policy using group-relative policy optimization (GRPO) with teacher-forced episodes: for each input we sample multiple counterfactual compute schedules that share the same token path, and optimize a reward that balances language-model quality against soft penalties that encourage the episode-average budget usage to match the requested targets. Our contributions are: • We introduce learnable per-token budget allocation for LLM decoding: a lightweight policy allocates compute at each step while keeping the base LLM frozen. • Our policy acts on unified action space that controls multiple efficiency mechanisms: token-level attention sparsity, structured MLP activation pruning, and activation quantization. SOL tracks a better quality-efficiency pareto-front when compared to fixed budget allocation across all our experiments. • We present a practical training recipe based on group-relative policy optimization with teacher-forced counterfactual trajectories, conditioning the policy on requested budget targets and using soft budget-matching penalties to achieve controllable efficiency and model quality trade-offs. 2 Related Work and Motivation Prior research on methods to optimize LLM inference focus on (i) reducing the memory and computational footprint of the LLM, by quantizing the weights, activations ( 20 ; 19 ; 11 ) , or pruning linear projections and feed-forward network (FFN) layers ( 26 ; 1 ; 14 ) , (ii) context compression by sparsifying the attention itself, using importance-based techniques to retain only important tokens in the KV-Cache ( 42 ; 23 ; 36 ; 25 ; 35 ) , or exploiting the low-rank nature of the KV-Cache to reduce its memory footprint ( 5 ; 6 ) . (iii) reducing memory bandwidth by employing query-aware sparsity ( 2 ; 33 ; 34 ) . These methods typically apply a uniform budget to every token, meaning, the compression level is the same for every generated token. A second line of work focuses on how much compute to spend per token, but skipping layers of computation. These “early-exit” methods dynamically decide how many layers to execute for each token, stopping computation once a confidence criterion is met; this predates transformers and has been applied to convolutional networks as well. In sequence-to-sequence and autoregressive LMs, this “depth-adaptive” execution is orthogonal to our focus: we keep the depth fixed and instead adapt how much context, precision, and sparsity to use for each generated token. ( 30 ; 12 ; 13 ) introduce a token-wise early exit strategies. ( 32 ) demonstrates that later layers actually demonstrate a higher attention span than earlier layers, and over long-decode tasks, early-exit of several tokens can harm the global-attention pattern as tokens that exit early do not explicitly commit to KV-Cache of subsequent layers. Beyond early exit of tokens, recent research has also proven that splitting token generation across models ( 3 ; 39 ; 15 ; 22 ) by learning to offload difficult tokens or spans to larger models can significantly improving reasoning performance. Our contribution is

Read the complete text