Adaptive Computation Depth via Learned Token Routing in Transformers

Ahmed Abdelmuniem Abdalla Mohammed · 2026

Read Adaptive Computation Depth via Learned Token Routing in Transformers by Ahmed Abdelmuniem Abdalla Mohammed (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05222.txt.

Introduction

Ahmed Abdelmuniem Abdalla Mohammed presents Token-Selective Attention, a way for transformers to spend fewer layers on easy tokens and more on hard ones. Standard transformers run every token through every layer regardless of difficulty. His mechanism adds a small learned gate between blocks that outputs a continuous halting probability for each token, trained end to end with modest extra parameters.

Remarkably, the gates learn difficulty-based routing from the task loss alone, with no explicit pressure to save depth. On character-level language modelling the method saved a fifth or so of token-layer operations at tiny quality cost. At matched efficiency it beat early-exit baselines, and the learned routing transfers to faster sparse execution at inference time.

Catalog record

CREATORAhmed Abdelmuniem Abdalla Mohammed
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05222

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptive Computation Depth via Learned Token Routing in Transformers Ahmed Abdelmuniem Abdalla Mohammed Abstract Standard transformer architectures apply the same number of layers to every token regardless of contextual difficulty. We present Token-Selective Attention (TSA) , a learned per-token gate on residual updates between consecutive transformer blocks. Each gate is a lightweight two-layer multi-layer perceptron (MLP) that produces a continuous halting probability, making the mechanism end-to-end differentiable with 1.7% parameter overhead and no changes to the base architecture. Notably, TSA learns difficulty-proportional routing without any explicit depth pressure: even at λ = 0 \lambda{=}0 (no depth regularisation), the task-loss gradient alone drives the router to skip 20% of token-layer operations. On character-level language modeling, TSA saved 14–23% of token-layer operations (TLOps) across Tiny-Shakespeare and enwik8 at < < 0.5% quality loss. At matched efficiency, TSA achieved 0.7% lower validation loss than early exit, and the learned routing transfers directly to inference-time sparse execution for real wall-clock speedup. Keywords: adaptive computation, token routing, sparse transformers, efficient inference, depth regularisation 1 Introduction Transformer language models ( 15 ) apply a fixed number of layers to every token in every sequence. This design trades per-token adaptability for architectural simplicity. In practice, the trade-off is costly: a common token in a predictable context requires far less processing than a rare token in a novel construction, yet both receive identical compute at every layer of every forward pass. The inefficiency is particularly consequential at inference scale. For large deployed models, the dominant cost is the forward pass through all layers for all tokens. If a significant fraction of tokens could exit early without quality loss, the savings would translate directly to reduced latency and throughput gains. Several approaches have addressed this problem. 9 introduced Adaptive Computation Time (ACT) for recurrent neural networks (RNNs), accumulating a halting probability across recurrent steps. 6 extended the idea to depth-shared transformer layers with the Universal Transformer. More recently, 14 proposed Mixture-of-Depths (MoD), which routes tokens through a fixed subset of layers using hard top- k selection; 3 introduced Mixture of Recursions, which applies recursive blocks for a learned number of steps per token; and 5 presented the Inner Thinking Transformer, which inserts additional computation steps at high-stakes positions. We present Token-Selective Attention (TSA) : a continuous soft gate on residual updates, conditioned per token on its current hidden state. The mechanism is architecturally minimal—a two-layer MLP per inter-block gap—and fully differentiable, requiring no straight-through estimators, Gumbel sampling, or reinforcement learning. Our contributions are: • A simple, differentiable token routing mechanism that gates residual updates softly per token per layer (§ 2 ). • Evidence that routing emerges from the task-loss gradient alone: at \lambda{=}0 (no depth regularisation), the router learns to skip 20% of token-layer operations without any explicit depth pressure (§ 3.4 ). • Cross-dataset validation on character-level language modeling: 14–23% token-layer operations saved across Tiny-Shakespeare and enwik8 at < 0.5% quality loss (§ 3.2 , § 3.3 ). • Ablations showing robustness to \lambda across two orders of magnitude (§ 3.4 ), quality advantage over early exit at matched efficiency (§ 3.5 ), and real wall-clock speedup via sparse inference on commodity hardware (§ 3.6 ). 2 Method 2.1 Architecture Let a pre-norm decoder-only transformer have blocks f_{0},f_{1},\ldots,f_{L-1} , where each block applies multi-head self-attention and a feed-forward network (FFN) with residual connections and LayerNorm ( 2 ) . In TSA, a lightweight router r_{l} is inserted after each block f_{l} for l=0,\ldots,L-2 . Block f_{0} is the stem and always executes unconditionally: a bare token embedding carries no contextual signal, making a routing decision at step zero uninformative and potentially degenerate. The routing begins after the stem: h\leftarrow f_{0}(h),\quad p_{l}=r_{l}(h),\quad h\leftarrow f_{l+1}(h,\,p_{l}),\quad l=0,\ldots,L-2. (1) Figure 1 illustrates the dual-mode mechanism: soft gating during training (differentiable) and hard-threshold sparse execution at inference (real FLOPs savings). Figure 1 : TSA dual-mode architecture. A router r_{l} reads hidden state h and produces a per-token halting probability p_{l} . Left (training): all tokens always pass through attn + FFN; the residual update is soft-scaled by (1{-}p_{l}) , keeping the gate differentiable so the router learns. Right (inference): attention remains dense, but tokens with p_{l}>0.5 skip the FFN entirely via gather/scatter, yielding real FLOPs savings. The stem block f_{0} always executes unconditionally. 2.2 Router Architecture Each router is a two-layer MLP with sigmoid output: r_{l}(h)=\sigma\!\bigl(W_{l}^{(2)}\,\mathrm{ReLU}(W_{l}^{(1)}h+b_{l}^{(1)})+b_{l}^{(2)}\bigr),\quad r_{l}(h)\in(0,1)^{B\times T}, (2) where the hidden dimension is d/4 (floored at 16). Each router adds d^{2}/4+d/2+1 parameters; at d=256 , L=6 , this totals \approx 83K on a 4.78M parameter base model (1.7% overhead). The final bias b_{l}^{(2)} is initialised to -1.0 , giving \sigma(-1)\approx 0.27 at initialisation. This bias prevents early collapse to “halt everything” before the model has learned useful representations. 2.3 Gated Block Update For each routing decision l=0,\ldots,L{-}2 , the gated update of block f_{l+1} is: \displaystyle h \displaystyle\leftarrow h+(1-p_{l})\odot\Delta_{l+1}^{\mathrm{attn}}(h), (3) \displaystyle h \displaystyle\leftarrow h+(1-p_{l})\odot\Delta_{l+1}^{\mathrm{ffn}}(h), (4) where p_{l}\in(0,1)^{B\times T} is broadcast over the model dimension d , and \Delta_{l+1}^{\mathrm{attn}} , \Delta_{l+1}^{\mathrm{ffn}} are the pre-norm attention and feed-forward residual deltas of block f_{l+1} respectively. When p_{l}=0 , the update is identical to the standard transformer. When p_{l}=1 , the state is unchanged—the block is skipped. The interpolation is smooth, preserving gradient flow through p_{l} during training. 2.4 Depth Regularisation Without any incentive to halt, routers default to p_{l}\approx 0 and TSA reduces to a standard transformer with extra parameters. We added a depth regularisation term that gently encourages early halting: \mathcal{L}_{\mathrm{depth}}=\lambda\cdot\frac{1}{L-1}\sum_{l=0}^{L-2}\overline{1-p_{l}}, (5) where \overline{1-p_{l}} is the mean active fraction at layer l (averaged over batch and sequence position). The total training loss is \mathcal{L}=\mathcal{L}_{\mathrm{task}}+\mathcal{L}_{\mathrm{depth}} . We used \lambda=0.001 for language experiments; Section 3.4 demonstrates that TSA is robust across \lambda\in[0,\,0.1] . 2.5 Compute Metric We measure compute using token-layer operations (TLOps) : for each block, TLOps equals the number of tokens processed at that block. The mean active fraction across routing decisions is: \alpha=\frac{1}{L-1}\sum_{l=0}^{L-2}\overline{1-p_{l}}. (6) TLOps savings relative to the fixed-depth baseline are: \Delta=1-\frac{1+(L-1)\,\alpha}{L}. (7) The stem block (always active) is included in both numerator and denominator, making \Delta a conservative estimate. Note on training compute. During training, all layers execute fully: the gate scales residual updates but does not skip computation. TLOps therefore measures the effective contribution of each layer to the final representation, not actual FLOPs saved. At inference, sparse-TSA (Section 3.6 ) exploits low-contribution positions via gather/scatter to achieve real compute savings. 3 Experiments 3.1 Synthetic Algorithmic Tasks Setup. We used decoder-only transformers trained on copy and sort tasks over length-10 sequences from a 32-token vocabulary. Inputs followed the format [BOS] src [SEP] tgt [EOS] with loss masked on source tokens. Both baseline and TSA used d{=}128 , L{=}6 , H{=}4 , d_{\mathrm{ff}}{=}512 (baseline: 1.20M params; TSA: 1.22M params, + 1.7%). Training employed AdamW ( 12 ) with \beta{=}(0.9,0.95) , \mathrm{lr}{=}3{\times}10^{-4} , \lambda_{\mathrm{wd}}{=}0.1 for 10K gradient steps on 10K training sequences. We report token-level sequence accuracy on 1K held-out sequences. Results. Table 1 : Synthetic Task Results ( d{=}128 , L{=}6 , Toy Vocabulary) Task Model Accuracy Perplexity Active Frac \alpha TLOps saved † Copy Baseline 1.0000 1.012 1.000 — Copy TSA 1.0000 1.012 0.341 54.9% Sort Baseline 0.9915 1.079 1.000 — Sort TSA 0.9878 1.087 0.730 22.5% † TLOps

Read the complete text