Efficient Reasoning at Fixed Test-Time Cost via Length Aware Attention Priors and Gain Aware Training

Rian Atri · 2026

Read Efficient Reasoning at Fixed Test-Time Cost via Length Aware Attention Priors and Gain Aware Training by Rian Atri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09253.txt.

Introduction

Rian Atri studies how small and medium Transformers can reason well without raising test-time cost. Two training-only parts do the work. A length-aware attention prior, built by fuzzy regime position alignment, biases attention like a structured regularizer. A gain-aware controller sharpens attention only when validation gains warrant it. Both switch off at inference, leaving a cached bias as the sole overhead. Under matched compute on WikiText-2, validation loss falls with no real latency or memory cost.

Catalog record

CREATORRian Atri
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53544
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.09253

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Reasoning at Fixed Test-Time Cost via Length Aware Attention Priors and Gain Aware Training Rian Atri Abstract We study efficient reasoning under tight compute: how to make structured, correct decisions without increasing test-time cost. We add two training-time only components to small/medium Transformers that also transfer to broader differentiable optimizers. First, a length-aware attention prior built via fuzzy regime-position alignment (RPA) yields a normalized pre-softmax bias that guides attention like a structured regularizer while adding no new inference parameters. Second, a minimal gain-aware controller (Guardian) nudges attention sharpness only when validation improvements warrant it, following a two-timescale policy-gradient view of nonconvex optimization; it is disabled at inference. A KL perspective shows softmax ⁡ ( z + log ⁡ π ) \operatorname{softmax}(z+\log\pi) as MAP with KL regularization, grounding the prior in a principled objective. Under strict compute parity on WikiText-2, we reduce validation cross-entropy while matching baseline latency and memory. At inference, we add a precomputed, cached prior B ⁡ ( T ) B(T) as a single additive bias per head; the controller does not run. In practice, this incurs negligible overhead (a cached bias add per head; no measurable p50 latency shift). Our results suggest that length-aware priors and late-phase gain control preserve scarce improvements,especially in long-span, noisy-logit regimes,while keeping test-time costs effectively unchanged. \workshoptitle NeurIPS 2025 Workshop on Efficient Reasoning 1 Introduction Reasoning systems are ultimately optimizers : they allocate probability mass, select actions, and propagate constraints under memory and time budgets. At small/medium scale, training often plateaus late: as the learning rate decays and averages dominate, short bursts of genuine progress get washed out. In parallel, inductive bias on where to attend or route is frequently either rigid (fixed sinusoids) or ad-hoc (relative/rotary heuristics), which can misalign with structure the model is in fact discovering. We frame efficient reasoning over general optimization as preserving scarce, high-value improvements without increasing test-time cost. Concretely, we couple two levers: 1. Regime–Position Alignment (RPA): fuzzy token-to-regime memberships aligned to a length-aware positional basis via Sinkhorn, yielding a data-driven, zero-parameter pre-softmax attention prior. 2. Gain-aware control (Guardian): a tiny controller that adjusts a bounded attention temperature only when validation gains justify it; otherwise it relaxes. The controller is training-only and disabled at inference. 3. Tail-optimized schedules: nonzero LR floors and selective SWA that preserve late-phase gains under fixed compute. Our perspective is modular and optimization-centric. The RPA prior acts as a structured regularizer on attention allocations, derived from a KL-regularized MAP view; the controller is a projected, slow-timescale policy-gradient update to a scalar hyperparameter. Together they protect marginal improvements in regimes where content logits are noisy and reasoning requires long-span links, all while keeping inference latency and memory unchanged. Contributions (i) A principled KL view connecting pre-softmax priors to MAP with KL regularization, explaining when and why a prior steers attention. (ii) A concrete, length-aware RPA construction from fuzzy memberships and soft positional blocks aligned by entropic transport. (iii) A minimal, gain-aware controller for late-phase optimization, disabled at inference. (iv) Compute-parity experiments on WT2 and diagnostics for long-context linkage, plus full code listings for reproduction. Scope Our goal is efficient reasoning: stronger structure and stability for the same test-time budget. We do not claim SOTA or zero runtime overhead; the prior adds a fixed bias at test time and the controller does not run. 2 Attention with a Prior is KL-Regularized MAP Let z\in\mathbb{R}^{L} be pre-softmax content logits for a query row and \pi\in\Delta^{L-1} a strictly positive prior over keys. Theorem 1 (KL-regularized MAP) . Define a^{\pi}(z)=\mathrm{softmax}(z+\log\pi) . Then a^{\pi}(z)=\arg\max_{a\in\Delta^{L-1}}\;a^{\top}z\ -\ \mathrm{KL}(a\|\pi), with a unique maximizer. Sketch. Lagrangian L(a,\lambda)=a^{\top}z-\sum_{j}a_{j}\log\tfrac{a_{j}}{\pi_{j}}+\lambda(1-\sum_{j}a_{j}) . KKT \Rightarrow\log\tfrac{a_{j}}{\pi_{j}}=z_{j}-\lambda\Rightarrow a_{j}\propto\pi_{j}e^{z_{j}} . Strict concavity gives uniqueness. Implications (1) The prior acts as a directional regularizer on row-wise attention distributions, biasing entropy toward \mathrm{H}(\pi) . (2) Standardizing and clipping the prior calibrates its effective temperature against QK^{\top}/\sqrt{d} without breaking softmax’s row-shift invariance. 3 Method: Fuzzy Regime Prior for Attention (RPA) We first induce graded memberships \mu_{t}\in\Delta^{R-1} over a small set of “regimes” using Gaussian memberships (stable, interpretable, entropy-exposable). We then align these regimes to a length-aware basis. Fuzzy regimes (intuition) Instead of forcing each token to pick a single expert or locality bucket, we infer a soft membership vector \mu_{t}\!\in\!\Delta^{R} that encodes which coarse “regimes” best explain the current representation. Gaussian memberships are a convenient parameterization: they act like learnable centroids with scale, are stable under end-to-end training, and expose an entropy signal H(\mu_{t}) we can regularize to avoid collapse. In practice, R\!\in\![3,8] already yields interpretable partitions (e.g., near/local vs. far/global patterns). This connects to mixture-of-experts routing while avoiding brittle hard top- k assignments [ 16 , 7 ] . Fuzzy regimes Each token’s hidden state h_{t} produces \mu_{t}(r)\ \propto\ \exp\left(-\tfrac{1}{2\sigma_{r}^{2}}\|Wh_{t}-c_{r}\|_{2}^{2}\right),\qquad r\in\{1,\dots,R\}, with centers c_{r} and scales \sigma_{r} learned end-to-end. We regularize the membership entropy H(\mu_{t}) to avoid early collapse. Length-aware positional basis We form soft raised-cosine blocks \Phi(T)\in\mathbb{R}^{T\times K} that tile \{1,\dots,T\} with row sums 1 . This provides a vocabulary to express where regimes tend to live (prefix, middle, suffix, long-span bands), and adapts smoothly as T varies. Entropic alignment and prior Let S=\tfrac{1}{B}\sum_{b}\mu_{b}^{\top}\Phi(T)\in\mathbb{R}^{R\times K} . We compute P=\mathrm{Sinkhorn}(\exp(S/\tau_{\text{align}})) (approximately doubly-stochastic). The aligned prior is B(T)\;=\;\Big(\tfrac{1}{B}\sum_{b}\mu_{b}\,P\,\Phi(T)^{\top}\Big)\in\mathbb{R}^{T\times T}, standardized and lightly clipped, then added to attention logits. We warm-in the bias over early steps. Motivation B(T) captures second-order co-assignment across positions: indices that tend to share regimes receive a positive prior, stabilizing heads when QK^{\top} is noisy (low data, small models). Because the basis is length-aware, the prior remains calibrated under varying context lengths. Core code (Gaussian memberships) ⬇ def gaussian_membership ( h , centers , log_sigma ): # h:[B,T,D], centers:[R,D], log_sigma:[R] z2 = ( h . unsqueeze (2) - centers . view (1,1,* centers . shape )). pow (2). sum (-1) # [B,T,R] inv2sig2 = torch . exp (-2* log_sigma ). view (1,1,-1). clamp (1 e -3,1 e3 ) logits = torch . clamp (-0.5 * z2 * inv2sig2 , -30.0, 30.0) mu = F . softmax ( logits , dim =-1) return torch . nan_to_num ( mu , nan =1.0/ mu . size (-1)) Listing 1: Compute fuzzy memberships \mu via squared distances and softmax. Core code (RPA bias) ⬇ def rpa_bias ( mu , K , tau =0.7, iters =6): B , T , R = mu . shape t = torch . arange ( T , device = mu . device ). float () c = torch . linspace (0, T -1, K , device = mu . device ). float () w = max (1.0, ( T / max (1, K ))*1.5) Phi = 0.5*(1+ torch . cos ( math . pi * torch . clamp (( t [:, None ]- c [ None ,:]). abs ()/ w ,0,1))) Phi = Phi /( Phi . sum (1, True )+1 e -6) S = torch . einsum ( 'btr,tk->rk' , mu , Phi )/ B X = torch . exp ( S / tau ) for _ in range ( iters ): X = X /( X . sum (1, True )+1 e -9); X = X /( X . sum (0, True )+1 e -9) M = torch . einsum ( 'btr,rk->btk' , mu , X ) # [B,T,K] Bmat = torch . einsum (

Read the complete text