VORT: Adaptive Power-Law Memory for NLP Transformers

Nabil Mlaiki · 2026

Read VORT: Adaptive Power-Law Memory for NLP Transformers by Nabil Mlaiki (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08966.txt.

Introduction

Nabil Mlaiki gives word models a memory that fades the way speech does. Stock focus lets far words fade at an exp rate. Real speech holds long links that fade by power laws. The new build gives each word a learned part order. That order sets a power law keep curve. The curve is near matched by a sum of exp parts. Each part steps in one plain loop step, so cost stays in hand.

Four proof claims back the build. They span the near match, a code size bound, an energy case on lost reach, and the pace of a shape rule. Two made tests back the edge. One is a skewed fetch task. One is a tag copy task with flat lags.

Catalog record

CREATORNabil Mlaiki
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08966

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VORT: Adaptive Power-Law Memory for NLP Transformers Nabil Mlaiki Abstract Standard Transformers impose near-exponential decay on the influence of distant tokens, conflicting with the power-law structure of long-range dependencies in natural language. We introduce the Variable-Order Retention Transformer ( vort ), a memory architecture in which each ingested token is assigned a learnable fractional order α i ∈ [ δ , 1 ] \alpha_{i}\in[\delta,1] that governs a Grünwald–Letnikov power-law retention kernel. Because the fractional weighted sum is non-Markovian, we approximate it through a sum-of-exponentials (SOE) decomposition computed by Gauss–Laguerre quadrature on a Laplace-type integral representation of the kernel weights. Each exponential component admits a one-step Markovian recurrence at O ⁡ ( S ​ d v ) O(Sd_{v}) per step, where S = O ⁡ ( log ⁡ ( T / ε ) ) S=O(\log(T/\varepsilon)) terms suffice for ε \varepsilon -uniform accuracy on horizon [ 1 , T ] [1,T] . Retrieval is keyed and associative via a linear-attention accumulator with an exact O ⁡ ( K ​ S ​ d ϕ ​ d v ) O(KSd_{\phi}d_{v}) -per-step recurrence. Four results are established: (i) an SOE approximation theorem with geometric convergence rate from the analyticity of the integrand after a log-change of variables; (ii) a quantisation bound valid on [ δ , 1 ] [\delta,1] with correct analysis near α = 0 \alpha=0 ; (iii) a direct L 2 L^{2} energy argument (Proposition) showing that for α > 1 / 2 \alpha>1/2 any mixture with fixed minimum decay rate Λ > 0 \Lambda>0 incurs L 2 ​ ( [ 1 , T ] ) L^{2}([1,T]) error at least N α ​ ( T ) − C ⁡ ( Λ ) → ∞ N_{\alpha}(T)-C(\Lambda)\to\infty , with the Λ \Lambda -dependence made explicit; and (iv) linear convergence of a gradient plasticity rule under the Polyak–Łojasiewicz condition. Two synthetic experiments confirm the architectural advantage: a Zipf-distributed retrieval benchmark and an entity label-copy task with uniform lag distribution, the latter ruling out prior-matching as an explanation for the power-law kernel’s advantage. Keywords: Transformers; long-range dependence; fractional calculus; Grünwald–Letnikov; sum-of-exponentials; power-law memory; linear attention; sequence modeling. 1 Introduction The Transformer [ 43 ] underlies modern large language models [ 6 , 42 , 9 , 2 ] , vision systems [ 35 ] , and scientific applications [ 21 ] . Its core operation is \mathrm{Attn}(Q,K,V)=\operatorname{softmax}\!\left(\tfrac{QK^{\top}}{\sqrt{d_{k}}}\right)V,\qquad Q,K\in\mathbb{R}^{n\times d_{k}},\quad V\in\mathbb{R}^{n\times d_{v}}, (1) whose quadratic cost in sequence length n has driven a large body of efficient approximations including sparse attention [ 7 ] , local-window methods [ 3 , 49 ] , low-rank factorisation [ 44 ] , and kernel-feature approximations [ 8 , 22 ] . A statistical mismatch underlies these computational challenges. Mutual information between tokens at lag \ell decays as a power law I(\ell)\sim c\,\ell^{-\gamma} with \gamma\in(0,1) [ 1 , 10 ] —a hallmark of long-range dependence [ 4 , 14 , 19 ] . For ARFIMA (0,d,0) processes, whose spectral density satisfies S(\lambda)\sim c_{0}|\lambda|^{-2d} near \lambda=0 , the optimal linear predictor assigns weights \pi_{j}\sim j^{d-1}/\Gamma(d) to lag- j observations [ 5 , 47 ] : power-law weights. Zipf’s law [ 51 ] and algebraic decay of character-level entropy production [ 12 ] reinforce this picture. Positional biases such as ALiBi [ 34 ] , RoPE [ 41 ] , and relative encodings [ 39 , 37 ] modulate attention scores but do not change the underlying memory update rule, leaving the mismatch intact. Current long-context methods partition memory into discrete tiers: eviction caches (H2O [ 50 ] , SnapKV [ 25 ] ) make binary keep-or-evict decisions; state-space models (S4 [ 16 ] , Mamba [ 15 ] , RWKV [ 32 ] ) use fixed or mildly input-dependent exponential recurrences; quantised caches [ 18 ] reduce precision uniformly; compressive memory [ 36 ] applies a fixed schedule. None represents a per-token, content-adaptive power-law decay. Fractional calculus offers the missing ingredient. The Grünwald–Letnikov (GL) weighted sum of order \alpha\in(0,1) assigns weight w_{j}^{(\alpha)}\sim j^{\alpha-1}/\Gamma(\alpha) to the value at lag j , matching the ARFIMA-optimal prediction weights. The obstacle is that this sum is non-Markovian : it cannot be reduced to a fixed-dimensional recurrence. The sum-of-exponentials (SOE) method, developed for numerical fractional differential equations by [ 27 ] and [ 17 ] , resolves this by decomposing the kernel into decaying exponentials, each updating via an exact one-step Markovian recurrence, with provably small error. We introduce Variable-Order Retention Transformer ( vort ) , grounded in the observation that tokens participating in long-range dependencies should receive high fractional orders so their retention kernels align with the power-law decay of those dependencies, while local connectives receive low orders for efficient compression. Our contributions are: (i) An architecturally coherent mechanism: per-token fractional states via SOE auxiliary recurrences, routed to K fixed-order banks, with keyed linear-attention retrieval admitting an exact O(KSd_{\phi}d_{v}) -per-step recurrence (Sections 2 – 3 ; Figures 1 – 2 ). (ii) Theorem 3.1 : SOE approximation with geometric convergence from analyticity of the Laplace-transformed integrand. (iii) Theorem 4.1 : quantisation bound on [\delta,1] with correct near- \alpha=0 behaviour. (iv) Proposition 4.2 : a direct L^{2} energy argument showing that for \alpha>1/2 any mixture with minimum decay rate \Lambda>0 incurs L^{2}([1,T]) error at least N_{\alpha}(T)-C(\Lambda)\to\infty as T\to\infty , where C(\Lambda) is a constant depending on \Lambda but not on T ; the \Lambda -dependence is stated explicitly as a limitation. (v) Theorem 5.1 : linear convergence of a plasticity update under the Polyak–Łojasiewicz (PL) condition. (vi) A synthetic pilot experiment on Zipf-distributed dependency retrieval (Section 6 ). 2 Background Throughout, d\in\mathbb{N} is embedding dimension, n is sequence length, \Gamma the Euler gamma function, B the beta function, and \psi=\Gamma^{\prime}/\Gamma the digamma function. Grünwald–Letnikov fractional sum. For \alpha\in(0,1) the GL weighted sum of a sequence \{f_{t}\} is \bigl(I^{\alpha}_{\mathrm{GL}}f\bigr)(t):=\sum_{j=0}^{t-1}w_{j}^{(\alpha)}\,f_{t-j},\qquad w_{j}^{(\alpha)}:=\frac{\Gamma(j+\alpha)}{\Gamma(\alpha)\,\Gamma(j+1)}, (2) with w_{j}^{(\alpha)}>0 for all j\geq 0 and \alpha>0 . By Stirling’s formula, w_{j}^{(\alpha)}\sim\frac{j^{\alpha-1}}{\Gamma(\alpha)}\quad\text{as }j\to\infty, (3) so the kernel is heavy-tailed: \sum_{j\geq 0}w_{j}^{(\alpha)}=\infty for all \alpha>0 (see Figure 4 a for log-log plots confirming the algebraic slope \alpha-1 ). The z -transform identity W_{\alpha}(z)=\sum_{j=0}^{\infty}w_{j}^{(\alpha)}\,z^{j}=(1-z)^{-\alpha},\quad|z|<1, (4) follows from the generalised binomial series [ 30 , 33 ] . Non-Markovian structure. The GL sum ( 2 ) has no exact two-term recurrence S_{t}=c\,S_{t-1}+b\,f_{t} with fixed scalars c,b : any such recurrence has z -transform b/(1-cz) , a rational function, while (1-z)^{-\alpha} is irrational for \alpha\notin\mathbb{Z} . More generally, no rational function of any finite degree equals (1-z)^{-\alpha} identically, since the latter has a branch point at z=1 and infinitely many non-zero Taylor coefficients. Long-memory processes. A stationary process \{X_{t}\} has long memory with parameter d\in(0,\tfrac{1}{2}) if S(\lambda)\sim c_{0}|\lambda|^{-2d} near \lambda=0 [ 4 ] . The ARFIMA (0,d,0) model [ 14 , 19 ] generates such processes via (1-B)^{d}X_{t}=\varepsilon_{t} . Its optimal one-step predictor has coefficients \pi_{j}\sim j^{d-1}/\Gamma(d) [ 5 ] , establishing that statistically optimal prediction of long-memory sequences requires power-law weights. 3 The VORT Mechanism Sum-of-Exponentials Approximation The GL weight admits the Laplace-type integral representation w_{j}^{(\alpha)}=\int_{0}^{\infty}e^{-\lambda j}\,\rho_{\alpha}(\lambda)\,d\lambda,\qquad\rho_{\alpha}(\lambda):=\frac{e^{-\alpha\lambda}(1-e^{-\lambda})^{-\alpha}e^{-\lambda}}{\Gamma(\alpha)\Gamma(1-\alpha)}, (5) obtained from the beta-function representation w_{j}^{(\alpha)}=B(j,1-\alpha)/B(\alpha,1-\alpha) via the substitution \lambda=-\log\mu . The density \rho_{\alpha} satisfies \rho_{\alpha}\geq 0 , \rho_{\alpha}(\lambda)\sim\lambda^{-\alpha}/(\Gamma(\alpha)\Gamma(1-\alpha)) as \lambda\to 0^{+} , and \rho_{\alpha}(\lambda)\leq Ce^{-\lambda} for large \lambda . An S -point SOE approximation is \hat{w}_{j}^{(\alpha)}=\sum_{s=1}^{S}c_{s}^{(\alpha)}\,\bigl(\lambda_{s}^{(\alpha)}\bigr)^{j},\qquad c_{s}^{(\alpha)}>0,\quad\lambda_{s}^{(\alpha)}\in(0,1), (6) where \lambda_{s}=e^{-\xi_{s}} ,

Read the complete text