EMA Is Not All You Need: Mapping the Boundary Between Structure and Content in Recurrent Context
Arth Singh · 2026
Read EMA Is Not All You Need: Mapping the Boundary Between Structure and Content in Recurrent Context by Arth Singh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08556.txt.
Introduction
Singh probes what simple exponential moving average traces can and cannot encode in sequence models, using them as the plainest recurrent context. A Hebbian design with multi timescale traces reaches 96 percent of a supervised BiGRU on grammatical role assignment with no labels, beating it on structure dependent roles. Yet traces erase token identity: a 130 million parameter model using only EMA context reaches perplexity 260, eight times GPT-2's, a gap no downstream predictor can undo.
Catalog record
| CREATOR | Arth Singh |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39270 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.08556
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EMA Is Not All You Need: Mapping the Boundary Between Structure and Content in Recurrent Context Arth Singh Abstract What exactly do efficient sequence models gain over simple temporal averaging? We use exponential moving average (EMA) traces, the simplest recurrent context (no gating, no content-based retrieval), as a controlled probe to map the boundary between what fixed-coefficient accumulation can and cannot represent. EMA traces encode temporal structure: a Hebbian architecture with multi-timescale traces achieves 96% of a supervised BiGRU on grammatical role assignment with zero labels, surpassing the supervised model on structure-dependent roles. EMA traces destroy token identity: a 130M-parameter language model using only EMA context reaches C4 perplexity 260 ( 8 × 8\times GPT-2), and a predictor ablation (replacing the linear predictor with full softmax attention) yields identical loss, localizing the entire gap to the traces. The traces apply lossy, data-independent compression; by the data processing inequality, no downstream predictor can recover the discarded information. Fixed-coefficient accumulation, whether across time or depth, suffers irreversible information dilution that only learned, input-dependent selection can resolve. 1 Introduction \mathbf{x}_{t} EMA Traces ( \alpha{=}.5,.1,.02 ) Predict \hat{\mathbf{x}}_{t} from \mathbf{h}^{(s)}_{t} Error \mathbf{e}_{t}=\mathbf{x}_{t}{-}\hat{\mathbf{x}}_{t} Context Mix Sparse FFN (Top- k , 6%) \mathbf{x}^{\prime}_{t} + \times 12 blocks bottleneck Figure 1: One SPEN block. All temporal context enters through fixed-decay EMA traces (red box). No attention is used. The predictor ablation (§ 7 ) confirms the traces, not any downstream component, limit performance. Efficient sequence models, including state-space models ( 5 ) , linear attention ( 8 ) , and gated recurrences ( 15 ; 4 ) , replace the full attention matrix with compressed recurrent states, trading expressiveness for efficiency. Each mechanism makes different tradeoffs: Mamba adds input-dependent transitions, RWKV adds time-decay gating, linear attention adds data-dependent outer products. What do all of these mechanisms buy over the simplest possible baseline? We use exponential moving average (EMA) traces as a controlled probe to answer that question. An EMA trace computes \mathbf{h}_{t}=(1{-}\alpha)\mathbf{h}_{t-1}+\alpha\mathbf{x}_{t} : a weighted sum of past inputs with fixed exponential decay. EMA traces have no gating, no content-based retrieval, and no learned state transitions. EMA occupies the bottom of the expressiveness hierarchy. EMA traces also rank among the most biologically plausible context mechanisms: cortical neurons integrate synaptic input with exponential decay at multiple timescales ( 13 ) . By building systems that use only EMA traces for context and measuring exactly where they succeed and fail, we establish a precise lower bound that clarifies what the richer mechanisms are buying. We conduct the probe at two scales. At small scale (§ 3 – 4 ), we build Sparse Predictive Column Networks (SPCN), a Hebbian architecture with frozen random projections, sparse top- k activation, and multi-timescale EMA traces. On a 20-role grammatical role assignment task, probing the traces instead of instantaneous activations raises within-grammar accuracy from 0.80 to 0.96, reaching 96% of a supervised BiGRU with zero labels. On structural roles where temporal patterns are vocabulary-independent, SPCN surpasses the supervised model. EMA traces encode temporal structure with high fidelity. At large scale (§ 5 – 7 ), we build SPEN, a 130M-parameter language model that replaces attention entirely with three EMA traces and a sparse feedforward network. SPEN establishes the lower bound at C4 perplexity 260, an 8\times gap to GPT-2 small (33). A predictor ablation localizes the source of the gap: replacing the linear predictor with full causal softmax attention yields identical cross-entropy loss. The traces destroy fine-grained token identity through data-independent averaging before any predictor can query the trace output. EMA traces fail at content retrieval. The two scales together characterize a sharp representational boundary. EMA traces preserve temporal structure (the order and pattern of sparse activations), sufficient for grammatical roles. EMA traces lose token identity (which specific word appeared where), required for language modeling. Concurrent work on Attention Residuals ( 9 ) identifies the same failure in the depth dimension; we develop the connection in § 8 . Our contributions: 1. We establish EMA traces as a controlled lower bound for recurrent context mechanisms, characterizing the boundary between structure and content (§ 4 , § 7 ). 2. We show temporal traces serve as unsupervised structural representations, achieving 96% of supervised accuracy and surpassing supervision on structure-dependent roles (§ 4 ). 3. We train a 130M-parameter EMA-only language model that quantifies the cost of data-independent context: an 8\times perplexity gap localized entirely to the trace mechanism (§ 6 , § 7 ). 4. We connect the temporal and depth dimensions through a general principle about fixed-coefficient accumulation (§ 8 ). 2 Related Work Predictive coding and Hebbian learning. Predictive coding ( 16 ; 3 ) proposes that cortical circuits learn by predicting bottom-up input and propagating prediction errors upward. 10 surveyed biologically plausible alternatives to backpropagation, including Hebbian rules and target propagation. SPCN implements predictive coding with a precision-gated Hebbian update (PGHU), a three-factor learning rule in which precision (the inverse variance of prediction error) modulates the learning rate at each synapse. The precision gate is analogous to dopaminergic modulation of plasticity ( 19 ) : dimensions with high prediction error variance receive lower precision and therefore smaller weight updates, while dimensions with consistent, informative errors receive focused learning. Reservoir computing. Echo State Networks ( 6 ) and Liquid State Machines ( 12 ) use frozen random recurrent projections with trained linear readouts, achieving strong performance on temporal tasks despite having no learned dynamics. SPCN shares the frozen-projection principle (the feedforward weights \mathbf{W}_{\text{ff}} are never updated) but differs in three ways: sparse top- k activation instead of dense states, multi-timescale trace integration instead of a single recurrent timescale, and learned feedback weights via PGHU that modulate the settling dynamics. Efficient sequence models. S4 ( 5 ) parameterizes structured linear recurrences for long-range modeling. Mamba ( 4 ) extends S4 with input-dependent state transitions, closing most of the gap to transformers. RWKV ( 15 ) reformulates attention as a linear recurrence. MEGA ( 11 ) combines EMA with gated attention in a single layer. Linear attention ( 8 ) removes the softmax normalization for O(n) cost. Each of these mechanisms sits above EMA in the expressiveness hierarchy: linear attention adds data-dependent outer products, SSMs add structured state matrices, Mamba adds input-dependent transitions. SPEN’s EMA traces are a degenerate case: scalar, data-independent transitions that provide a controlled lower bound on what the richer mechanisms buy. Fast weights and online adaptation. 17 proved that linear attention is mathematically equivalent to fast weight programmers ( 18 ) : associative memories updated via outer-product rules at each timestep. SPEN includes a precision-gated Hebbian fast weight mechanism (PGHU) for inference-time adaptation. Our experiments show the mechanism faces the same stability–plasticity tradeoff as gradient-based fine-tuning: aggressive updates destabilize in-distribution predictions, while conservative updates provide negligible adaptation (Appendix D ). Depth-wise accumulation. 9 identify the same information dilution problem in the depth dimension and propose Attention Residuals as a fix. We develop the parallel between time and depth in § 8 . 3 SPCN: Architecture Sparse Predictive Column Networks (SPCN) stack cortical-column-inspired processing units into a four-level hierarchy. Each column at level \ell maintains a sparse state vector \mathbf{x}^{(\ell)}\in\mathbb{R}^{d_{\ell}} and three weight matrices: feedforward \mathbf{W}_{\text{ff}}^{(\ell)}\in\mathbb{R}^{d_{\ell}\times d_{\ell-1}} (fixed at initialization), feedback \mathbf{W}_{\text{fb}}^{(\ell)}\in\mathbb{R}^{d_{\ell}\times