Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling
Petr Nyoma · 2026
Read Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling by Petr Nyoma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24650.txt.
Introduction
Nyoma stacks state space models into a hierarchy that reads at three speeds. Language models need long context, and attention's cost grows brutally with length. Harmonic, presented in 2026, stacks three recurrent levels that run at progressively slower timescales. Each level receives the prediction error from the level below, rather than its raw hidden state, so higher levels only process what lower levels failed to predict.
On a standard character level benchmark with equal token budgets, Harmonic beats a comparable transformer at every context length, and beats Mamba at every length too. The gap widens dramatically as sequences lengthen. At the longest setting, both rivals run out of memory on a large accelerator, while Harmonic, with linear memory, trains successfully. Scaling studies show the transformer winning at short context and small scale, but losing as length grows. Replacing all attention in a billion parameter model removes its positional encoding limit entirely.
Catalog record
| CREATOR | Petr Nyoma |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24650
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling Petr Nyoma Abstract We present Harmonic , a hierarchical state space model (SSM) for language modeling. The architecture stacks three recurrent levels at progressively slower timescales; each level receives the prediction error of the level below as input, rather than its raw hidden state. On enwiki8 with equal token budgets, Harmonic outperforms a comparable Transformer (28M params) by +1.4% at 1K tokens, +6.7% at 8K tokens, and +11.4% at 32K tokens (bpt, lower is better). It also outperforms Mamba at every tested length by 0.7–1.8%. At 64K tokens, both Mamba and Transformer run out of memory on an 80GB H100; Harmonic trains successfully, reaching 6.169 bpt — a direct consequence of O ( L ) O(L) memory. Results replicate on WikiText-103 (H–TF gap + 1.7 % +1.7\% to + 7.2 % +7.2\% across 1K–32K), the standard benchmark used by Mamba and S4. At ∼ \sim 100M parameters and 1K tokens the Transformer wins (by 3.2%); at 8K tokens the same model favors Harmonic by 6.6%. At ∼ \sim 112M parameters the same pattern holds: TF wins at 1K ( − 1.5 % -1.5\% ) and loses at 8K ( − 7.0 % -7.0\% ). At 1B parameter scale, replacing all attention layers in TinyLlama 1.1B with HarmonicBlock (the SSM module from this work) eliminates the RoPE positional encoding limit: the resulting Hallamonic model maintains stable loss across sequence lengths 1K–8K on two independent clean benchmarks (Lambada and fineweb-edu held-out), while TinyLlama degrades catastrophically past its 2K-token RoPE limit (gap: + 9.4 +9.4 bpt at seq=8K on Lambada). Compute is O ( L ) O(L) per forward pass vs. O ( L 2 ) O(L^{2}) for attention. Logs: https://github.com/Omibranch/harmonic-logs . 1 Introduction Motivation. This project started with a musical observation. When producing a track, you hear “something is off” before you can say what. That perception works across timescales at once: the note, the phrase, the harmonic arc of the piece. Standard language models have no equivalent. A Transformer processes all token pairs with equal weight regardless of distance, and produces no internal signal when its output is temporally inconsistent. The question behind this work: does making a model explicitly aware of multiple timescales improve its language modeling, and by how much? The computational problem. Self-attention requires O(L^{2}) computation per layer. At L=32{,}768 , attention costs 1{,}024\times more per token than at L=1{,}024 . State space models (SSMs) compute in O(L) , but prior work has not consistently shown quality advantages over Transformers under fair equal-budget comparisons at long context. This paper. We show that a three-level SSM hierarchy with multi-timescale recurrence outperforms both Transformer and Mamba baselines at every tested sequence length from 1K to 32K (Figure 2 ). The H–TF gap grows from +1.4\% to +11.4\% with context; the H–Mamba gap is smaller (0.7–1.8%) but consistent. At 64K tokens, both baselines run out of memory on an H100 80GB; Harmonic trains successfully. The long-context quality advantage holds across 7M–100M parameters and replicates on WikiText-103 (H–TF gap: +1.7\% at 1K to +7.2\% at 32K). At short context (1K, \geq 100M params) the Transformer wins; we report these results without qualification. At 1B parameter scale (Section 5 ), we transplant the HarmonicBlock SSM module into TinyLlama 1.1B by replacing all 22 attention layers. The resulting model (Hallamonic) has no positional encoding limit and maintains stable loss at seq=1K, 4K, and 8K on independent benchmarks, while the unmodified TinyLlama degrades by +9 – 10 bpt past its 2K RoPE limit. 2 Related Work Linear and sub-quadratic sequence models. S4 ( Gu et al. 2022 ) introduced structured state space models for long-range sequence modeling. Mamba ( Gu and Dao 2023 ) added input-selective state transitions. RWKV ( Peng et al. 2023 ) recasts attention in RNN form. Griffin ( De et al. 2024 ) mixes gated linear recurrences with local attention. H3 ( Fu et al. 2023 ) combines SSMs with a small attention component for associative recall. These models achieve competitive perplexity but most published comparisons against Transformers use unequal token budgets or different tuning, making it hard to isolate architectural effects. Hierarchical recurrence. Clockwork RNNs ( Koutnik et al. 2014 ) and hierarchical multiscale RNNs ( Chung et al. 2017 ) showed that multiple temporal scales can be handled by separate recurrent modules. Harmonic applies this idea to SSMs with a structured timescale hierarchy and prediction-error inter-level signals. Predictive coding. In predictive coding ( Rao and Ballard 1999 ) each level predicts the activity of the level below; only errors propagate upward. Neural network applications include video prediction ( Lotter et al. 2017 ) and contrastive representation learning ( van den Oord et al. 2018 ) . Harmonic uses prediction-error signals for inter-level communication; the timescale hierarchy rather than the prediction errors accounts for the performance advantage (Section 4.5 ). Attention-free large language models. Recent work has explored replacing attention in pretrained LLMs. RWKV-7 achieves competitive performance by converting attention to linear recurrence post-training. Mamba-based models at 1B–7B scale have been trained from scratch but require significantly more compute budget than fine-tuning. We take a direct approach: warm-start from TinyLlama’s pretrained FFN and embedding weights, train only the new SSM layers for a small fraction of the original budget, and evaluate whether the resulting model removes the positional encoding constraint without retraining from scratch. Comparison protocol. We use a strict equal-budget protocol: same dataset, tokenizer, optimizer, schedule, gradient clipping, and total training tokens across all models and sequence lengths. This is the minimum condition for attributing performance differences to architecture. 3 Harmonic Architecture Figure 1 : Harmonic architecture. Three SSM levels operate at progressively slower timescales \tau_{1}\ll\tau_{2}\ll\tau_{3} . Each level receives the prediction error from the level below as part of its input. The outputs of all three levels are summed before the language model head. Harmonic consists of three stacked SSM levels with a shared embedding and a single linear head. Figure 1 shows the overall structure. Timescale hierarchy. Each level \ell\in\{1,2,3\} maintains a hidden state h_{\ell}\in\mathbb{R}^{d} updated by a learned recurrence: h_{\ell}(t)=A_{\ell}(t)\odot h_{\ell}(t-1)+b_{\ell}(t), where A_{\ell}(t)\in(0,1)^{d} is the data-dependent decay gate and b_{\ell}(t) is the input contribution. A_{\ell}(t) is computed from the current input token, making the decay rate input-selective rather than fixed — analogous to the selective mechanism in Mamba ( Gu and Dao 2023 ) , but constrained by initialization to a level-specific timescale range [\tau_{\ell}^{\min},\tau_{\ell}^{\max}] with \tau_{1}\ll\tau_{2}\ll\tau_{3} : level 1 decays fast (local context), level 3 decays slowly (long-range context). Input-selectivity allows the model to gate out irrelevant context without relying on hard-coded decay, which addresses the associative recall limitation of fixed-timescale recurrences. Prediction-error inter-level signals. Each level \ell produces a prediction of the level below via a learned linear map P_{\ell}:\mathbb{R}^{d}\to\mathbb{R}^{d} . The prediction error e_{\ell}=h_{\ell-1}-P_{\ell}(h_{\ell}) is normalized and passed to level \ell{+}1 as part of its input, instead of h_{\ell-1} directly. Higher levels therefore receive residual signals — what lower levels failed to predict — rather than raw activations. The projectors P_{\ell} are trained end-to-end through the top-level language modeling loss; no local reconstruction losses are used at intermediate levels. This design is structurally consistent with predictive coding ( Rao and