HRM-Text: Efficient Pretraining Beyond Scaling

Guan Wang, Changling Liu, Chenyu Wang, Cai Zhou, Yuhao Sun, Yifei Wu, Shuai Zhen, Luca Scimeca, Yasin Abbasi Yadkori, [1ex] Sapient Intelligence · 2026

Read HRM-Text: Efficient Pretraining Beyond Scaling by Guan Wang, Changling Liu, Chenyu Wang, Cai Zhou, Yuhao Sun, Yifei Wu, Shuai Zhen, Luca Scimeca, Yasin Abbasi Yadkori, [1ex] Sapient Intelligence (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20613.txt.

Introduction

Guan Wang and eight colleagues present HRM-Text, a bid to pretrain language models efficiently without internet-scale compute. It displaces standard transformers with a hierarchical recurrent model that splits slow strategic work from fast execution. MagicNorm and warmup credit assignment stabilize the deep recurrence, and training uses instruction-response pairs only. A one-billion-parameter model trained on 40 billion tokens for 1,500 dollars reached 60.7 percent on MMLU and 84.5 percent on GSM8K.

Catalog record

CREATORGuan Wang, Changling Liu, Chenyu Wang, Cai Zhou, Yuhao Sun, Yifei Wu, Shuai Zhen, Luca Scimeca, Yasin Abbasi Yadkori, [1ex] Sapient Intelligence
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53249
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20613

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HRM-Text: Efficient Pretraining Beyond Scaling Guan Wang, Changling Liu, Chenyu Wang, Cai Zhou, Yuhao Sun, Yifei Wu, Shuai Zhen, Luca Scimeca, Yasin Abbasi Yadkori, [1ex] Sapient Intelligence Abstract The current pretraining paradigm for large language models relies on massive compute and internet-scale raw text, creating a significant barrier to foundational research. In contrast, biological systems demonstrate highly sample-efficient learning through multi-timescale processing, such as the functional organization of the frontoparietal loop. Taking this as inspiration, we introduce HRM-Text, which replaces standard Transformers with a Hierarchical Recurrent Model (HRM) that decouples computation into slow-evolving strategic and fast-evolving execution layers. To stabilize this deep recurrence for language modeling, we introduce MagicNorm and warmup deep credit assignment. Furthermore, instead of standard raw-text pretraining, we train exclusively on instruction-response pairs using a task-completion objective and PrefixLM masking. Serving as an empirical existence proof of efficient pretraining, a 1B-parameter HRM-Text model trained from scratch on only 40 billion unique tokens and $1,500 budget achieves 60.7% on MMLU, 81.9% on ARC-C, 82.2% on DROP, 84.5% on GSM8K, and 56.2% on MATH. Despite utilizing roughly 100-900x fewer training tokens and 96-432x less estimated compute than standard baselines, HRM-Text performs competitively with 2–7B parameter open models. These results demonstrate that co-designing architectures and objectives can radically reduce the compute-to-performance ratio, making pretraining from scratch accessible to the broader research community. † † footnotetext: † Corresponding author. ∗ Equal Contribution. Contact: [email protected] . Code available at: github.com/sapientinc/HRM-Text Figure 1 : Pretraining efficiency. Trained from scratch in 1.9 days on 16 GPUs, HRM-Text 1B achieved performance competitive with substantially larger 2–7B foundation models while utilizing up to 432 \times less compute and 900 \times fewer training tokens. 1 Introduction The remarkable success of large language models (LLMs) is currently driven by a monolithic recipe: massive, multi-stage pipelines that begin with broad unsupervised pretraining over internet-scale raw text. While undeniably effective, this brute-force scaling paradigm is highly inefficient in data-limited regimes. Massive compute is spent predicting prompt-like or task-irrelevant text simply to build generalized representations 37 , 31 , 63 . Consequently, this extreme computational barrier has largely locked the broader research community out of foundational pretraining exploration. The prevailing assumption is that without immense compute clusters and trillions of tokens, investigating new architectures or training from scratch is futile. This brute-force data hunger stands in stark contrast to human intelligence, which can grasp governing rules and perform heuristic-guided search from only a few examples. In our previous work, we introduced the Hierarchical Recurrent Model (HRM), a dual-timescale architecture inspired by the functional organization of the biological frontoparietal loop 69 . By decoupling deliberation into a slow-evolving strategic layer and a fast-evolving execution layer, HRM provided a structural inductive bias that helped avoid local stagnation and successfully guided symbolic search on combinatorial tasks. However, scaling recurrent architectures to the open-ended complexities of language modeling introduces severe gradient-instability risks 6 , 13 , 34 , 78 . A structural prior alone is insufficient; achieving competitive open-domain performance requires a holistic codesign. In this paper, we demonstrate that architecture and training methods are profoundly important once again. We explore two major, synergistic directions to realize this sample-efficient engine: • Architectural Exploration: To achieve deep computation without a proportional explosion in parameter counts, we build upon HRM’s modular, multi-timescale recurrence. The fast L -module performs local iterative refinement, while the slow H -module maintains stable semantic context across cycles 69 . To make this deep recurrence mathematically viable for language, we introduce stabilization techniques like MagicNorm and warmup deep credit assignment, which bound forward activation variance while maintaining backward optimization stability 71 , 44 , 62 . • Objective Exploration: We challenge the dogma of autoregressive pretraining on raw text. Since models are primarily used for conditional generation at inference time, we pretrain HRM-Text directly from scratch on instruction-response pairs 70 , 55 , 46 . We optimize a task-completion objective, computing the negative log-likelihood loss exclusively over the response: -\log P(x_{a}\mid x_{q}) 61 , 53 , 55 . We pair this with a PrefixLM attention mask, which allows full bidirectional (encoder-like) attention across the instruction tokens while preserving standard causal generation for the response 45 , 17 , 53 , 63 . When these two directions are combined, the result is an empirical existence proof that defies the current scaling dogma. Trained from scratch on a low budget of only 40B unique tokens, HRM-Text achieves strong performance on most benchmarks against contemporary open models like Llama, Qwen, Gemma, OLMo, Ouro and Huginn 48 , 72 , 64 , 50 , 77 , 23 . Strikingly, it reaches this performance neighborhood using roughly 100\text{-}900\times fewer training tokens and 96\text{-}432\times less estimated training compute than these baselines, as shown in Figure 1 and Table 4 . We do not present HRM-Text as the final or optimal language model, but rather as proof that specific structural priors and targeted training objectives can radically alter the compute-to-performance ratio. Because the entry price is vastly reduced, this methodology democratizes foundational AI research. Pretraining from scratch is accessible again—we invite the community to join us in exploring how far smart architectures and focused objectives can go. 2 Methods Figure 2 : HRM-Text architecture. (a) Dual-timescale recurrent design comprising L and H modules. (b) L/H module internals featuring MagicNorm —PreNorm blocks followed by final norm. (c) Sigmoid-gated multi head self-attention. (d) PrefixLM mask enabling bidirectional attention on instruction. HRM-Text builds upon an improved HRM architecture, featuring a dual-timescale recurrence 69 . The forward pass is initialized with a high-level state, z_{H}^{0} , derived from the input token embeddings, alongside a fixed low-level state, z_{L}^{0} . The core processing sequence consists of two high-level cycles. Each cycle executes three fast L module updates followed by a single slow H module update. Token logits are generated by applying a linear head to the output of the final H module state. We employ a warmup deep credit assignment strategy: gradients are initially backpropagated through only the final two recurrent steps, expanding to the final five steps as training progresses. Internally, both the H and L recurrent modules are structured using MagicNorm . Additionally, we utilize parameterless RMSNorm (omitting the learnable \gamma parameter) 74 , SwiGLU activation functions 58 , Rotary Position Embeddings (RoPE) 60 , and a sigmoid-gated self-attention mechanism 52 . In contrast to standard autoregressive pretraining on raw text, we optimize a task-completion objective. The model is pretrained directly on instruction-response pairs (x_{q},x_{a}) from scratch using a negative log-likelihood (NLL) loss computed exclusively over the response, -\log P(x_{a}|x_{q}) . This objective is naturally paired with a PrefixLM attention mask, enabling full bidirectional attention across the instruction tokens. In the following sections, we detail the specific mechanics that enable HRM-Text’s extreme efficiency. Section 2.1 delves into our novel stabilization techniques, while Section 2.2 explores the task-completion pretraining objective and PrefixLM masking strategy. 2.1 Scaling to language with recurrence 2.1.1 Stabilization via MagicNorm Although the original HRM demonstrated strong performance on symbolic tasks, scaling recurrent architectures to language modeling introduces severe gradient-instability risks. Transformer design already involves a compromise in the placement of normalization layers 71 , 44 ; recurrence amplifies this compromise because the same transformation is repeatedly applied over many steps.

Read the complete text