Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
Zeyi Huang, Xuehai He, LiLiang Ren, Yiping Wang, Baolin Peng, Hao Cheng, Shuohang Wang, Pengcheng He, Jianfeng Gao, Yong Jae Lee, Yelong Shen, Microsoft University of Wisconsin-Madison University of Washington · 2026
Read Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior by Zeyi Huang, Xuehai He, LiLiang Ren, Yiping Wang, Baolin Peng, Hao Cheng, Shuohang Wang, Pengcheng He, Jianfeng Gao, Yong Jae Lee, Yelong Shen, Microsoft University of Wisconsin-Madison University of Washington (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26797.txt.
Introduction
Huang and ten colleagues study the Latent Recurrent Transformer, which reuses a high-level hidden state from the prior token as memory for the next token. To train this recurrence without unrolling full sequences, they introduce interleaved parallel training, mixing one full-sequence pass with parallel refinement of position subsets. Across 1.3B- and 2.1B-parameter backbones, the design improves bits-per-byte and CORE scores at matched compute, beats two-forward PonderLM-2, and matches a three-loop transformer at one forward per token.
Catalog record
| CREATOR | Zeyi Huang, Xuehai He, LiLiang Ren, Yiping Wang, Baolin Peng, Hao Cheng, Shuohang Wang, Pengcheng He, Jianfeng Gao, Yong Jae Lee, Yelong Shen, Microsoft University of Wisconsin-Madison University of Washington |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52605 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26797
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior Zeyi Huang, Xuehai He, LiLiang Ren, Yiping Wang, Baolin Peng, Hao Cheng, Shuohang Wang, Pengcheng He, Jianfeng Gao, Yong Jae Lee, Yelong Shen, Microsoft University of Wisconsin-Madison University of Washington Abstract We study Latent Recurrent Transformer (LRT), a lightweight augmentation of autoregressive transformers that reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token. Because this source state is already computed during ordinary decoding, LRT adds a cross-layer recurrent latent pathway across positions without inserting pause tokens or extra depth loops, and the standard attention mechanism and KV-cache interface are preserved. To pretrain this recurrence at scale without sequentially unrolling the transformer, we introduce interleaved parallel training : a single full-sequence initialization forward builds a shared buffer, then disjoint position subsets are refined in parallel and written back, so all tokens receive recurrent-memory-aware supervision at roughly 2 × 2\times baseline compute. Across nanochat-style backbones and a wide range of tokens-per-parameter budgets, LRT improves both language-modeling loss and in-context learning under matched effective compute while adding as little as 0.3 % 0.3\% parameters. 1 Introduction † † Work done during Zeyi’s internship at Microsoft. Correspondence to {zeyihuang,yongjaelee}@cs.wisc.edu and {xuehaihe,yeshe}@microsoft.com. Autoregressive transformers are the standard architecture for language modeling ( 40 ; 32 ; 3 ) , but each generated token is still produced by a fixed-depth feedforward computation. A natural way to increase computation is to introduce recurrence. Existing approaches often add recurrence either in depth, by repeatedly applying blocks to the same token ( 7 ; 11 ) , or in time, by inserting pause or thinking tokens before emitting each real token ( 12 ) . While these methods can enable iterative refinement, they also increase inference cost through extra block applications or additional decoding steps. We observe that autoregressive decoding already computes high-level recurrent signals for free: the hidden states of the previous token. Upper-layer states are trained toward next-token prediction and can provide useful latent context for processing the next position. This motivates a simple question: can we reuse an already-computed high-level representation from the previous tokens as recurrent memory, without adding extra decoding steps? We study Latent Recurrent Transformer (LRT), a lightweight augmentation of standard autoregressive transformers. At token position t , LRT reuses a source-layer hidden state from the previous position as recurrent memory, \mathbf{m}_{t-1}=\mathbf{h}^{\ell_{\mathrm{src}}}_{t-1} , where \ell_{\mathrm{src}} is the source layer. It processes the usual token input with the standard KV cache, while injecting \mathbf{m}_{t-1} into transformer layers through lightweight mechanisms such as KV Projection and Residual Injection . LRT preserves the decoder-only backbone, attention mechanism, feedforward layers, and KV-cache interface; the memory acts as an auxiliary latent pathway across adjacent autoregressive steps. This latent pathway gives LRT a cross-layer route that standard causal attention lacks. In a KV-cached transformer, layer \ell at position t can attend to previous tokens only through cached states from the same layer \ell ; early layers therefore cannot directly use higher-layer representations of previous tokens. By injecting the source-layer memory into the current token’s layers, LRT lets early computation at position t access higher-level information already computed for position t-1 , while still using the standard KV cache and one normal forward pass per generated token. The main challenge is pretraining. At inference time, LRT naturally forms a token-level recurrent chain, \mathbf{m}_{1}\rightarrow\mathbf{m}_{2}\rightarrow\cdots\rightarrow\mathbf{m}_{T} . Exactly reproducing this chain during training would require sequentially unrolling the transformer over the full sequence, destroying the parallelism that makes transformer pretraining efficient. Chunked training preserves parallelism within chunks, similar to segment- or block-level recurrent computation ( 5 ; 17 ; 37 ) , but recurrent memory is only propagated across chunk boundaries. We therefore introduce interleaved parallel training . It first builds a full-sequence buffer with an initialization pass, then refines disjoint interleaved subsets of positions and writes their updated states back to the buffer. Later subsets can consume memory updated by earlier subsets, while every position receives a recurrent-memory-aware refinement step under a fixed training budget. Compared with chunked training, this gives a finer approximation to token-level recurrence while retaining parallel computation within each subset. Empirically, LRT shifts scaling curves toward lower bits per byte (BPB) ( 22 ) and higher CORE ( 22 ) under baseline-equivalent training compute across model sizes and tokens-per-parameter budgets. The default shared-projection variant adds only 0.3\% parameters. Ablations show that KV Projection and Residual Injection are complementary, and that an upper-middle source layer can provide stronger recurrent memory than the final layer, suggesting that useful memory should be high-level but not overly specialized for logits. In summary, we make three contributions. First, we propose Latent Recurrent Transformer, a lightweight recurrent extension of autoregressive transformers that reuses a high-level source-layer hidden state from the previous token as memory for the next position, without adding extra decoding steps in its default form. Second, we introduce interleaved parallel training, a parallel approximation that refines disjoint token subsets to mimic token-level recurrence under a fixed training budget. Third, we show that LRT improves the compute–quality trade-off over matched-depth transformer baselines across model sizes and tokens-per-parameter budgets, with the default shared-projection variant adding only 0.3\% parameters. Figure 1 : Overview of Latent Recurrent Transformer (LRT). (a) Depth-recurrent methods increase computation by repeatedly applying model blocks to the same token before emitting an output. (b) Default LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token, creating a cross-layer, cross-token latent recurrence without adding extra autoregressive decoding steps. Each generated token still uses one normal transformer forward pass. 2 Latent Recurrent Transformer Architecture Latent Recurrent Transformer (LRT) augments a standard autoregressive transformer with a lightweight recurrent memory across adjacent token positions. Let L be the number of transformer layers, d the hidden dimension, and \mathbf{h}^{\ell}_{t}\in\mathbb{R}^{d} the hidden state at position t after layer \ell . We choose a source layer \ell_{\mathrm{src}} and define the recurrent memory as \mathbf{m}_{t}=\mathbf{h}^{\ell_{\mathrm{src}}}_{t} . This memory is already computed during ordinary autoregressive decoding and can be reused when processing the next token without adding extra decoding steps. At position t , LRT processes the current token with the standard KV cache \mathcal{C}_{<t} while additionally injecting \mathbf{m}_{t-1} into the transformer: \mathbf{h}^{L}_{t},\mathbf{z}_{t},\mathbf{m}_{t},\mathcal{C}_{\leq t}=f_{\theta}\!\left(x_{t},\mathbf{m}_{t-1},\mathcal{C}_{<t}\right), (1) where \mathbf{z}_{t} are the next-token logits, \mathbf{m}_{t}=\mathbf{h}^{\ell_{\mathrm{src}}}_{t} is the updated recurrent memory, and \mathcal{C}_{\leq t} is the updated KV cache. At the first position, \mathbf{m}_{t-1} is initialized to zero. LRT adds a cross-token, cross-layer pathway that is not available in a standard KV-cached transformer. Standard causal attention lets layer \ell at position t attend to cached states from previous positions at the same layer; LRT lets target layers at position t access the source-layer memory from position t-1 . Thus, even an intermediate source layer can provide high-level feedback to earlier layers of the next token. The standard attention mechanism and KV-cache interface are preserved: LRT adds no memory tokens and does not change the cache shape, but only changes how current-token layer inputs or key-value vectors are formed. Our default LRT combines two lightweight injection mechanisms: KV Projection , which injects memory through the attention key-value pathway, and Residual Injection , which adds memory to the residual stream. Architecture