TNT: Improving Chunkwise Training for Test-Time Memorization
Zeman Li 1,2 Ali Behrouz 2 Yuan Deng 2 Peilin Zhong 2 Praneeth Kacham 2 Mahdi Karami 2 Meisam Razaviyayn 1,2 Vahab Mirrokni 2 1 University of Southern California 2 Google Research · 2025
Read TNT: Improving Chunkwise Training for Test-Time Memorization by Zeman Li 1,2 Ali Behrouz 2 Yuan Deng 2 Peilin Zhong 2 Praneeth Kacham 2 Mahdi Karami 2 Meisam Razaviyayn 1,2 Vahab Mirrokni 2 1 University of Southern California 2 Google Research (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07343.txt.
Introduction
Zeman Li and seven co-authors remove the training bottleneck of recurrent language models with deep test-time memory, such as Titans and TTT. These models scale linearly with sequence length, unlike transformers, but train painfully slowly because a chunk-size compromise rules everything: big chunks are fast but inaccurate, small chunks accurate but slow. Their scheme, TNT, decouples the two stages.
Stage one pre-trains efficiently with a hierarchical memory: a global module digests large hardware-friendly chunks while parallel local modules handle detail, with periodic memory resets breaking sequential dependencies for massive parallelization. Stage two briefly fine-tunes only the local modules at a small, high-resolution chunk size. On Titans and TTT, TNT trained up to seventeen times faster than the most accurate baseline configuration while together improving accuracy.
Catalog record
| CREATOR | Zeman Li 1,2 Ali Behrouz 2 Yuan Deng 2 Peilin Zhong 2 Praneeth Kacham 2 Mahdi Karami 2 Meisam Razaviyayn 1,2 Vahab Mirrokni 2 1 University of Southern California 2 Google Research |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07343
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TNT: Improving Chunkwise Training for Test-Time Memorization Zeman Li 1,2 Ali Behrouz 2 Yuan Deng 2 Peilin Zhong 2 Praneeth Kacham 2 Mahdi Karami 2 Meisam Razaviyayn 1,2 Vahab Mirrokni 2 1 University of Southern California 2 Google Research Abstract Recurrent neural networks (RNNs) with deep test-time memorization modules, such as Titans and TTT, represent a promising, linearly-scaling paradigm distinct from Transformers. While these expressive models do not yet match the peak performance of state-of-the-art Transformers, their potential has been largely untapped due to prohibitively slow training and low hardware utilization. Existing parallelization methods force a fundamental conflict governed by the chunksize hyperparameter: large chunks boost speed but degrade performance, necessitating a fixed, suboptimal compromise. To solve this challenge, we introduce TNT, a novel training paradigm that decouples training efficiency from inference performance through a two-stage process. Stage one is an efficiency-focused pre-training phase utilizing a hierarchical memory. A global module processes large, hardware-friendly chunks for long-range context, while multiple parallel local modules handle fine-grained details. Crucially, by periodically resetting local memory states, we break sequential dependencies to enable massive context parallelization. Stage two is a brief fine-tuning phase where only the local memory modules are adapted to a smaller, high-resolution chunksize, maximizing accuracy with minimal overhead. Evaluated on Titans and TTT models, TNT achieves a substantial acceleration in training speed—up to 17 × \times faster than the most accurate baseline configuration—while simultaneously improving model accuracy. This improvement removes a critical scalability barrier, establishing a practical foundation for developing expressive RNNs and facilitating future work to close the performance gap with Transformers. 1 Introduction The demand for modeling long sequences highlights a fundamental limitation of standard softmax attention ( Vaswani et al. 2017 ) : its quadratic complexity bottlenecks scaling. This has spurred extensive research into more efficient architectures. Among these emerging paradigms, a particularly powerful approach is rooted in test-time memorization ( Sun et al. 2024 ) . Architectures leveraging this principle, which we refer to as deep memory modules , utilize a deep, online-adapted sub-network whose parameters are rapidly updated to encode in-context information. Prominent examples include Titans ( Behrouz et al. 2025c ) and Atlas ( Behrouz et al. 2025 ) . This method stands in sharp contrast to linear memory modules ( Yang et al. 2024 ; Yang et al. 2024b ; Dao & Gu 2024 ; Sun et al. 2023 ) , which, despite their efficiency, are constrained by matrix-valued hidden states and linear state transitions. By leveraging expressive non-linear objectives and update rules, deep memory modules can theoretically overcome these limitations. While these methods generally do not yet achieve the state-of-the-art performance of Transformers, they represent a potentially promising paradigm for efficient sequence modeling, provided their training bottlenecks can be resolved. Figure 1 : The basic diagram for illustraing TNT memory hierarchy. In each row, the updates at the same value of t ran at the same time (run in parrallel). t=0 is the initialization of the memory. Despite their expressive advantages, deep memory modules lack the efficient training algorithms of their linear counterparts, leading to low hardware utilization. Unlike linear memory modules, which utilize hardware-efficient parallelization, deep memory modules face challenges stemming from non-linear recurrences (e.g., LayerNorm between chunks) and the complexity of their deep structures. In practice, these challenges constrain their training to more frequent online updates on small data segments, resulting in poor computational throughput in training. This creates an inherent tension, as these models typically rely on a fixed, small chunk size (e.g., 16 to 64 tokens) to balance memory layer expressiveness against training efficiency. Consequently, this trade-off between in-context learning capability and computational performance has become a critical bottleneck preventing the application of these models to truly long sequences in practice. Resolving this fundamental tension is the primary goal of this work. Recent work attempts to mitigate this issue. Zhang et al. 2025 combines large chunks with local attention to enhance parallelism. However, this circumvents the inefficiency rather than solving it, complicates the analysis by mixing memory and attention, and neglects the need for small chunks (ideally 1) during inference. Concurrently, Guo et al. 2025 proposed a hierarchical memory system, but it is limited to linear memory modules and does not support short-term memories. To resolve this tension, we introduce TNT 0 0 0 TNT can be viewed as an abbreviation of Titans iNside Titans or TTT iNside TTT . It also hints to its “explosive” impact on training efficiency. , a novel training paradigm for deep memory modules. Our core insight is that different components of the model should process information at different granularities during distinct training stages. TNT begins with an efficiency-focused pre-training stage designed to maximize throughput. This is achieved via a hierarchical memory system: a global memory module operates on large, hardware-friendly chunks to capture long-range context, while multiple local memory modules handle fine-grained details in parallel. Crucially, we introduce a periodic reset mechanism for the local memory states. This breaks the sequential dependencies inherent even in non-linear RNNs (e.g., those with normalization between steps), enabling massive context parallelization. This is a key innovation, as efficiently parallelizing non-linear recurrences across the sequence length is a long-standing challenge, largely unsolved outside of Transformers and specialized linear RNNs (where parallel scans apply). Subsequently, a performance-focused fine-tuning stage adapts the model for optimal inference. During this stage, only the local memory modules are adjusted to use smaller chunk sizes, achieving high-resolution accuracy with minimal additional computational cost. This two-stage approach effectively decouples training efficiency from inference performance, significantly improving training scalability while addressing a key limitation of prior architectures. Furthermore, the local memory system itself can be hierarchical, employing multiple modules operating at different resolutions. This multi-resolution approach allows the model to capture complex, multi-scale temporal dynamics more effectively than a single fixed chunk size. TNT is a general training paradigm applicable to any deep memory module rather than a specific architecture. By decoupling training throughput from inference accuracy, we resolve a fundamental tension constraining prior work. This removes dependency on hardware-specific optimizations for small chunks and enables flexible exploration of the architectural design space. We believe this paradigm will open new research avenues towards replacing softmax attention. Our main contributions are summarized as follows: • We identify three fundamental challenges limiting the scalability and performance of deep memory modules: 1) domain mismatch between memory compression and retrieval; 2) tradeoff between memory performance and computational efficiency; 3) chunksize mismatch between training and inference (Section 3 ). • We introduce Q-K Projection, an efficient mechanism to resolve the domain mismatch between memory compression and retrieval (Section 4.1.2 ). • We introduce a novel hierarchical memory architecture with periodic state resets, enabling context parallelism for non-linear deep memory modules (Section 4.1 ). • We introduce an efficient fine-tuning mechanism to address chunksize mismatch between training and inference in deep memory modules (Section 4.2 ). • Putting all above together, we introduce TNT, a general two-stage training paradigm that decouples training efficiency from inference performance by combining efficient pre-training with high-resolution fine-tuning (Figure 1 , Figure 3 , Section 4 ). • We validate TNT on the Titans architecture, achieving up to a 17.37\times training speedup while improving accuracy, significantly advancing