End-to-End Test-Time Training for Long Context

Arnuv Tandon, Karan Dalal, Xinhao Li, Daniel Koceja, Marcel Rød, Sam Buchanan, Xiaolong Wang, Jure Leskovec, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin, Jed McCaleb, Yejin Choi, Yu Sun · 2025

Read End-to-End Test-Time Training for Long Context by Arnuv Tandon, Karan Dalal, Xinhao Li, Daniel Koceja, Marcel Rød, Sam Buchanan, Xiaolong Wang, Jure Leskovec, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin, Jed McCaleb, Yejin Choi, Yu Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23675.txt.

Introduction

Arnuv Tandon and thirteen co-authors reframe long-context language modeling as continual learning rather than architecture design. Instead of building ever more elaborate attention mechanisms, their model is a standard transformer with sliding-window attention that keeps learning while it reads. At test time it trains on the very context it is given, compressing that context into its own weights through next-token prediction.

Meta-learning during training teaches the model how to learn well at test time, making the method end-to-end in both phases. In scaling experiments, models trained this way tracked the context-length scaling of full-attention transformers, which rival sequence models failed to match, while inference latency stayed constant like a recurrent network's. For very long contexts it ran nearly three times faster than full attention. Code is public.

Catalog record

CREATORArnuv Tandon, Karan Dalal, Xinhao Li, Daniel Koceja, Marcel Rød, Sam Buchanan, Xiaolong Wang, Jure Leskovec, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin, Jed McCaleb, Yejin Choi, Yu Sun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-23675

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

End-to-End Test-Time Training for Long Context Arnuv Tandon, Karan Dalal, Xinhao Li, Daniel Koceja, Marcel Rød, Sam Buchanan, Xiaolong Wang, Jure Leskovec, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin, Jed McCaleb, Yejin Choi, Yu Sun Abstract We formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture – a Transformer with sliding-window attention. However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights. In addition, we improve the model’s initialization for learning at test time via meta-learning at training time. Overall, our method, a form of Test-Time Training (TTT), is End-to-End (E2E) both at test time (via next-token prediction) and training time (via meta-learning), in contrast to previous forms. We conduct extensive experiments with a focus on scaling properties. In particular, for 3B models trained with 164B tokens, our method (TTT-E2E) scales with context length in the same way as Transformer with full attention, while others, such as Mamba 2 and Gated DeltaNet, do not. However, similar to RNNs, TTT-E2E has constant inference latency regardless of context length, making it 2.7 × 2.7\times faster than full attention for 128K context. Our code is publicly available. Figure 1 : Scaling with context length, in terms of test loss (left) and latency (right). Left: Our method (TTT-E2E) turns the worst line (green) into the best (blue) at 128K context length. Loss \Delta ( \downarrow ), the y -value, is computed as (loss of the reported method) - (loss of Transformer with full attention), so loss \Delta of full attention itself (orange) is the flat line at y=0 . While other methods produce worse loss \Delta in longer context, TTT-E2E maintains the same advantage over full attention. All models have 3B parameters and are trained with 164B tokens. Right: Similar to SWA and the RNN baselines, TTT-E2E has constant inference latency regardless of context length, making it 2.7\times faster than full attention for 128K context on an H100. 1 Introduction Humans are able to improve themselves with more experience throughout their lives, despite their imperfect recall of the exact details. Consider your first lecture in machine learning: You might not recall the instructor’s first word during the lecture, but the intuition you learned is probably helping you understand this paper, even if that lecture happened years ago. On the other hand, Transformers with self-attention still struggle to efficiently process long context equivalent to years of human experience, in part because they are designed for nearly lossless recall. Self-attention over the full context, also known as full attention, must scan through the keys and values of all previous tokens for every new token. As a consequence, it readily attends to every detail, but its cost per token grows linearly with context length and quickly becomes prohibitive. As an alternative to Transformers, RNNs such as Mamba 2 [ 32 ] and Gated DeltaNet [ 104 ] have constant cost per token, but become less effective in longer context, as shown in Figure 1 . Some modern architectures approximate full attention with a sliding window [ 1 , 107 ] , or stack attention and RNN layers together [ 91 , 11 ] . However, these techniques are still less effective than full attention in using longer context to achieve better performance in language modeling. How can we design an effective method for language modeling with only constant cost per token? Specifically, how can we achieve better performance in longer context without recalling every detail, as in the opening example? The key mechanism is compression. For example, humans compress a massive amount of experience into their brains, which preserve the important information while leaving out many details. For language models, training with next-token prediction also compresses a massive amount of data into their weights. So what if we just continue training the language model at test time via next-token prediction on the given context? This form of Test-Time Training (TTT), similar to an old idea known as dynamic evaluation [ 72 , 60 ] , still has a missing piece: At training time, we were optimizing the model for its loss out of the box, not for its loss after TTT. To resolve this mismatch, we prepare the model’s initialization for TTT via meta-learning [ 38 , 79 , 58 ] instead of standard pre-training. Specifically, each training sequence is first treated as if it were a test sequence, so we perform TTT on it in the inner loop. Then we average the loss after TTT over many independent training sequences, and optimize this average w.r.t. the model’s initialization for TTT through gradients of gradients in the outer loop [ 71 , 3 , 27 ] . In summary, our method is end-to-end in two ways. Our inner loop directly optimizes the next-token prediction loss at the end of the network, in contrast to prior work on long-context TTT [ 86 , 110 ] ; Subsection 2.4 explains this difference through an alternative derivation of our method. Moreover, our outer loop directly optimizes the final loss after TTT, in contrast to dynamic evaluation [ 72 , 60 ] , as discussed. Our key results are highlighted in Figure 1 , with the rest presented in Section 3 . The conceptual framework of TTT has a long history with many applications beyond long context, and many forms without meta-learning [ 85 , 12 , 45 , 2 ] . Our work is also inspired by the literature on fast weights [ 38 , 79 , 77 , 49 ] , especially [ 17 ] by Clark et al., which shares our high-level approach. Section 4 discusses related work in detail. 2 Method Figure 2 : Toy example. Left: Given x_{1} and x_{2} as context, we want to predict the unknown x_{3} . Our toy baseline, a Transformer without self-attention (using only the upward arrows), is effectively a bigram since it has no memory of x_{1} . TTT (using all the arrows) first tries to predict x_{2} from x_{1} as an exercise: It computes the loss \ell_{2} between x_{2} and the prediction \hat{p}_{2} , then takes a gradient step on \ell_{2} . Now information of x_{1} is stored in the updated MLPs (blue). Right: Token-level test loss \ell_{t} for various methods in our toy example, as discussed in Subsection 2.2 , except for TTT-E2E b=16 discussed in Subsection 2.3 . In particular, TTT-E2E b=1 turns the green line (our toy baseline) into the blue line, which performs almost as well as orange (using full attention). Consider the standard task of next-token prediction, which consists of two phases at test time: • Prefill: conditioning on T+1 given tokens x_{0},x_{1},\dots,x_{T} , where x_{0} is the Beginning of Sequence ( <BOS> ) token. • Decode: predicting a distribution \hat{p}_{T+1} over all possible instantiations of the next token. The test loss is then \texttt{CE}\left(\hat{p}_{T+1},x_{T+1}\right) , where CE is the cross entropy and x_{T+1} is generated by nature. For ease of exposition, we first focus on the task of prefilling T+1 tokens and then decoding a single token. In this setting, self-attention over the full context, also known

Read the complete text