Absorber LLM: Harnessing Causal Synchronization for Test-Time Training

Zhixin Zhang, Shabo Zhang, Chengcan Wu, Zeming Wei, Meng Sun · 2026

Read Absorber LLM: Harnessing Causal Synchronization for Test-Time Training by Zhixin Zhang, Shabo Zhang, Chengcan Wu, Zeming Wei, Meng Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20915.txt.

Introduction

Zhang, Zhang, Wu, Wei, and Sun propose Absorber LLM, a way to hold long context in model parameters without the memory cost of full attention. After absorbing past context, a context-free model is trained to match the original model's behavior on future text, syncing internal responses so causal effects survive. On long-context and streaming benchmarks it cuts inference memory and beats earlier parameter-as-memory baselines on accuracy.

Catalog record

CREATORZhixin Zhang, Shabo Zhang, Chengcan Wu, Zeming Wei, Meng Sun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37059
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.20915

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Absorber LLM: Harnessing Causal Synchronization for Test-Time Training Zhixin Zhang, Shabo Zhang, Chengcan Wu, Zeming Wei, Meng Sun Abstract Transformers suffer from a high computational cost that grows with sequence length for self-attention, making inference in long streams prohibited by memory consumption. Constant-memory alternatives such as RNNs and SSMs compress history into states with fixed size and thus lose long-tail dependencies, while methods that memorize contexts into parameters, such as Test-Time Training (TTT), are prone to overfitting token-level projection and fail to preserve the causal effect of context in pretrained LLMs. We propose Absorber LLM, which formulates long-context retention as a self-supervised causal synchronization: after absorbing historical contexts into parameters, a contextless model should match the original model with full context on future generations. We optimize this objective by synchronizing internal behaviors of the updated model with the original one, ensuring context absorption and generalization. Experiments on long-context and streaming benchmarks show that Absorber LLM reduces inference memory and improves accuracy over prior parameter-as-memory baselines. † † affiliationnotice: Equal contribution 1 Introduction Transformers ( Vaswani et al. 2017 ) underpin modern AI. However, in real-world deployments, transformer models struggle to satisfy further expectations, such as maintaining lifelong memories of user interactions across multiple domains ( Park et al. 2023 ; Wang et al. 2024 ; Jimenez et al. 2024 ; Bairi et al. 2024 ) , and achieving the ultimate goal of continuously absorbing a never-ending stream of data without resetting their state. This bottleneck stems from their inability to effectively process super-long contextual streams. Specifically, the self-attention mechanism of transformers ( Vaswani et al. 2017 ; Dao et al. 2022 ) incurs a quadratic computational complexity \mathcal{O}(N^{2}) to generate texts. Although variants like sparse ( Beltagy et al. 2020 ) and linearized attention ( Katharopoulos et al. 2020 ) mitigate this overhead, they merely delay the inevitable exhaustion of cache and computing resources in long-stream scenarios. To resolve these memory constraints, prior works have proposed linear-time models, including Recurrent Neural Networks (RNNs) ( Peng et al. 2023 ) and State Space Models (SSMs) ( Gu et al. 2021 ; Gu & Dao 2024 ) . These models achieve efficiency by compressing history into fixed-size hidden states. However, recent theoretical and empirical analyses reveal that this efficiency comes at the cost of model expressive power, leading to significant information loss compared to full-attention transformers ( Merrill et al. 2024 ; Hsieh et al. 2024 ) . Alternatively, Test-Time Training (TTT) ( Sun et al. 2025 ; Feng et al. 2026 ) methods utilize model parameters for context storage to overcome state capacity limits. Specifically, given a sequence X to be stored in the parameters of a model, TTT methods train the model to project X into a specifically constructed target V . Such methods optimize the model’s performance on X by writing it directly into the model parameters. Yet, when learning X , they ignore the model’s performance on subsequent sequences, failing to help the model absorb X in a way that contributes to future inferences. Crucially, such reconstruction methods overlook the causal mechanisms between historical contexts and subsequent inferences, which are essential for effective model deduction ( Geiger et al. 2024 ) . Figure 1 : A brief overview of our method. We propose to absorb historical contexts and move beyond simple context memorization, focusing on preserving the causal relationship between historical context and future generations rather than directly reconstructing historical tokens. Specifically, we fine-tune the context-less model to behave identically to the original model with full contexts. This is achieved through a self-supervised optimization that synchronizes the hidden states or output behaviors of the two models. This example shows the context absorption process, where the model absorbs 3 historical tokens by synchronization on 5 future tokens. Our method ensures that the absorbed context retains its semantic and causal influence on future deductions. In this paper, we draw inspiration from TTT and further propose to preserve the causal relation between historical contexts and the subsequent generation process. We introduce Absorber LLM (Fig. 1 ), which ensures causal effect preservation by training the contextless model to behave identically to the full-context model in future inferences. Note that compared to prior TTT methods, our approach focuses on causal effect synchronization rather than simple target projection. We maintain the contribution of removed historical sequences to future inferences, which is crucial for more effective deductions. Our main contributions are as follows: • We move beyond directly memorizing contexts to preserving their causal effects on subsequent deductions, and formally define such causal effect preservation as a functional equivalence problem, providing a standard for information transfer from contexts to parameters. • We introduce an update mechanism to satisfy the equivalence condition in transformer models, successfully synchronizing the behavior of the updated contextless model with the ideal full-context model, and realizing causal effect preservation and context absorption. • We validate the efficiency and effectiveness of our method on long-context benchmarks. Our results demonstrate that Absorber LLM not only outperforms traditional transformer models in computational efficiency but also achieves superior performance compared to prior linear-time and parameter memory methods, paving the way for scalable, infinite-stream inference in real-world deployments. 2 Motivation Figure 2 : A demonstration of the necessity of inference with a limited number of contexts. Normally, transformers compute f_{W}(x_{0}x_{1}\dots x_{n}) to generate the next token x_{n+1} with a computational complexity of \mathcal{O}(n) . If we absorb history into the parameters to get f_{W^{*}} and deduct with a limited context length, the deduction with discarded history has a reduced complexity of \mathcal{O}(1) , which suits scenarios with long sequence lengths. 2.1 Memory issue of Transformers Transformers exhibit quadratic computational complexity \mathcal{O}(N^{2}) due to their attention mechanism ( Vaswani et al. 2017 ) . Specifically, given a pretrained LLM f with parameter W , generating a single token x_{n+1}=f_{W}(x_{1},\dots,x_{n}) from an input sequence X=(x_{0},\dots,x_{n}) costs \mathcal{O}(n) with KV cache. Autoregressively, producing N tokens results in a cumulative \mathcal{O}(N^{2}) computational overhead and \mathcal{O}(N) memory footprint (Fig. 2 ). Numerous variants ( Child 2019 ; Beltagy et al. 2020 ; Katharopoulos et al. 2020 ; Choromanski et al. 2021 ; Liu et al. 2024 ) have been proposed to resolve this memory issue, but they merely delay the inevitable exhaustion of cache memory and computing resources. 2.2 Superiority of parameter memory The primary cause of the high computational cost of transformers in long contexts is their attention mechanism, which must process all historical contexts during deduction. To address this memory issue, prior works have proposed linear-time sequence models ( Peng et al. 2023 ; Gu et al. 2021 ; Gu & Dao 2024 ) that reduce the computational cost to \mathcal{O}(N) . Note that the linear-time models gain efficiency by sacrificing model capacity ( Merrill et al. 2024 ) , as they compress arbitrarily rich history into a fixed-size hidden state h_{t}\in\mathbb{R}^{d} . By contrast, the paradigm that restores contexts into the model parameters is more promising, since the parameter space \theta\in\mathbb{R}^{d\times d} is orders of magnitude larger than the activation space h\in\mathbb{R}^{d} , offering a vastly superior capacity for encoding history. Consequently, computationally costly contexts can be safely omitted from long-stream inferences. Formally, let X denote the historical

Read the complete text