Fast Spatial Memory with Elastic Test-Time Training

Ziqiao Ma 1 , 2 ∗ Xueyang Yu 3 ∗ Haoyu Zhen 3 Yuncong Yang 3 Joyce Chai 2 Chuang Gan 1 , 3, 2 University of Michigan, 3 University of Massachusetts Amherst · 2026

Read Fast Spatial Memory with Elastic Test-Time Training by Ziqiao Ma 1 , 2 ∗ Xueyang Yu 3 ∗ Haoyu Zhen 3 Yuncong Yang 3 Joyce Chai 2 Chuang Gan 1 , 3, 2 University of Michigan, 3 University of Massachusetts Amherst (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-07350.txt.

Introduction

Ma and five colleagues stabilize test-time training for long 3D and 4D reconstruction, where fully plastic updates forget and overfit. Their elastic method anchors fast-weight updates with a Fisher-weighted prior around a slowly moving average of past weights, balancing stability against plasticity. The resulting Fast Spatial Memory model learns space-time scenes from long observation runs and renders new view-time combinations. It adapts fast over long sequences, rebuilds scenes well with smaller chunks, and eases the activation-memory bottleneck.

Catalog record

CREATORZiqiao Ma 1 , 2 ∗ Xueyang Yu 3 ∗ Haoyu Zhen 3 Yuncong Yang 3 Joyce Chai 2 Chuang Gan 1 , 3, 2 University of Michigan, 3 University of Massachusetts Amherst
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52921
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.07350

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fast Spatial Memory with Elastic Test-Time Training Ziqiao Ma 1 , 2 ∗ Xueyang Yu 3 ∗ Haoyu Zhen 3 Yuncong Yang 3 Joyce Chai 2 Chuang Gan 1 , 3, 2 University of Michigan, 3 University of Massachusetts Amherst Abstract Large Chunk Test-Time Training (LaCT) has shown strong performance on long-context 3D reconstruction, but its fully plastic inference-time updates remain vulnerable to catastrophic forgetting and overfitting. As a result, LaCT is typically instantiated with a single large chunk spanning the full input sequence, falling short of the broader goal of handling arbitrarily long sequences in a single pass. We propose Elastic Test-Time Training inspired by elastic weight consolidation, that stabilizes LaCT fast-weight updates with a Fisher-weighted elastic prior around a maintained anchor state. The anchor evolves as an exponential moving average of past fast weights to balance stability and plasticity. Based on this updated architecture, we introduce Fast Spatial Memory ( FSM ), an efficient and scalable model for 4D reconstruction that learns spatiotemporal representations from long observation sequences and renders novel view-time combinations. We pre-trained FSM on large-scale curated 3D/4D data to capture the dynamics and semantics of complex spatial environments. Extensive experiments show that FSM supports fast adaptation over long sequences and delivers high-quality 3D/4D reconstruction with smaller chunks and mitigating the camera-interpolation shortcut. Overall, we hope to advance LaCT beyond the bounded single-chunk setting toward robust multi-chunk adaptation, a necessary step for generalization to genuinely longer sequences, while substantially alleviating the activation-memory bottleneck. Figure 1 : Fast Spatial Memory ( FSM ) is an efficient, scalable 4D reconstruction model that learns spatiotemporal representations from long sequences to render novel views at novel times. The model is powered by Large Chunk Elastic Test-Time Training (LaCET) blocks and is compatible with a range of rendering decoders, including LRM-style and LVSM-style decoders. ${}^{*}$ ${}^{*}$ footnotetext: Authors contribute equally to this work. 1 Introduction Building a spatial memory would require learning to compress visual observations across viewpoints and time into a unified 4D representation that preserves both spatial structure and temporal dynamics. This capability would advance applications in 4D asset generation 58 ; 39 for video games, film production, and AR/VR, as well as world modeling 22 ; 74 for embodied AI and robotics. Especially, reconstructing dynamic scenes from temporally extended and dynamically sampled observations (e.g., long videos captured by moving cameras) remains a central challenge. Recent advances in Large Reconstruction Models (LRMs) 14 ; 70 and Large View Synthesis Models (LVSM) 17 ; 23 offer promising rendering-based alternatives to efficient and high-quality 3D/4D reconstruction. Typically built on Transformer-based sequence modeling, these methods achieve strong reconstruction performance by learning powerful priors over structure and appearance from large-scale multi-view data. Despite these advances, these models remain constrained by the amount of activation memory available for a single forward pass, leaving long-context modeling largely unresolved. This is particularly the case in 4D domain, where videos that are temporally extended yet spatially sparsely observed, and their reconstruction quality degrades sharply beyond the training context length, indicating limited temporal scalability 34 . While several 3D reconstruction works have explored hybrid sequence models that combine linear-time state-based mixers with full attention 80 ; 79 , the central question for practical 4D modeling remains open: How can we design a simple, scalable, and efficient spatial memory architecture that learns scene-level spatiotemporal representations from long sequences? Test-Time Training (TTT) 41 ; 44 has shown promise in addressing the long-context issue in geometric reconstruction and view synthesis 8 ; 69 ; 49 . Especially, Large Chunk Test-Time Training (LaCT) 73 enables in-forward, chunk-wise fast-weight adaptation that lets a transformer recalibrate its internal representations during inference, efficiently updating small parameters from key-value statistics without backpropagation to achieve self-refining, test-time adaptation. Yet, these techniques do not directly generalize to the 4D regime, where scene dynamics evolve across space and time during inference, since the fully plastic nature of continuous LaCT updates leads to uncontrolled fast-weight drift, leading to overfitting in training and unstable updates at test time. This is analogous to catastrophic forgetting at inference time. To address this issue, we introduce Elastic Test-Time Training that executes an additional consolidate operation after LaCT update , inspired by the Elastic Weight Consolidation (EWC) 24 in continual learning. Each fast-weight module keeps a reference set of anchor parameters (the values before adaptation) and continuously estimates their importance through an online Fisher-style statistic. During inference, important parameters are softly pulled back toward their anchors, while less critical ones remain free to adjust. This elastic behavior acts as an adaptive spring: it constrains unstable drift without sacrificing responsiveness to new lighting, pose, or scene conditions, transforming the base transformer into a fast, self-refining yet elastic 4D learner, one that keeps adapting to the stream while remembering where it came from. We refer to this new architecture as Large Chunk Elastic Test-Time Training (LaCET). We scale LaCET up to pretrain a Fast Spatial Memory ( FSM ) on a curated set of 3D/4D datasets with posed images captured over time and from different cameras. We primarily evaluated FSM on the novel view synthesis (NVS) and demonstrated its competitive performance on a variety of benchmarks and the scalability of LaCET. The model scales effectively with more data and larger model size and generalizes well to novel scenes. With careful ablation studies, we show that LaCET can effectively mitigate the overfitting and undesirable inference time behaviors of LaCT, e.g., camera interpolation. To our knowledge, FSM is the first large-scale 4D reconstruction model design that supports input from long sequences of views and arbitrary timestamps and renders arbitrary novel view-time combinations. Overall, we hope to advance LaCT beyond the bounded single-chunk setting toward robust multi-chunk adaptation, a necessary step for generalization to genuinely longer sequences, while substantially alleviating the activation-memory bottleneck. 2 Algorithmic Preliminaries Figure 2 : (Left) Overview of FSM . The model takes a sequence of posed images captured at different times and learns to infer novel view-time combinations. Camera information is converted into Plücker ray maps as geometric augmentation for visual tokens. The model directly predict the target view with decoders. (Right) The LaCET Block. It maintains two sets of parameters, anchor weights and fast weights . During adaptation, the fast weights are updated using information from the current chunk (queries, keys, and values), while the anchor weights act as a stable reference. The model tracks parameter importance online and softly restores critical weights toward their anchors to prevent drift. This stabilizes rapid updates while preserving the adaptability of TTT, addressing the plasticity issue. 2.1 Fast Weights and Test-Time Training Test-Time Training (TTT) 44 introduces fast weights 41 with rapidly adaptable parameters, which get updated at both training and inference time. This is in sharp contrast to slow weights (conventional model parameters) that remain fixed at inference time. In the context of attention, we consider a sequence of N tokens \mathbf{x}=[x_{1},x_{2},\dots,x_{N}] , where each token x_{i} is projected into key k_{i} , query q_{i} , and value v_{i} vectors. Formally, TTT defines a function f_{\bm{\theta}}(\cdot) parameterized by the fast weights \bm{\theta} , and it involves an update and an apply operation. The (per-token) update operation defines: \bm{\theta}^{\prime}=\bm{\theta}-\eta\,\nabla_{\bm{\theta}}\mathcal{L}\big(f_{\bm{\theta}}(k_{i}),v_{i}\big), (1) where \eta represents the learning rate and

Read the complete text