Phasor Memory Networks: Stable Backpropagation Through Time for Scalable Explicit Memory

Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung · 2026

Read Phasor Memory Networks: Stable Backpropagation Through Time for Scalable Explicit Memory by Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13370.txt.

Introduction

Goo, Yun and Jung present Phasor Memory Networks, a fix for unstable explicit memory in sequence models. Classic memory designs suffer exploding or vanishing gradients during training through time, blocking practical language modelling. PMNet limits state changes to phase turns around a complex unit circle, preserving gradient norms without special initialization. With an 85 slot memory tree, it nears perfect retrieval on a copy paste task and matches a model three times larger on long context robustness.

Catalog record

CREATORSungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38270
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13370

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Phasor Memory Networks: Stable Backpropagation Through Time for Scalable Explicit Memory Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung Abstract For over a decade, explicit memory architectures like the Neural Turing Machine have remained theoretically appealing yet practically intractable for language modeling due to catastrophic gradient instability during Backpropagation Through Time. In this work, we break this stalemate with Phasor Memory Network (PMNet), a novel architecture that structurally resolves memory volatility through Unitary Phasor Dynamics and Hierarchical Learnable Anchors . Rather than relying on brute-force scaling, we present a mechanistic proof-of-concept in a controlled byte-level setting. By constraining recurrent state updates to phase rotations on a complex unit circle, PMNet preserves gradient norms and inherently prevents divergence without the need for specialized initialization. We empirically demonstrate the active actuation of the memory module through a synthetic Copy-Paste task, where PMNet utilizes an expansive 85-slot hierarchical memory tree ( = ∑ h = 1 4 4 h − 1 =\sum^{4}_{h=1}4^{h-1} ) to achieve near 100% exact retrieval across temporal distances that completely exceed the local sliding window attention’s receptive field. Furthermore, despite being a compact 119M parameter model trained on 18.8B tokens, PMNet matches the zero-shot long-context robustness of a Mamba model that is three times larger. Our ablation studies and gradient analyses confirm that the historical failure of explicit memory was a structural alignment problem, which PMNet effectively overcomes, providing a theoretically grounded foundation for scalable sequence modeling. Figure 1 : Evaluation of Zero-shot Generalization on PG-19. Comparison of Byte-level Perplexity (BPB, lower is better) on the PG-19 test set. (Green) PMNet (119M) (Ours): Trained on FineWeb-Edu. Despite the dual disadvantage of being 3\times smaller than the baseline and evaluated in a zero-shot setting, PMNet maintains stable extrapolation up to 512k bytes. This confirms that our Phasor Dynamics capture universal linguistic structures irrespective of domain training. (Cyan) MambaByte_Arxiv (353M): An OOD baseline. PMNet achieves parity with this significantly larger model, demonstrating superior parameter efficiency. (Blue) MambaByte_PG19 (353M): Shown as an in-domain oracle reference. (Red Dashed) Ablation: Performance degrades without memory updates, proving that the stability stems from the memory mechanism, not the SWA. (Gray) SmolLM (135M): Demonstrates the catastrophic failure of standard Transformers beyond their context window ( \approx 2k ), contrasting sharply with PMNet’s linear stability. 1 Introduction The recent evolution of Large Language Models (LLMs) has fundamentally transformed the landscape of artificial intelligence. However, as the demand for processing massive documents and long-form reasoning grows, the standard Transformer architecture faces a critical bottleneck: the quadratic computational complexity of its self-attention mechanism. This O(L^{2}) scaling makes it prohibitively expensive to maintain ultra-long context windows. To address this, alternative architectures such as Recurrent Neural Networks (RNNs) and State Space Models (SSMs) like Mamba have been proposed to achieve linear-time complexity. Despite their inference efficiency, training these models on extremely long sequences remains a formidable challenge. Rather than inherent flaws in gating mechanisms, the primary difficulties arise from managing non-unitary state dynamics and the accumulation of numerical errors during Backpropagation Through Time (BPTT). Without strict structural constraints, the recurrent states in these linear models are prone to unbounded growth or catastrophic decay. Consequently, modern linear architectures often rely on meticulous initialization or forced exponential decay to prevent gradients from exploding, which inherently limits their ability to flawlessly retrieve exact information across extended temporal horizons. In this paper, we argue that stabilizing BPTT requires a fundamental shift in learning mechanics rather than heuristic optimization patches. We introduce Phasor Memory Network (PMNet) , a novel architecture that provides a mechanistic proof-of-concept for stable, linear-time explicit memory. PMNet goes beyond merely storing hidden states; it explicitly separates static semantic anchors (memory embeddings) from dynamic context recurrence, allowing the model to directly access memory slots without catastrophic addressing drift. PMNet is built upon three core technical innovations: Phasor Dynamics for Unitary Recurrence We address the spectral instability of recurrent states by encoding information as phase angles on a complex unit circle. This enables purely additive updates that are structurally bounded and unitary, ensuring numerical stability and strictly preserving gradient norms without vanishing or exploding gradients, even over ultra-long sequences. Hierarchical Memory with Learnable Embeddings Because Phasor Dynamics guarantee stable BPTT, we can afford to implement a massive, tree-structured explicit memory hierarchy without the risk of gradient collapse. Each node is associated with a learnable embedding that acts as a semantic anchor, enabling efficient O(1) routing. Unlike traditional Neural Turing Machines (NTMs) 6 that suffer from volatile content-based addressing, these learnable anchors provide structurally permanent reference points. This guarantees that historical knowledge remains reliably retrievable, while the phasor states handle the continuous injection of new context. Parallel Segmented Scan & Adaptive Scaling To prevent gradient explosions caused by imbalanced routing collisions within the tree structure, we implement a parallelized segmented scan mechanism. By introducing segment-aware gradient normalization based on update frequency, we ensure robust optimization and strictly bounded energy flow across the memory hierarchy. Crucially, we validate the necessity and efficacy of Phasor Dynamics through rigorous empirical analysis of learning dynamics. Rather than solely relying on final perplexity scores, we analyze the training gradient norms to reveal a clear phase transition in the model’s behavior. We observed that local Sliding Window Attention (SWA) dominates early optimization, but as the required lookback distance exceeds the SWA’s receptive field, the global Phasor Memory actively awakens to route and retrieve exact information. Leveraging this structurally guaranteed stability, we adopt a byte-level modeling approach to eliminate tokenizer bias. Although raw byte sequences result in significantly longer contexts that typically cripple standard models, PMNet flawlessly executes a synthetic associative recall (Copy-Paste) task over thousands of steps. Furthermore, when scaled to 18.8B training tokens, a compact 119M parameter PMNet matches the zero-shot long-context robustness of an in-domain MambaByte model three times its size. We provide the source code at our repository 1 1 1 https://github.com/swgoo/pmnet . 2 Related Works The pursuit of efficient long-context modeling has led to several distinct research trajectories, ranging from linearizing attention to reviving recurrent dynamics and external memory systems. PMNet stands at the intersection of these fields, addressing the fundamental trade-offs between computational efficiency and stable information retention. 2.1 Linear Transformers and Recurrent Models To overcome the O(L^{2}) complexity of standard Transformers, various linear-time architectures have emerged. Linear Transformers 10 and models like RWKV 13 and RetNet 19 reformulate attention into a dual form that allows for recurrent inference. While these models achieve O(L) efficiency, they frequently rely on sophisticated, magnitude-based gating mechanisms to prevent state collapse. PMNet diverges from this trend by eliminating traditional gates entirely. Instead, it utilizes Phasor Dynamics to ensure state stability through rotation on a complex unit circle, providing a structural guarantee against the vanishing gradient issues often found in purely additive recurrent structures. 2.2 Long-context Transformers. Several approaches have been proposed to extend the context window of Transformers. 5 introduced segment-level recurrence (Transformer-XL) to capture longer dependencies, while 14 proposed ALiBi to enable length extrapolation beyond the training horizon. However, these methods still rely on attention mechanisms with quadratic or linear-growing complexity. In the realm of explicit memory, 21 utilized k-NN based lookup tables (Memorizing Transformers). Unlike these approaches, PMNet employs a fully differentiable, hierarchical phasor memory that achieves O(1) inference complexity with

Read the complete text