MANAR: Memory-augmented Attention with Navigational Abstract Conceptual Representation Journal: XXXXXXXXXX
Zuher Jahshan, Ben Ben Ishay, Leonid Yavits · 2026
Read MANAR: Memory-augmented Attention with Navigational Abstract Conceptual Representation Journal: XXXXXXXXXX by Zuher Jahshan, Ben Ben Ishay, Leonid Yavits (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18676.txt.
Introduction
Jahshan, Ishay, and Yavits propose MANAR, an attention layer shaped by Global Workspace Theory. Standard attention lets all tokens talk at once, with no bottleneck. MANAR routes global context through a small trainable memory of abstract concepts. Retrieved concepts merge into one shared state, which then guides each token. Routing through that fixed-size state gives linear-time scaling. Weight copying from trained transformers eases adoption. Tests match or beat strong baselines on language, vision, and speech tasks, the authors report.
Catalog record
| CREATOR | Zuher Jahshan, Ben Ben Ishay, Leonid Yavits |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52387 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.18676
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MANAR: Memory-augmented Attention with Navigational Abstract Conceptual Representation Journal: XXXXXXXXXX Zuher Jahshan, Ben Ben Ishay, Leonid Yavits Abstract MANAR ( M emory-augmented A ttention with N avigational A bstract Conceptual R epresentation), contextualization layer generalizes standard multi-head attention (MHA) by instantiating the principles of Global Workspace Theory (GWT) . While MHA enables unconstrained all-to-all communication, it lacks the functional bottleneck and global integration mechanisms hypothesized in cognitive models of consciousness. MANAR addresses this by implementing a central workspace through a trainable memory of abstract concepts and an Abstract Conceptual Representation (ACR) . The architecture follows a two-stage logic that maps directly to GWT mechanics: (i) an integration phase , where retrieved memory concepts converge to form a collective "mental image" (the ACR) based on input stimuli; and (ii) a broadcasting phase , where this global state navigates and informs the contextualization of individual local tokens. We demonstrate that efficient linear-time scaling is a fundamental architectural byproduct of instantiating GWT functional bottleneck, as routing global information through a constant-sized ACR resolves the quadratic complexity inherent in standard attention. MANAR is a compatible re-parameterization of MHA with identical semantic roles for its projections, enabling knowledge transfer from pretrained transformers via weight-copy and thus overcoming the adoption barriers of structurally incompatible linear-time alternatives. MANAR enables non-convex contextualization , synthesizing representations that provably lie outside the convex hull of input tokens - a mathematical reflection of the creative synthesis described in GWT. Empirical evaluations confirm that MANAR matches or exceeds strong baselines across language (GLUE score of 85.1), vision (83.9% ImageNet-1K), and speech (2.7% WER on LibriSpeech), positioning it as an efficient and expressive alternative to quadratic attention. † † highlights: Brain-Inspired GWT Architecture : MANAR instantiates Global Workspace Theory via an Abstract Conceptual Representation (ACR) as a functional bottleneck for global information integration. Knowledge transfer and deployment : Compatible re-parameterization of MHA enables weight-copy transfer from pretrained transformers and drop-in replacement, reducing training cost and supporting deployment in existing systems (e.g. document understanding, information retrieval). Linear scaling and cross-modal performance : Linear time and memory scaling via the ACR workspace; MANAR matches or exceeds baselines on language, vision, and speech with up to 14.8 \times speedups and 9.3 \times memory reductions at scale. Non-Convex Synthesis : MANAR produces representations outside the convex hull of input tokens, enabling creative synthesis beyond standard attention. 1 Introduction Since its introduction, the transformer architecture [ 51 ] has achieved remarkable success across a wide spectrum of domains, including natural language processing [ 51 , 28 , 50 , 33 , 52 ] , computer vision [ 19 , 2 , 46 ] , speech recognition [ 44 , 6 , 34 ] , bioinformatics [ 11 , 1 , 26 ] , and many other domains. At the heart of this success lies the attention mechanism, which enables every token to attend to all other tokens in a sequence, yielding highly expressive, sequence-wide contextualizations and facilitating efficient parallel training. However, the same all-to-all contextualization that powers the expressivity of multi-head attention (MHA) introduces significant scalability bottlenecks. The quadratic time and memory complexity with respect to sequence length, together with the need to store a linearly growing, unbounded context in autoregressive generation, limits both the efficiency and the reach of contemporary attention-based models. This challenge has become increasingly pressing as workloads shift toward longer contexts, higher-resolution inputs, and larger-scale models, motivating numerous architectural and algorithmic strategies to address these limitations, including hierarchical attention, recurrence, compression, and explicit memory mechanisms. Among the most widely adopted approaches are quantization and knowledge distillation, which have succeeded in compressing the memory and compute footprint of transformer models. Quantization [ 3 , 37 , 38 , 56 ] enables more efficient computations and reduced memory footprint by lowering precision, sometimes as low as 4-bit per element [ 37 ] . Distillation [ 10 , 23 , 41 ] transfers knowledge from large models to smaller ones. Despite their practical utility, these methods fall short of solving the core issue: the direct all-to-all token contextualization is preserved, so context size remains unbounded and computational complexity quadratic. To address the unbounded-context problem, several lines of work introduce recurrence, compression, or explicit memory to extend context length without quadratic growth. Transformer-XL [ 13 ] introduces segment-level recurrence, allowing hidden states from previous segments to be reused as extended context while avoiding full recomputation. Building on this idea, the Compressive Transformer [ 43 ] augments recurrence with a compressed long-term memory, enabling retention of distant context at reduced resolution. Memory Transformer [ 12 ] augments standard transformers by adding trainable memory tokens that accumulate non-local representations and help capture properties of the entire sequence beyond what element-wise contextual embeddings provide. Such memory tokens can act as dedicated slots for storing global patterns, and bottlenecks can restrict global information propagation to emphasize essential representations. More recently, many works [ 55 , 20 , 47 , 32 , 35 , 40 , 54 ] focus on offloading the KV cache into lower memory tiers (e.g., CPU DRAM) and sparsely selecting KV pairs to attend to. These techniques exploit the empirical observation that, during contextualization, only a small subset of tokens significantly contributes due to highly peaked attention distributions. To capitalize on this behavior, several approaches augment multi-head attention with explicit memory units that store contextual blocks (i.e., KV blocks) for later retrieval. For example, InfLLM [ 55 ] contextualizes each token using both a local context window and a set of highly related KV blocks retrieved via approximate nearest-neighbor search. The retrieved blocks and the local context jointly participate in token contextualization, enabling scalable long-context inference. However, while these strategies effectively reduce active computation time, they do not resolve the fundamental architectural burden of maintaining a linearly growing and unbounded KV cache, which remains a primary bottleneck for scaling to ultra-long sequences. Many works seek to completely replace the standard attention mechanism with alternative architectures designed for enhanced scalability. For example, Titans [ 9 ] and ATLAS [ 8 ] augment standard attention (used as short-term memory) with explicit long-term neural memory modules, which are updated using test-time training and optional persistent memory to retrieve and fuse past information alongside current-context attention for scalable long-range modeling. State space models [ 22 ] formulate sequence modeling as linear dynamical systems, allowing fast, parallel computation and effective modeling of long-range context. Mamba [ 21 ] extends this paradigm by introducing selective sequence modeling through dynamic state selection and gating, enabling efficient and expressive representations for extremely long inputs [ 14 ] . Other notable advances, such as RetNet [ 48 ] , propose recurrent architectures with retention mechanisms that further boost performance, demonstrating strong results on long-context benchmarks. These approaches fundamentally rethink sequential model design and show particular promise in ultra-long sequence regimes. Nonetheless, a significant practical drawback of these alternative linear-time architectures is that they fundamentally alter the parameterization of the contextualization mechanism. Because they replace the standard attention mechanism with structurally different formulations, such models cannot directly inherit or easily transfer knowledge from existing pretrained transformer attention weights. This structural incompatibility creates a substantial barrier to practical adoption, as it prevents these architectures from leveraging the vast representational knowledge