MoVE: Mixture of Value Embeddings A New Axis for Scaling Parametric Memory in Auto-Regressive Models
Yangyan Li · 2026
Read MoVE: Mixture of Value Embeddings A New Axis for Scaling Parametric Memory in Auto-Regressive Models by Yangyan Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22887.txt.
Introduction
Yangyan Li proposes a new way to give autoregressive models more memory without more computing cost. These models, which predict one step at a time, power most modern AI for text and images. Normally, adding memory means making the network deeper or wider, which raises the cost of every step.
The proposal, MoVE, mixes many value embeddings to store knowledge in a new dimension of scale. If memory can grow while active compute stays flat, models could know more for the same price. That trade-off matters for serving AI to millions of users.
Catalog record
| CREATOR | Yangyan Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22887
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MoVE: Mixture of Value Embeddings A New Axis for Scaling Parametric Memory in Auto-Regressive Models Yangyan Li Abstract Autoregressive sequence modeling stands as the cornerstone of modern Generative AI, powering results across diverse modalities ranging from text generation to image generation. However, a fundamental limitation of this paradigm is the rigid structural coupling of model capacity to computational cost: expanding a model’s parametric memory—its repository of factual knowledge or visual patterns—traditionally requires deepening or widening the network, which incurs a proportional rise in active FLOPs. In this work, we introduce MoVE (Mixture of Value Embeddings) , a mechanism that breaks this coupling and establishes a new axis for scaling capacity. MoVE decouples memory from compute by introducing a global bank of learnable value embeddings shared across all attention layers. For every step in the sequence, the model employs a differentiable soft gating mechanism to dynamically mix retrieved concepts from this bank into the standard value projection. This architecture allows parametric memory to be scaled independently of network depth by simply increasing the number of embedding slots. We validate MoVE through strictly controlled experiments on two representative applications of autoregressive modeling: Text Generation and Image Generation. In both domains, MoVE yields consistent performance improvements over standard and layer-wise memory baselines, enabling the construction of “memory-dense” models that achieve lower perplexity and higher fidelity than their dense counterparts at comparable compute budgets. 1 Introduction The Transformer architecture ( 27 ) and the autoregressive modeling paradigm have expanded beyond their origins in Natural Language Processing (NLP) to become the unified engine of Generative AI. Whether predicting the next subword in a sentence or the next discrete visual code in an image ( 26 ) , the underlying principle remains the same: modeling the joint probability distribution of a sequence via conditional “next-token prediction.” Across these modalities, performance is governed by scaling laws ( 12 ; 9 ) , which traditionally impose a rigid tradeoff: increasing a model’s capacity to store and utilize information (its parametric memory) requires increasing the number of dense parameters by deepening or widening the network. This creates a structural bottleneck where scaling memory capacity incurs a linear penalty in training and inference FLOPs. Breaking this coupling is a significant area of interest for advancing GenAI, as it would allow for “memory-dense” models that possess vast encyclopedic or visual knowledge without the prohibitive compute cost of massive dense networks. In this work, we introduce MoVE (Mixture of Value Embeddings) , a modality-agnostic mechanism designed to address this efficiency challenge. MoVE adapts the spirit of memory augmentation ( 24 ) into a fully differentiable, dense form compatible with standard Transformers (see Figure 1 ). MoVE operates by augmenting the Attention mechanism’s Value stream —which mechanistic interpretability research identifies as a key carrier of semantic content ( 7 ) —with a global bank of learnable “concept vectors” shared across all attention layers. For each step in the sequence, the model computes a soft gating coefficient for every slot in this bank, dynamically mixing these retrieved concepts with the standard value projection. This allows the model to potentially offload the storage of static information (e.g., definitions of words, textures of objects) to the embedding bank, freeing up the dense attention layers for complex reasoning and structure building. We validate the efficacy of MoVE through strictly controlled comparisons on the two most prominent pillars of autoregressive modeling: Text Generation. We train Large Language Models (LLMs) ( 2 ) using standard open-source datasets. We compare MoVE against standard and strong baselines, maintaining identical hyperparameters to isolate the architectural differences. Image Generation. We integrate MoVE into the LlamaGen framework ( 26 ) , a representative autoregressive image generator. We show that MoVE improves the fidelity of generated images compared to the vanilla LlamaGen baseline, demonstrating that the mechanism’s benefits transfer effectively to visual memory. Furthermore, we demonstrate the architectural versatility of MoVE by extending it to Multi-Head Latent Attention (MLA) ( 4 ) . MLA is an optimized architecture that compresses Key-Value heads into a low-rank latent vector for inference efficiency. We show that MoVE extends seamlessly to this setting by injecting memory directly into the compressed latent space, enhancing capacity without disrupting the efficiency gains of MLA. By decoupling parametric memory from computational depth, MoVE establishes a distinct new axis for scaling. It serves as a general-purpose booster for autoregressive modeling, allowing practitioners to scale model capacity by simply increasing the number of mixture components. This enables the construction of “memory-dense” generative models that are smarter and richer in knowledge without the prohibitive cost of deepening the reasoning engine. 2 Related Work Unified Autoregressive Modeling. The autoregressive “next-token prediction” paradigm has proven to be a universal framework for generative AI. In NLP, this powers the dominant GPT series ( 2 ) , where performance scales predictably with compute. This paradigm has successfully transferred to Computer Vision, where works like ImageGPT ( 3 ) , VQGAN ( 6 ) , and Parti ( 29 ) demonstrate that visual representations can be learned via sequences of discrete tokens. Most recently, frameworks like LlamaGen ( 26 ) show that standard Transformers—without modality-specific biases—achieve effective synthesis purely through autoregressive scaling. This convergence implies that architectural innovations such as MoVE are inherently modality-agnostic. Scaling Laws and Efficient Architectures. Model performance is governed by rigorous scaling laws ( 12 ; 9 ) . However, traditional dense scaling requires a linear increase in training and inference FLOPs. To circumvent this, research has pivoted toward decoupled architectures. Universal Transformers ( 5 ) and ALBERT ( 16 ) attempted to decouple size from compute via cross-layer parameter sharing. While parameter-efficient, they do not inherently increase the model’s memory capacity relative to compute. MoVE explores an alternative direction by introducing a dedicated, scalable embedding bank that adds capacity without adding depth. Specifically regarding attention efficiency, Multi-Head Latent Attention (MLA) , introduced in DeepSeek-V2 ( 4 ) , addresses the memory bottleneck of the KV cache by projecting Keys and Values into a low-rank latent vector. Our work is complementary to MLA; while MLA reduces the memory footprint of inference, MoVE increases the parametric memory capacity of the model. We show that these two goals can coexist by applying MoVE directly to the compressed latent space. Figure 1 : Architectural Comparison. Left: Standard Attention, where the Value stream is a linear projection of the local hidden state X . Center: Layer-wise Value Embeddings (LaVE), a baseline that augments attention with a layer-specific memory vector gated by the input. Right: MoVE (Ours) , which decouples memory from depth by introducing a global bank of value embeddings. Crucially, both layers fetch value embeddings from this central shared bank, employing a differentiable router to dynamically mix these global concepts into the value stream. Conditional Computation and Mixture-of-Experts. A prominent success in decoupling parameters from compute is the Mixture-of-Experts (MoE) paradigm ( 23 ; 17 ; 10 ) , which routes tokens to sparse subsets of FFN “experts.” While successful, standard MoE relies on hard routing (top- k ), which often introduces training instability. Recent work on Soft MoE ( 21 ) solves this by performing a differentiable weighted average of experts. MoVE adopts a similar philosophy but applies it