Scaling Embeddings Outperforms Scaling Experts in Language Models

Hong Liu, Jiaqi Zhang, Chao Wang, Xing Hu, Linkun Lyu · 2026

Read Scaling Embeddings Outperforms Scaling Experts in Language Models by Hong Liu, Jiaqi Zhang, Chao Wang, Xing Hu, Linkun Lyu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21204.txt.

Introduction

Hong Liu and four co-authors argue that a neglected dimension beats the fashionable one for scaling language models. Their 2026 paper compares embedding scaling with expert scaling. Mixture-of-Experts designs are the standard for sparsity scaling. They route each token to a few specialist subnetworks. Such designs face shrinking returns and system bottlenecks. The authors explore embedding scaling as a strong, orthogonal dimension.

Through analysis and tests, they find regimes where embedding scaling wins. It reaches a superior Pareto frontier there. A Pareto edge means better quality for the same cost. Widening embeddings enriches each token's picture at once. The result challenges where labs should spend their next budget. It is an architecture argument with direct cost stakes.

Catalog record

CREATORHong Liu, Jiaqi Zhang, Chao Wang, Xing Hu, Linkun Lyu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21204

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scaling Embeddings Outperforms Scaling Experts in Language Models Hong Liu, Jiaqi Zhang, Chao Wang, Xing Hu, Linkun Lyu Abstract While Mixture-of-Experts (MoE) architectures have become the standard for sparsity scaling in large language models, they increasingly face diminishing returns and system-level bottlenecks. In this work, we explore embedding scaling as a potent, orthogonal dimension for scaling sparsity. Through a comprehensive analysis and experiments, we identify specific regimes where embedding scaling achieves a superior Pareto frontier compared to expert scaling. We systematically characterize the critical architectural factors governing this efficacy—ranging from parameter budgeting to the interplay with model width and depth. Moreover, by integrating tailored system optimizations and speculative decoding, we effectively convert this sparsity into tangible inference speedups. Guided by these insights, we introduce LongCat-Flash-Lite, a 68.5B parameter model with ∼ \sim 3B activated trained from scratch. Despite allocating over 30B parameters to embeddings, LongCat-Flash-Lite not only surpasses parameter-equivalent MoE baselines but also exhibits exceptional competitiveness against existing models of comparable scale, particularly in agentic and coding domains. Hugging Face : https://huggingface.co/meituan-longcat/LongCat-Flash-Lite Figure 1: The architecture of a N-gram Embedding layer ( 14 ) . The embedding of each token is augmented by the N-gram Embedding branch. 1 Introduction The Mixture-of-Experts (MoE) architecture has firmly established itself as the dominant paradigm for scaling Large Language Models (LLMs), enabling massive parameter counts while maintaining manageable computational costs 18 . By dynamically routing tokens to a subset of experts, models decouple parameter capacity from computational cost, allowing LLMs to scale to trillons of parameters while keeping modest inference latency. However, as the model size and sparsity level increase, the marginal gain in performance diminishes, eventually approaching an efficiency saturation point ( 1 ) . Furthermore, the practical expansion of experts is constrained by system-level bottlenecks, particularly the escalating communication overhead and memory bandwidth pressure in distributed training. This necessitates the exploration of alternative, orthogonal dimensions for scaling sparse parameters beyond the Feed-Forward Networks (FFNs). In contrast to MoE, the embedding layer offers an overlooked, inherently sparse dimension with O(1) lookup complexity. This allows for massive parameter expansion without routing overheads—effectively achieving parameter extension without computation explosion. Theoretical foundations for this dimension have been established by scaling laws with vocabulary ( 36 ) , which posit that larger models necessitate proportionally larger vocabularies to maximize computation efficiency. To exploit this potential, diverse strategies have been proposed. One prominent direction is structural expansion, exemplified by Per-Layer Embedding (PLE) ( 11 ; 33 ; 3 ) , which allocates independent embedding parameters to each layer to scale capacity. Another key direction is vocabulary expansion via n-grams to densify information per token. This concept traces back to lookup-table language models ( 15 ) in the RNN era and has recently been advanced in LLMs ( 6 ; 14 ; 29 ; 5 ) . These approaches collectively highlight the embedding layer as a fertile ground for scaling. Despite the recent interest in expanding embedding parameters in LLMs, several key challenges remain underexplored. First, the comparative scaling efficiency between expert parameters and embedding parameters is not well understood, leaving the optimal allocation of capacity between these two sparse dimensions ambiguous. Second, the constraints of scaling embeddings are still not systematically characterized: it remains unclear how factors such as the total parameter budget, vocabulary size, initialization schemes, and the trade-offs between model width and depth jointly influence the effectiveness and stability of embedding scaling. Third, while some methods for scaling embeddings have been proposed, it is still unclear which scaling strategy is more effective and efficient under different regimes. Finally, scaling embeddings alters the input/output characteristics of the model during decoding, potentially impacting the overall I/O efficiency, yet its consequences for end-to-end inference performance remain insufficiently analyzed and optimized. In this technical report, we present a study to address these challenges and establish a robust framework for embedding scaling. Our contributions are as follows: • Comparison of Embedding Scaling vs. Expert Scaling : Through comprehensive scaling experiments across diverse scenarios, we identify specific regimes where embedding scaling achieves a superior Pareto frontier compared to increasing expert numbers, offering a high-efficiency alternative for model scaling. • Impact Analysis of Architectural Factors : We establish the complete set of architectural factors determining embedding scaling efficacy, covering the integration timing, parameter budgeting, hash collisions, hyperparameter settings and initialization of embedding, together with the effects of model width and depth. Besides, we investigate different methods of scaling embedding and find that N-gram Embedding offers the most robust scalability. • Inference Efficiency and System Optimization : We demonstrate that N-gram Embedding largely reduce I/O bottlenecks in MoE layers, particularly when paired with speculative decoding to maximize hardware utilization. Addressing the concomitant embedding overhead, we propose a specialized N-gram Cache and synchronized kernels, ensuring that the reduction in active parameters translates directly to lower latency and higher throughput. Based on these findings, we introduce and open-source LongCat-Flash-Lite, a model trained from scratch with 68.5B total parameters and 2.9B \sim 4.5B activated parameters depending on the context. Our evaluation demonstrates that LongCat-Flash-Lite not only surpasses a parameter-equivalent MoE baseline—validating the superior efficacy of allocating over 30B parameters to embeddings rather than experts—but also exhibits competitive performance against existing models of similar scale, particularly in agentic and coding tasks. 2 N-gram Embedding Layer To scale the embedding parameters, we adopt the N-gram Embedding introduced in 6 ; 14 ; 29 , which augments the representation of the embedding module by expanding a vocabulary-free n-gram embedding table. Specifically, for the i -th token t_{i} in a sequence, the augmented embedding e_{i} is calculated as follows \displaystyle e_{i}=\frac{1}{N}\big(E_{0}(t_{i})+\sum_{n=2}^{N}E_{n}(\mathcal{H}_{n}({t_{i-n+1},...,t_{i}}))\big), (1) \displaystyle\text{where}~t_{j}=0~\text{if}~j\leq 0, where E_{0}\in\mathbb{R}^{V_{0}\times D} is the original base embedding table with hidden size D , E_{n}\in\mathbb{R}^{V_{n}\times D} is the expanded embedding table, N denotes the maximum n-gram order and \mathcal{H}_{n} denotes the hash mapping function. We use the polynomial rolling hash funciton: \displaystyle\mathcal{H}_{n}({t_{i-n+1},...,t_{i}})=(\sum_{j=0}^{n-1}t_{i-j}*V_{0}^{j})\%V_{n}. (2) To further enhance the model’s expressive ability and reduce hash collisions, 6 ; 14 decompose each n-gram embedding table into K sub-tables with different vocabulary size. 14 further incorporate additional linear projection to map the outputs back to the original embedding space. The final version of N-gram Embedding (also referred to as Over-Encoding in 14 ) is shown in Figure 1 and can be written as \displaystyle e_{i} \displaystyle=\frac{1}{(N-1)K+1}\Big(E_{0}(t_{i})+\sum_{n=2}^{N}\sum_{k=1}^{K}W_{n,k}E_{n,k}(\mathcal{H}_{n,k}({t_{i-n+1},...,t_{i}}))\Big), (3) where E_{n,k}\in\mathbb{R}^{V_{n,k}\times D/((N-1)K)} is a sub-table and W_{n,k}\in\mathbb{R}^{D\times D/((N-1)K)} is the linear projection matrix. By setting the hidden size of sub-tables to be inversely proportional to the number of sub-tables, this design ensures that the parameter count of N-gram Embedding remains invariant with respect to N and K . 3 Comparative Analysis of Expert and Embedding Scaling This section presents our empirical findings regarding the comparison between scaling embeddings and scaling experts. Experiment Settings We integrate N-gram Embedding into the Longcat-Flash architecture ( 26 ) and conduct scaling experiments via from-scratch pre-training across varying activated parameter budgets (280M, 790M, and 1.3B). To rigorously compare scaling strategies, we establish a framework that contrasts scaling via N-gram Embedding against scaling experts. Specifically, for N-gram Embedding scaling, we first train MoE models with varying base sparsity levels ranging from 35% to 98% and incrementally incorporate N-gram Embedding from specific sparsity levels. Crucially, at each sparsity level, the

Read the complete text