Beyond N-gram: Data-Aware X-gram Extraction for Efficient Embedding Parameter Scaling
Yilong Chen 1,2,4,∗ , Yanxi Xie 3,∗ , Zitian Gao 4 , He Xin 4 , Yihao Xiao 4 , Jason Klein Liu 4 ,Haoming Luo 4 , Yifan Luo 4 , Zhengmao Ye 4 , Tingwen Liu 1,2 , Xin Zhao 4 , Ran Tao 4,† , Bryan Dai 4,† 1 Institute of Information Engineering, Chinese Academy of Sciences 2 School of Cyber Security, University of Chinese Academy of Sciences 3 Peking University 4 IQuest Research Code: https://github.com/Longyichen/X-gram · 2026
Read Beyond N-gram: Data-Aware X-gram Extraction for Efficient Embedding Parameter Scaling by Yilong Chen 1,2,4,∗ , Yanxi Xie 3,∗ , Zitian Gao 4 , He Xin 4 , Yihao Xiao 4 , Jason Klein Liu 4 ,Haoming Luo 4 , Yifan Luo 4 , Zhengmao Ye 4 , Tingwen Liu 1,2 , Xin Zhao 4 , Ran Tao 4,† , Bryan Dai 4,† 1 Institute of Information Engineering, Chinese Academy of Sciences 2 School of Cyber Security, University of Chinese Academy of Sciences 3 Peking University 4 IQuest Research Code: https://github.com/Longyichen/X-gram (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21724.txt.
Introduction
Chen and twelve colleagues propose X-GRAM, a frequency-aware way to scale language-model memory without scaling compute. Large token lookup tables waste parameters on rarely seen tokens and grow memory fast. X-GRAM compresses the long tail with hybrid hashing and alias mixing, then refines vectors to pull out local n-gram features. At sub-billion and just-over-billion scales, it gains up to 4.4 points over the plain backbone and about 3 points over strong retrieval baselines, with smaller tables.
Catalog record
| CREATOR | Yilong Chen 1,2,4,∗ , Yanxi Xie 3,∗ , Zitian Gao 4 , He Xin 4 , Yihao Xiao 4 , Jason Klein Liu 4 ,Haoming Luo 4 , Yifan Luo 4 , Zhengmao Ye 4 , Tingwen Liu 1,2 , Xin Zhao 4 , Ran Tao 4,† , Bryan Dai 4,† 1 Institute of Information Engineering, Chinese Academy of Sciences 2 School of Cyber Security, University of Chinese Academy of Sciences 3 Peking University 4 IQuest Research Code: https://github.com/Longyichen/X-gram |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81602 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.21724
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond N-gram: Data-Aware X-gram Extraction for Efficient Embedding Parameter Scaling Yilong Chen 1,2,4,∗ , Yanxi Xie 3,∗ , Zitian Gao 4 , He Xin 4 , Yihao Xiao 4 , Jason Klein Liu 4 ,Haoming Luo 4 , Yifan Luo 4 , Zhengmao Ye 4 , Tingwen Liu 1,2 , Xin Zhao 4 , Ran Tao 4,† , Bryan Dai 4,† 1 Institute of Information Engineering, Chinese Academy of Sciences 2 School of Cyber Security, University of Chinese Academy of Sciences 3 Peking University 4 IQuest Research Code: https://github.com/Longyichen/X-gram Abstract Large token-indexed lookup tables provide a compute-decoupled scaling path, but their practical gains are often limited by poor parameter efficiency and rapid memory growth. We attribute these limitations to Zipfian under-training of the long tail, heterogeneous demand across layers, and “slot collapse” that produces redundant embeddings. To address this, we propose X-gram , a frequency-aware dynamic token-injection framework . X-gram employs hybrid hashing and alias mixing to compress the tail while preserving head capacity, and refines retrieved vectors via normalized SwiGLU ShortConv to extract diverse local n n -gram features . These signals are integrated into attention value streams and inter-layer residuals using depth-aware gating, effectively aligning static memory with dynamic context. This design introduces a memory-centric scaling axis that decouples model capacity from FLOPs. Extensive evaluations at the 0.73B and 1.15B scales show that X-gram improves average accuracy by as much as 4.4 points over the vanilla backbone and 3.2 points over strong retrieval baselines, while using substantially smaller tables in the 50% configuration . Overall, by decoupling capacity from compute through efficient memory management, X-gram offers a scalable and practical paradigm for future memory-augmented architectures. 1 Introduction Increasing model capacity in large language models usually comes with higher inference-time compute, making further scaling increasingly costly in practice. Token-indexed memory offers an alternative scaling path: instead of relying solely on dense computation, the model can retrieve token-conditioned features from a lookup table, thereby expanding static capacity without proportionally increasing compute [ 12 , 2 ] . In practice, however, prior lookup-based designs often require very large tables and substantial memory traffic, making these gains difficult to realize efficiently [ 12 , 2 ] . Figure 1 : Traditional n-gram approaches ignore token frequency, resulting in inefficient parameter allocation—most long-tail slots are under-trained, and slot representations collapse to homogeneous features. In contrast, X-gram introduces frequency-aware hybrid routing and diverse contextual feature extraction, enabling efficient parameter usage and richer context modeling. Yet, in practice, lookup-based scaling shifts the bottleneck from FLOPs to memory traffic, and naive parameter growth can be wasteful. Prior work often requires tables that are several times larger than the backbone (e.g., MoLE reports 2.4 – 7.4\times larger re-parameterized tables), while n -gram extensions can demand terabytes of external storage (e.g., 7.6 TB for 10^{9} n -gram embeddings) [ 12 , 2 ] . Such tables rarely fit in HBM; they must be placed in host memory or even SSD, and the resulting parameter traffic is difficult to fully hide behind computation, especially during decoding. More importantly, empirical gains from enlarging lookup tables often diminish rapidly, suggesting low activation efficiency and information density per byte. To make this scaling axis practical, we ask a more basic question: what does an efficient lookup table need to optimize for under finite data and bandwidth? We analyze how token-indexed parameters are trained and how their representations behave, and find two consistent regularities. First, utilization is strongly frequency-driven under Zipf-like statistics: high-frequency items dominate updates and learned activations, while most long-tail rows remain under-trained as the table grows [ 27 ] . Second, scaling capacity by adding parallel fixed slots is not sufficient: different slots are trained on highly overlapping events, encouraging redundant subspaces and rapid saturation [ 12 ] . These findings indicate that the bottleneck is not the lack of parameters, but the mismatch between table capacity, update density, and representational diversity. Motivated by this analysis, we propose X-gram , a data-aware framework for fine-grained information extraction and injection. We name it X-gram sinceit converts token-indexed 1-gram retrieval into variable-length local x-gram features via multi-scale ShortConv refinement. We factor lookup-based augmentation into three stages—vocabulary-to-memory mapping, information extraction, and injection—and design each stage to increase effective capacity without sacrificing stability or system efficiency. Specifically, X-gram makes three targeted design choices to fully exploit the parameter space of lookup memory while keeping training stable and the system offloading-friendly. First, to mitigate long-tail under-training, X-gram introduces probability-balanced hybrid hashing : it assigns dedicated capacity to high-frequency tokens and routes the long tail into shared hash buckets, mitigating hotspot concentration and improving row-level training balance under a fixed budget. Second, to prevent fixed-slot collapse and encourage diversity, X-gram adds a lightweight ShortConv refinement module that enriches static retrieved vectors with multi-scale local structure, extracting differentiated n -gram features even when the underlying tables would otherwise become redundant. Third, to match heterogeneous injection demand across depth and components, X-gram uses a mixed injection design centered on attention content pathways, especially the value stream, together with inter-layer residual injection when beneficial; the value pathway provides contextual matching with a smaller injected width, while depth-aware gating modulates injection strength across layers. Finally, guided by training dynamics, we stabilize the post-injection residual pathway and design a sparse-aware learning-rate schedule for token-indexed parameters, forming an end-to-end system that is both trainable and scalable. We evaluate X-gram with large-scale pretraining across two model scales. Under identical backbones and training budgets, X-gram consistently lowers validation perplexity, improves downstream accuracy, and substantially reduces lookup-table size. At 0.73B scale, X-gram -50% reaches 48.5 at 1 \times , beating Retoken (45.0), MoRT (45.8), and Engram (47.2) with a much smaller table, and it scales to 49.1 at 4 \times . At 1.15B scale, X-gram -50% reaches 48.2 at 1 \times and 50.8 at 4 \times , surpassing strong baselines by 2.3 points (Table 3 ). Moreover, the ShortConv n -gram extractor strengthens scaling behavior: as the lookup table grows, X-gram continues to gain, whereas baselines often plateau, indicating that additional memory capacity is converted into useful, distinct features rather than redundant slots. These results, together with our mechanistic analyses, support X-gram as a practical lookup-based scaling axis that is trainable under Zipf-like statistics and diversity-preserving under slot scaling. 2 General Framework Generic Injection Framework Given a tokenizer \tau , an input sequence is tokenized into token IDs \mathbf{x}=(x_{1},\ldots,x_{T})\in V^{T} , with embedding matrix \mathbf{E}\in\mathbb{R}^{|V|\times d} . A Transformer layer maps hidden states \mathbf{H}_{\ell-1}\in\mathbb{R}^{T\times d} to \mathbf{H}_{\ell} through self-attention and MLP blocks. We factor lookup-based injection into three operators: (i) token-to-parameter mapping \mathcal{R}_{\phi} , (ii) information extraction \mathcal{T}_{\psi} , and (iii) injection \mathcal{I}_{s} . Given token IDs \mathbf{x} , each view m retrieves a sequence \mathbf{E}_{\ell}^{(m)}=\mathcal{R}_{\phi}^{(m)}(\mathbf{x};\mathbf{B}_{\ell}^{(m)}), (1) where \mathbf{B}_{\ell}^{(m)}\in\mathbb{R}^{S\times d_{s}} is a (possibly compressed) lookup table and \phi parameterizes the routing from tokens (or n -grams) to physical rows. An extractor produces refined features \tilde{\mathbf{E}}_{\ell}^{(m)}=\mathcal{T}_{\psi}^{(m)}(\mathbf{E}_{\ell}^{(m)}), (2) and we fuse views to form a per-layer injection \Delta_{\ell}=\sum_{m=1}^{M_{\ell}}g_{\ell}^{(m)}\,\tilde{\mathbf{E}}_{\ell}^{(m)}, (3) Here, a view denotes an independent retrieval-and-extraction branch with dedicated parameters, while a site denotes the target pathway where the fused signal is injected. The fused signal is injected into a chosen site s by \mathcal{I}_{s} . We