STEM: Scaling Transformers with Embedding Modules
Ranajoy Sadhukhan, Sheng Cao, Harry Dong, Changsheng Zhao, Attiano Purpura-Pontoniere, Yuandong Tian, Zechun Liu *, Beidi Chen * · 2026
Read STEM: Scaling Transformers with Embedding Modules by Ranajoy Sadhukhan, Sheng Cao, Harry Dong, Changsheng Zhao, Attiano Purpura-Pontoniere, Yuandong Tian, Zechun Liu *, Beidi Chen * (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10639.txt.
Introduction
Ranajoy Sadhukhan and co-authors present STEM, a way to scale transformer models with less compute per token. Sparse designs promise more capacity without a matching rise in cost. In practice they often suffer unstable training, poor load balance, and heavy traffic between chips. STEM takes a static, token-indexed route instead.
The method swaps the up-projection in each feed-forward block for a lookup in a local embedding table. The gate and down-projection stay dense, as before. Lookups are cheap, so capacity can grow while token cost stays low. The design targets teams that train large language models and want size without runaway bills.
Catalog record
| CREATOR | Ranajoy Sadhukhan, Sheng Cao, Harry Dong, Changsheng Zhao, Attiano Purpura-Pontoniere, Yuandong Tian, Zechun Liu *, Beidi Chen * |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-10639
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
STEM: Scaling Transformers with Embedding Modules Ranajoy Sadhukhan, Sheng Cao, Harry Dong, Changsheng Zhao, Attiano Purpura-Pontoniere, Yuandong Tian, Zechun Liu *, Beidi Chen * Abstract Fine-grained sparsity promises higher parametric capacity without proportional per-token compute, but often suffers from training instability, load balancing, and communication overhead. We introduce STEM ( Scaling Transformers with Embedding Modules ), a static, token-indexed approach that replaces the FFN up-projection with a layer-local embedding lookup while keeping the gate and down-projection dense. This removes runtime routing, enables CPU offload with asynchronous prefetch, and decouples capacity from both per-token FLOPs and cross-device communication. Empirically, STEM trains stably despite extreme sparsity. It improves downstream performance over dense baselines while reducing per-token FLOPs and parameter accesses (eliminating roughly one-third of FFN parameters). STEM learns embedding spaces with large angular spread which enhances its knowledge storage capacity . More interestingly, this enhanced knowledge capacity comes with better interpretability . The token-indexed nature of STEM embeddings allows simple ways to perform knowledge editing and knowledge injection in an interpretable manner without any intervention in the input text or additional computation. In addition, STEM strengthens long-context performance: as sequence length grows, more distinct parameters are activated, yielding practical test-time capacity scaling. Across 350M and 1B model scales, STEM delivers up to ∼ \sim 3–4% accuracy improvements overall, with notable gains on knowledge and reasoning-heavy benchmarks (ARC-Challenge, OpenBookQA, GSM8K, MMLU). Overall, STEM is an effective way of scaling parametric memory while providing better interpretability , better training stability and improved efficiency . †]Carnegie Mellon University §]Meta AI \contribution [‡]Work done during internship at Meta AI \contribution [¶]Work done at Meta AI \contribution [*]Co-supervisors \contact \contact \contact \contact \contact \contact \contact \contact \metadata [Github] https://github.com/Infini-AI-Lab/STEM \metadata [Website] https://infini-ai-lab.github.io/STEM 1 Introduction (a) (b) (c) Figure 1 : (a) Validation PPL vs. training tokens for 1B STEM vs. dense; (b) Needle-in-a-Haystack at 8k/16k/32k; (c) STEM layer: embedding tables offloaded to CPU and token-indexed ones are prefetched to GPU. Sparse computation is a key mechanism for realizing the benefits predicted by parameter-scaling laws ( 26 ; 21 ) without proportionally increasing per-token compute. In particular, Mixture-of-Experts (MoE) ( 43 ; 1 ; 14 ) models have been adopted in several frontier LLMs ( 47 ; 48 ; 10 ) because they raise parametric capacity at roughly constant activated FLOPs by sparsely activating a small subset of experts per token. Recent work ( 5 ; 19 ; 11 ; 10 ) further advocate for finer-grained sparsity that employs large number of micro-experts to achieve better expressivity, enhanced knowledge storing capacity, and favorable efficiency metrics. However, finer granularity introduces nontrivial challenges in both optimization and systems. On the training side, even large fraction of experts can remain under-trained ( 23 ) due to a highly non-uniform routing and result in training instability. While load-balancing objectives ( 43 ; 14 ; 28 ) can address these issues, they may interfere with the primary objective if not carefully tuned ( 10 ; 38 ; 17 ) . On the systems side, increasing the number of experts typically raises the number of all-to-all messages while shrinking message sizes, degrading bandwidth utilization and amplifying communication overhead ( 22 ; 30 ) . Finer granularity can also reduce parameter-access locality and degrade kernel efficiency when expert subnetworks become too small for dense linear-algebra kernels to reach high occupancy, yielding suboptimal end-to-end performance. Finally, these large-scale fine-grained sparse networks are far from interpretable. It is difficult to understand the roles of each micro-expert. To harness the full potential of fine-grained sparsity, we require: (a) stable optimization , (b) broad expert utilization (each micro-expert learns useful representations), and (c) negligible expert-retrieval latency and communication overhead . Additionally, we would like our sparse architecture to be (e) more interpretable and ensure that we are using all the micro-experts in a more transparent manner. We identify static sparsity as a potential solution to achieve these desired properties. Static sparsity keeps the compute path predictable (no runtime routing latency), enables prefetch and CPU offloading (removing the need for inter-node communication). Recently, static sparsity via token-indexed routing has emerged as a promising direction ( 40 ; 18 ) with strong performance guarantees. Additionally, the token-indexed nature makes this static sparsity more interpretable as each micro-expert can correspond to a given token ID. However, such token-based selection strategy lacks context adaptivity. If applied naively, it can reduce the expressivity of the model and degrade quality despite more parameters. Our ablation study in sec. 4.4.2 highlights the criticality of selecting the suitable module for sparsification. Based on these observations, we introduce STEM , a static, token-indexed, fine-grained mechanism that replaces only the up-projection in gated FFNs with a token-specific vector retrieved from a layer-local embedding table. The gating and down-projection paths are preserved and shared across tokens. We observe that STEM achieves, Better Training Stability: Despite being extremely sparse, STEM does not exhibit any training instability issues as usually seen in MoE models. Figure 5(a) shows that unlike MoE models, STEM does not exhibit any loss spikes. Improved Performance with Larger Knowledge Capacity: STEM learns a representation space for the embeddings that is conducive to better information storage. The learned embeddings exhibit a large angular spread (i.e., low pairwise cosine similarity), which reduces representational interference and improves addressability of the parametric memory. As a result, it effectively increases the distinct “slots” available for storing and retrieving information. In our downstream evaluation benchmark, STEM consistently outperforms the dense baseline on knowledge-intensive tasks like, ARC-Challenge ( 8 ) , and OpenBookQA ( 35 ) by large margins ( \sim 9–10%) and the improvement usually increases with more STEM layer inclusion. Interpretability features: Because each STEM embedding in every layer is tied to a specific token ID, individual “micro-experts” have clear, token-level semantics. This structure not only makes their role more interpretable, but also gives STEM models a surprising degree of controllability: by simply swapping the STEM table index (illustrated in Fig. 3 ) while leaving the input text unchanged, we can systematically steer the model’s output distribution. Such interventions highlight how much factual knowledge is localized in these embeddings and how modular, editable, and attributable that knowledge becomes. Improved Long-context Inference: During long-context inference, STEM activates more distinct parameters as sequence length grows, yielding test-time capacity scaling. As shown in Figure 1(b) , the benefits strengthen with context: on Needle-in-a-Haystack (NIAH) ( 25 ) , the gap over the dense baseline increases from 8.4% to 13%. Training and Inference-time efficiency: STEM reduces both FLOPs as well as parameter loading cost by eliminating one-third of the parameters in FFN layers. Consequently, it is strictly more efficient during both computation-intensive training and prefilling, as well as in memory-intensive decoding. We benchmark STEM against the dense baseline with 350M MobileLLM ( 31 ) and Llama3.2-1B ( 34 ) model variants. Additionally, we compare with Hash Layer MoEs with the same total parameter count . We report results on standard downstream suites across pretraining, mid-training, and context-length extension. STEM improves downstream accuracy by up to \sim 3–4% while reducing per-token FLOPs and parameter accesses by up to one-third. It also strengthens knowledge retrieval and mathematical reasoning, with gains on GSM8K ( 9 ) and MMLU ( 20