\jbsans
Mellum 2 · 2026
Read \jbsans by Mellum 2 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31268.txt.
Introduction
Kojic and colleagues present Mellum 2, an open-weight language model built for software engineering. The 12B-parameter mixture-of-experts design activates a small share of its weights per token, spanning code generation, editing, debugging, tool use and agentic coding. Pretraining covers about 10.6 trillion tokens in three phases shifting toward code and maths, followed by fine-tuning and reinforcement stages that yield Instruct and Thinking variants. Benchmarks place it with open-weight rivals in the 4B to 14B range at the per-token cost of a small dense model.
Catalog record
| CREATOR | Mellum 2 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86966 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.31268
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
\jbsans Mellum 2 Abstract Correspondence: [email protected] Released under the Apache 2.0 license. 1 Introduction Large language models (LLMs) have reshaped how developers work with code. What began as inline autocomplete [ 40 ] has broadened into a much wider task surface: writing whole functions from natural-language specifications, editing and debugging existing code, reasoning through multi-step engineering tasks, calling external tools, navigating repositories as an agent, and serving as a conversational collaborator throughout the development loop. The competitive coding models today must do all of this at once, and at a serving cost that makes them practical to deploy at scale. Two regimes dominate the open-weights landscape on the quality-versus-cost trade-off. Dense models in the 4–14B range are cheap to serve but plateau on harder coding and reasoning workloads; very large Mixture-of-Experts (MoE) models [ 18 , 14 ] reach frontier quality but at deployment costs that strain everyday use. To strike a balance between knowledge scope and serving cost, we aim to extend the recent line of small MoE coding models—among them Qwen3-Coder-30B-A3B [ 74 ] and Ling-Coder-Lite [ 13 ] : sufficient parameters to absorb the long tail of programming language and reasoning knowledge but with enough sparsity to allow for deployment on commodity hardware (per-token compute in the 2–3B-dense range). We introduce Mellum 2 , an open-weight 12B-parameter MoE language model with 2.5B active parameters per token, a general-purpose successor to Mellum [ 55 ] — the 4B dense code-completion model previously deployed in JetBrains IDEs. While the original Mellum was trained to fill single completions inside an editor, Mellum 2 is a full-fledged coding assistant: it generates and edits code, calls tools, plans and executes multi-step agentic workflows, holds long conversations about code, and, in its thinking variant, produces explicit reasoning traces before answering. The model is built on the Qwen3-MoE recipe [ 74 ] (64 experts, 8 active) with three deployment-oriented modifications: Grouped-Query Attention [ 1 ] with only 4 KV heads, Sliding Window Attention [ 6 ] on three of every four layers, and a single Multi-Token Prediction (MTP) [ 22 ] head that is used both as an auxiliary pre-training objective and as a built-in draft for speculative decoding. Our key contributions are: • An efficiency-aware architecture. We systematically ablate dense versus MoE backbones, Grouped-Query Attention configurations, Multi-head Latent Attention [ 15 ] , Sliding Window Attention patterns, and expert sparsity ratios. The resulting 12B/2.5B-active configuration matches or exceeds the throughput of a 7B dense baseline while occupying a substantially larger total-parameter envelope. • A three-phase pre-training curriculum on {\sim} 10.6T tokens. Following the “web early, curated late” paradigm [ 23 ] , the data mixture progressively shifts from diverse web content toward curated code and mathematical content (code ratio 23 % \to 42 % \to 59 %), with batch-size doubling and an extended capability-sharpening phase that decays the learning rate linearly to zero. • A Muon + FP8 training recipe at production scale. We adopt the Muon optimizer [ 34 , 45 ] for large-scale MoE pre-training, combine it with FP8 hybrid mixed precision [ 49 ] and a Warmup-Hold-Decay schedule [ 30 , 25 ] with linear decay to zero, and report training-stability observations across the full ten-trillion-token run. • Long-context extension to 128K. We extend the pre-trained base to 131,072 tokens following the layer-selective scaling recipe of Gemma 3 [ 21 ] and OLMo 3 [ 71 ] with YaRN [ 57 ] as the scaling method, and report empirical findings on data-mix transfer and MoE router dynamics during this stage. • Two post-trained variants from a shared base. From the same long-context checkpoint we produce an Instruct model that answers directly and a Thinking model that emits an explicit reasoning trace, each refined further by reinforcement learning with verifiable rewards (RLVR) on math and executable coding tasks. • Open release. We release base, instruct, and thinking checkpoints under the Apache 2.0 license, together with this report documenting the architecture decisions, data pipeline, and training recipe behind them. In addition, we release a base model before the long context extension and SFT checkpoints. Across a panel of code generation, math and reasoning, tool use, knowledge, and safety benchmarks, Mellum 2 is competitive with open-weight baselines in the 4–14B range despite running at the per-token compute of a 2.5B dense model, and matches or exceeds the inference throughput of Qwen2.5-7B [ 58 ] on a single H100. The remainder of this report follows the contributions above: Section 2 traces the architecture design and ablations, Section 3 details the pre-training data and recipe, Section 4 describes the 128K context extension, Section 5 covers SFT, RL, and post-training evaluation, and Section 6 reports our inference benchmarks. 2 Model Architecture Mellum 2 is a decoder-only Transformer that closely follows the Qwen3-MoE recipe [ 74 ] : a Mixture-of-Experts (MoE) feed-forward network in every layer, Grouped-Query Attention (GQA) [ 1 ] with QK-Norm [ 28 ] , SiLU-gated MLPs [ 63 ] , RMSNorm [ 78 ] , and Rotary Position Embeddings (RoPE) [ 68 ] . On top of this backbone we add two latency- and quality-oriented modifications: Sliding Window Attention (SWA) on a fraction of the layers, and a single Multi-Token Prediction (MTP) head trained as an auxiliary objective. 2.1 Architecture Design Decisions As Mellum 2 is meant to be deployed in JetBrains IDEs, we approached the design space from the perspective of efficient inference. We targeted the latency and throughput budget of a Qwen2.5-7B dense model on a single H100 GPU as our baseline and conducted several architectural ablations to match it. 2.1.1 Dense vs. Sparse We first evaluated whether a dense architecture could outperform the baseline under our latency constraint. We explored multiple Qwen3-based dense configurations—varying depth (24–40 layers) and width (hidden sizes 2304–4096), as well as DeepSeek-style models with Multi-head Latent Attention (MLA) [ 15 ] . None of the dense configurations consistently outperformed Qwen2.5-7B on our evaluation benchmarks within the latency budget. MLA allowed scaling to approximately 5.5B parameters at equivalent speed, but the quality gains were insufficient to justify the additional training complexity, and the supported latent rank was too large for our model scale. We therefore adopted a Mixture-of-Experts (MoE) architecture, which enabled scaling to {\sim} 12B total parameters while keeping the per-token compute comparable to a 2.5B dense model. 2.1.2 Expert Configuration Starting from the Qwen3-30B-A3B architecture [ 74 ] , we scaled down the model proportionally to fit within a single H100 GPU ( < 18B total parameters). We fixed the number of experts at 64 as larger expert counts exceeded GPU memory constraints. We evaluated different sparsity levels (number of active experts) and found that higher sparsity (fewer active experts) yielded better inference performance. For example, 2 active experts achieved {\sim} 1.5 \times lower latency than 8 active experts. However, consistent with prior work suggesting that high sparsity can be detrimental at smaller scales [ 10 , 35 ] , our benchmark evaluations confirmed that models with lower sparsity (more active experts) produced better quality. We settled on 8 active out of 64 total experts as the optimal quality–latency trade-off. Under this configuration, the model supports up to {\sim} 15B total parameters while matching Qwen2.5-7B