xGen-small Technical Report

Erik Nijkamp, Bo Pang, Egor Pakhomov, Akash Gokul, Jin Qu, Silvio Savarese, Yingbo Zhou, Caiming Xiong · 2025

Read xGen-small Technical Report by Erik Nijkamp, Bo Pang, Egor Pakhomov, Akash Gokul, Jin Qu, Silvio Savarese, Yingbo Zhou, Caiming Xiong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06496.txt.

Introduction

Erik Nijkamp and seven co-authors introduce xGen-small, a family of compact language models built for long documents. Their technical report came out in 2025. The family has two sizes, each a transformer decoder, the standard architecture behind modern chat models.

Their pipeline unites several stages in one system. Training data is balanced across domains and filtered with frequency awareness. Pre-training runs in stages, with quality annealing and context length extended to a very long window. Targeted post-training then adds supervised fine-tuning, preference learning, and online reinforcement learning. The result is strong performance across many tasks, especially mathematics and coding. Long-context benchmarks, where the model must use information spread over a huge input, are a particular strength.

Catalog record

CREATORErik Nijkamp, Bo Pang, Egor Pakhomov, Akash Gokul, Jin Qu, Silvio Savarese, Yingbo Zhou, Caiming Xiong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-06496

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

xGen-small Technical Report Erik Nijkamp, Bo Pang, Egor Pakhomov, Akash Gokul, Jin Qu, Silvio Savarese, Yingbo Zhou, Caiming Xiong Abstract We introduce xGen-small , a family of 4B and 9B Transformer decoder models optimized for long - context applications. Our vertically integrated pipeline unites domain - balanced, frequency - aware data curation; multi - stage pre - training with quality annealing and length extension to 128k tokens; and targeted post - training via supervised fine - tuning, preference learning, and online reinforcement learning. xGen-small delivers strong performance across various tasks, especially in math and coding domains, while excelling at long context benchmarks. Figure 1: Summary of xGen model performance . We introduce the xGen-small series of language models which are available in two sizes – xGen-small-4B and xGen-small-9B. These open-source models provide strong performance across a variety of tasks and scale up to 128K context length. 1 Introduction Deep learning–based language models have rapidly transformed natural language processing and related fields by scaling up model size and training data. However, in enterprise settings, deploying large-scale models with hundreds of billions of parameters presents significant challenges: high latency, unpredictable serving costs, elevated energy consumption, and increased risk of exposing proprietary or sensitive data. Equally important, many enterprise use cases require processing long documents, multi-page reports, or extended conversational histories in a single pass, a capability often constrained by fixed context windows in existing LLMs. To address these constraints, we introduce xGen-small , a family of compact Transformer decoder [ 1 ] models available in two parameter scales — 4B and 9B — designed and optimized for both cost-efficient deployment and true long-context comprehension. Instead of pursuing ever-larger architectures, xGen-small leverages a holistic, vertically integrated pipeline that unifies: (1) Domain-balanced, frequency-aware data curation : rigorous filtering, deduplication, and targeted up-sampling across diverse web and domain-specific corpora to ensure both breadth and depth of high-quality content. (2) Pre-training with distributional sharpening : progressive training stages that shift from broad content coverage to high-quality subsets, sharpening the data distribution over time to improve performance across diverse tasks under aligned learning rate regimes. (3) Context length extension : a two-stage methodology to extend model context from 4K to 128K tokens, utilizing tuned Rotary Position Embeddings, cross-document attention masking, and sequence parallelism to maintain training efficiency at extreme lengths. (4) Targeted post-training : supervised fine-tuning with mixed instruction datasets, off-policy preference alignment, and online reinforcement learning via Group Relative Policy Optimization (GRPO) [ 2 ] to enhance alignment, reasoning rigor, and domain expertise. (5) Comprehensive evaluation : rigorous benchmarking across general reasoning, mathematics, coding, and long-context suites — including MMLU, GSM8K, HumanEval, and RULER — to demonstrate that xGen-small matches or outperforms similarly-sized models while retaining cost-effective serving characteristics. In summary, xGen-small shows that compact language models, when engineered with specialized data pipelines, efficient training strategies, dedicated length extension techniques, and targeted alignment procedures, can deliver state-of-the-art performance on both standard and long-context benchmarks without the operational overhead associated with larger models. The remainder of this report is organized as follows: Section 2 describes our data acquisition and curation pipeline; Section 3 details the pre-training methodology and the context length extension approach; Section 4 outlines the post-training alignment procedures; Section 5 covers evaluation results; and Section 6 concludes with discussion. 2 Data-Curation High - quality, domain - balanced pre - training data may be the single most important driver of model capability. We therefore combine large - scale raw web data with rigorous filtering, deduplication, and targeted up-sampling to produce a data distribution that is both diverse and highly informative for enterprise use cases. 2.1 Acquisition We follow the common practices for data processing pipelines: Extract documents from commonly available text corpora and split the cleaned text into self-contained documents. Each document is stored with only essential metadata (URL, crawl time, language, hash), forming a “raw-prepared” layer that downstream quality filtering and deduplication consume. In addition to common datasets, we incorporate relevant documents from open datasets with high concentrations of domain-specific content, such as code and math, to ensure broader and deeper domain coverage. 2.2 Deduplication Deduplication is an essential step in preparing the data. We take a slightly different approach compared to prior work. Previous studies consistently show that removing large clusters of duplicate or near-duplicate documents improves model performance by reducing memorization and increasing exposure to diverse content [ 3 , 4 ] . Yet follow - up studies warn that pushing deduplication too far can degrade quality, probably because it strips away meaningful signals embedded in the natural frequency of documents. Although the mechanics differ — some pipelines halt deduplication part - way (e.g., DCLM caps each document at roughly one copy per shard) [ 5 , 6 ] , while others keep a single canonical copy per duplicate cluster and then re - hydrate the corpus by repeating that copy according to a coarse, bucketed log - like recipe — often capping or even down - weighting very large clusters [ 7 ] — the common outcome is a sub - linear reweighting of raw frequencies, often approximating (or bounded by) a log - scaled function rather than preserving a strict linear count. Many real - world pipelines therefore finish deduplication in a deliberately “incomplete” state: fuzzy matching prunes the bulk of exact copies, yet enough near - duplicates remain to implicitly up - sample documents that appear more often in the wild. While this strategy can boost benchmark scores—higher - frequency documents are often higher quality, as our own ablations confirm even though formal evidence in the literature is limited — it also bakes a crucial sampling decision into preprocessing. As a result, the true impact of frequency is hard to disentangle in ablation studies. We instead record natural occurrences in metadata and treat it as one quality signal among many, rather than allowing it to covertly steer dataset composition. This practice may stem from the incentive structure around releasing CommonCrawl-based datasets: new datasets are often evaluated by comparing their performance on established benchmarks, and pre-baked frequency up-sampling can produce superficially improved results. We argue that natural frequency should be explicitly tracked and used at sampling time, not silently embedded during pre-processing. Additionally, standard deduplication pipelines discard all fuzzy duplicates. If such a document is later selected for up-sampling due to high quality, we are forced to replicate a single canonical version—missing an opportunity to use alternate variants that could enrich lexical diversity. To address this, our pipeline retains the top-k fuzzy duplicates per cluster, enabling more diverse document selection during sampling. We further enrich our deduplication metadata by saving frequency signals such as the number of times a document appears across Common Crawl snapshots (indicating longevity) and across different domains (indicating breadth). Additional signals are also extracted and retained in metadata, though they are beyond the scope of this report. These are preserved alongside raw occurrence counts to inform later sampling decisions on equal footing with classifier scores and domain tags. 2.3 Quality Filtering We use lightweight heuristics to discard absolutely low-quality documents not useful for training. We consider two groups of quality signals to determine which documents are retained: (1) Document natural occurrences, as described in the deduplication section,

Read the complete text