IMU-1: Sample-Efficient Pre-training of Small Language Models

George Grigorev · 2026

Read IMU-1: Sample-Efficient Pre-training of Small Language Models by George Grigorev (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02522.txt.

Description

George Grigorev presents a small language model trained to punch far above its data budget, in a 2026 arXiv paper. IMU-1 has four hundred thirty million parameters and trained on seventy-two billion tokens. It approaches the benchmark performance of models trained on fifty-six times more data.

The recipe combines recent architectural interventions, including normalized query-key attention, per-head gating, value residuals, and scaled normalization layers. Optimization advances include a newer optimizer with cautious weight decay and a parametrization scheme that transfers settings across scales. Training runs in three stages, followed by averaging checkpoints after the fact. Ablations validate each component separately. Grigorev releases the code, weights, and data for reproduction. The paper is a practical argument that careful recipes, not just scale, drive sample efficiency.

Catalog record

CREATORGeorge Grigorev
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-02522

Browse the full library · About the Tower of Babel