Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging

Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade · 2026

Read Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging by Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03702.txt.

Description

Nobody knows in advance how long a language model will keep training, yet most learning-rate schedules assume a fixed finish line. Alexandru Meterez and 4 co-authors study anytime pretraining in a 2026 paper, where the schedule must work at every stopping point. Their theory starts with overparameterized linear regression and isolates the role of weight averaging.

They show anytime schedules should decay polynomially with time, at a rate set by the problem's source and capacity conditions. Weight averaging, also called model merging, is what lets such schedules reach the best possible convergence rates. Empirically, constant and inverse-square-root schedules with averaging match a tuned cosine schedule. The test models reach 300 million parameters, trained far past the usual data scale.

Catalog record

CREATORAlexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-03702

Browse the full library · About the Tower of Babel