Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging
Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade · 2026
Read Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging by Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03702.txt.
Description
Nobody knows in advance how long a language model will keep training, yet most learning-rate schedules assume a fixed finish line. Alexandru Meterez and 4 co-authors study anytime pretraining in a 2026 paper, where the schedule must work at every stopping point. Their theory starts with overparameterized linear regression and isolates the role of weight averaging.
They show anytime schedules should decay polynomially with time, at a rate set by the problem's source and capacity conditions. Weight averaging, also called model merging, is what lets such schedules reach the best possible convergence rates. Empirically, constant and inverse-square-root schedules with averaging match a tuned cosine schedule. The test models reach 300 million parameters, trained far past the usual data scale.
Catalog record
| CREATOR | Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-03702