Celo2: Towards Learned Optimization Free Lunch
Abhinav Moudgil, Boris Knyazev, Eugene Belilovsky · 2026
Read Celo2: Towards Learned Optimization Free Lunch by Abhinav Moudgil, Boris Knyazev, Eugene Belilovsky (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-19142.txt.
Description
Learned optimizers promise much but generalise poorly. Abhinav Moudgil, with two co-authors, makes them practical in 2026 with Celo2. Hand-designed rules like Adam still dominate. Learned alternatives often fail beyond their training distribution and cost dearly to meta-train. One prior system burned enormous compute yet stalled below six hundred million parameters.
A simple normalised architecture plus augmented meta-training changes the picture. A performant general-purpose learned rule emerges from a tiny fraction of that compute, in just hours. It scales stably to a billion-scale pretraining task, vastly larger than its meta-training distribution. Strong performance holds across diverse out-of-distribution tasks. Compatibility covers modern harnesses, including orthogonalisation, distinct input-output and hidden rules, and decoupled weight decay. Learnable optimisation thus becomes practically explorable.
Catalog record
| CREATOR | Abhinav Moudgil, Boris Knyazev, Eugene Belilovsky |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-19142