Celo2: Towards Learned Optimization Free Lunch

Abhinav Moudgil, Boris Knyazev, Eugene Belilovsky · 2026

Read Celo2: Towards Learned Optimization Free Lunch by Abhinav Moudgil, Boris Knyazev, Eugene Belilovsky (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-19142.txt.

Description

Learned optimizers promise much but generalise poorly. Abhinav Moudgil, with two co-authors, makes them practical in 2026 with Celo2. Hand-designed rules like Adam still dominate. Learned alternatives often fail beyond their training distribution and cost dearly to meta-train. One prior system burned enormous compute yet stalled below six hundred million parameters.

A simple normalised architecture plus augmented meta-training changes the picture. A performant general-purpose learned rule emerges from a tiny fraction of that compute, in just hours. It scales stably to a billion-scale pretraining task, vastly larger than its meta-training distribution. Strong performance holds across diverse out-of-distribution tasks. Compatibility covers modern harnesses, including orthogonalisation, distinct input-output and hidden rules, and decoupled weight decay. Learnable optimisation thus becomes practically explorable.

Catalog record

CREATORAbhinav Moudgil, Boris Knyazev, Eugene Belilovsky
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-19142

Browse the full library · About the Tower of Babel