A Trainable Optimizer
Ruiqi Wang Diego Klabjan · 2025
Read A Trainable Optimizer by Ruiqi Wang Diego Klabjan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01764.txt.
Description
Wang and Klabjan present a 2025 paper on optimizers that learn how to optimize. Standard training leans on hand-built rules like ADAM to turn gradients into steps. Their framework trains the gradient estimator itself, alongside the model weights. A simplified linear version, they prove, matches the classic SGD method's convergence speed while cutting noise.
The cheap version adds almost no compute, needing only a few extra tensor products. Two leaner variants trim cost further. Across test problems, convex and not, the trainable optimizers converged faster than ADAM and peers. Even fine-tuning a large language model sped up. The authors pitch learned optimizers as a drop-in upgrade.
Catalog record
| CREATOR | Ruiqi Wang Diego Klabjan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01764