A Trainable Optimizer

Ruiqi Wang Diego Klabjan · 2025

Read A Trainable Optimizer by Ruiqi Wang Diego Klabjan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01764.txt.

Description

Wang and Klabjan present a 2025 paper on optimizers that learn how to optimize. Standard training leans on hand-built rules like ADAM to turn gradients into steps. Their framework trains the gradient estimator itself, alongside the model weights. A simplified linear version, they prove, matches the classic SGD method's convergence speed while cutting noise.

The cheap version adds almost no compute, needing only a few extra tensor products. Two leaner variants trim cost further. Across test problems, convex and not, the trainable optimizers converged faster than ADAM and peers. Even fine-tuning a large language model sped up. The authors pitch learned optimizers as a drop-in upgrade.

Catalog record

CREATORRuiqi Wang Diego Klabjan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-01764

Browse the full library · About the Tower of Babel