Faster Adaptive Optimization via Expected Gradient Outer Product Reparameterization

Adela DePavia, Jose Cruzado, Jiayou Liang, Vasileios Charisopoulos, Rebecca Willett · 2025

Read Faster Adaptive Optimization via Expected Gradient Outer Product Reparameterization by Adela DePavia, Jose Cruzado, Jiayou Liang, Vasileios Charisopoulos, Rebecca Willett (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-01594.txt.

Description

Adela DePavia and co-authors worked in 2025 on a quiet weakness of popular training methods for machine learning. Methods such as Adam adjust each step by past slopes, but they react to how the problem is framed. A mere change of axes can speed or slow them a great deal. No clear rule said which frame is best. The team put forth a way to pick that frame from the slopes themselves.

Their frame turns the axes to match a matrix built from the mean outer product of slopes. They show that how fast that matrix's values decay sets how much the frame matters. Common tasks with natural data show that decay, they argue. Tests back the gain in speed from the new frame.

Catalog record

CREATORAdela DePavia, Jose Cruzado, Jiayou Liang, Vasileios Charisopoulos, Rebecca Willett
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE186 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-01594

Browse the full library · About the Tower of Babel