Fast Compute for ML Optimization

Nick Polson, Vadim Sokolov · 2026

Read Fast Compute for ML Optimization by Nick Polson, Vadim Sokolov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-14280.txt.

Description

Nick Polson and one co-author present a 2026 arXiv optimizer for losses with a variance-mean scale-mixture form. Under that form, each EM step becomes a weighted least-squares update, with latent variables setting observation and parameter weights. Those weights play roles like Adam's second-moment scaling and AdamW's weight decay, but they arise from the model itself.

The resulting Scale Mixture EM needs no user-set learning-rate or momentum schedules. On ill-conditioned logistic regression benchmarks with 20 to 500 parameters, the Nesterov-accelerated version reached up to 13 times lower final loss than grid-searched Adam. Sharing sufficient statistics across a 40-point regularization path cut runtime tenfold versus the same Adam protocol, though Nesterov extrapolation trades EM monotonicity for speed.

Catalog record

CREATORNick Polson, Vadim Sokolov
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-14280

Browse the full library · About the Tower of Babel