Fast Compute for ML Optimization
Nick Polson, Vadim Sokolov · 2026
Read Fast Compute for ML Optimization by Nick Polson, Vadim Sokolov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-14280.txt.
Description
Nick Polson and one co-author present a 2026 arXiv optimizer for losses with a variance-mean scale-mixture form. Under that form, each EM step becomes a weighted least-squares update, with latent variables setting observation and parameter weights. Those weights play roles like Adam's second-moment scaling and AdamW's weight decay, but they arise from the model itself.
The resulting Scale Mixture EM needs no user-set learning-rate or momentum schedules. On ill-conditioned logistic regression benchmarks with 20 to 500 parameters, the Nesterov-accelerated version reached up to 13 times lower final loss than grid-searched Adam. Sharing sufficient statistics across a 40-point regularization path cut runtime tenfold versus the same Adam protocol, though Nesterov extrapolation trades EM monotonicity for speed.
Catalog record
| CREATOR | Nick Polson, Vadim Sokolov |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-14280