Less is More: Convergence Benefits of Fewer Data Weight Updates over Longer Horizon
Rudrajit Das, Neel Patel, Meisam Razaviyayn, Vahab Mirrokni · 2026
Read Less is More: Convergence Benefits of Fewer Data Weight Updates over Longer Horizon by Rudrajit Das, Neel Patel, Meisam Razaviyayn, Vahab Mirrokni (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-19510.txt.
Description
Data mixing means strategically reweighting training domains, and it is vital for robust machine learning models. Rudrajit Das and three co-authors analyze it as a bilevel optimization problem. An outer loop tunes domain weights to minimize validation loss, while an inner loop tunes model parameters to minimize weighted training loss. Classical bilevel methods need hypergradients, which in theory require the inner loop to converge.
State-of-the-art methods instead run a finite, often small, number of inner steps before updating weights, and the theory of that shortcut was unclear. The paper proves the greedy approach of a single inner step can fail even in a plain quadratic example. Under a fixed budget of parameter updates, the analysis clarifies when fewer weight updates over a longer horizon converge better, justifying the less-is-more title.
Catalog record
| CREATOR | Rudrajit Das, Neel Patel, Meisam Razaviyayn, Vahab Mirrokni |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 127 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-19510