Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descent

Behrad Moniri, Hamed Hassani · 2026

Read Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descent by Behrad Moniri, Hamed Hassani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17767.txt.

Description

Behrad Moniri and Hamed Hassani study feature learning in two-layer nets where width, sample size, and input size grow in proportion. Earlier work tracked one gradient step on first-layer weights, but that update is near rank-one and captures just one direction. The pair characterize the second step: weights behave like a spiked random matrix with outliers, each marking a learned direction. Reuse beats fresh batches: reuse lets the second step learn directions with information exponent above one, while fresh batches restrict it to exponent one.

Catalog record

CREATORBehrad Moniri, Hamed Hassani
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86564
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17767

Browse the full library · About the Tower of Babel