AYLA: Architecting a loss landscape in shallow neural networks to accelerate feature recovery
Behnam Gheshlaghi, Shahin Atakishiyev · 2025
Read AYLA: Architecting a loss landscape in shallow neural networks to accelerate feature recovery by Behnam Gheshlaghi, Shahin Atakishiyev (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01875.txt.
Description
Shallow neural networks learn features in fits and starts, stalling on flat stretches of their loss landscape before suddenly improving. Behnam Gheshlaghi and Shahin Atakishiyev introduce AYLA in a 2025 paper, a method that reshapes that landscape while training runs. It does not move the best solution or any resting point of the original problem.
AYLA bends the loss value through a smooth, sigmoid-gated power law, which acts like a learning rate that adapts to the current state. Descent speeds up across flat and saddle regions, then settles for the final approach. In teacher-student tests with two-layer networks on synthetic data, feature recovery improved across every measure the team tracked: weight alignment, neuron similarity, activation match, and spectrum. The gains held across gradient methods and exponent schedules.
Catalog record
| CREATOR | Behnam Gheshlaghi, Shahin Atakishiyev |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-01875