AYLA: Architecting a loss landscape in shallow neural networks to accelerate feature recovery

Behnam Gheshlaghi, Shahin Atakishiyev · 2025

Read AYLA: Architecting a loss landscape in shallow neural networks to accelerate feature recovery by Behnam Gheshlaghi, Shahin Atakishiyev (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01875.txt.

Description

Shallow neural networks learn features in fits and starts, stalling on flat stretches of their loss landscape before suddenly improving. Behnam Gheshlaghi and Shahin Atakishiyev introduce AYLA in a 2025 paper, a method that reshapes that landscape while training runs. It does not move the best solution or any resting point of the original problem.

AYLA bends the loss value through a smooth, sigmoid-gated power law, which acts like a learning rate that adapts to the current state. Descent speeds up across flat and saddle regions, then settles for the final approach. In teacher-student tests with two-layer networks on synthetic data, feature recovery improved across every measure the team tracked: weight alignment, neuron similarity, activation match, and spectrum. The gains held across gradient methods and exponent schedules.

Catalog record

CREATORBehnam Gheshlaghi, Shahin Atakishiyev
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-01875

Browse the full library · About the Tower of Babel