Dispelling the Curse of Singularities in Neural Network Optimizations

Hengjie Cao Mengyi Chen Yifeng Yang Fang Dong Ruijun Huang, Anrui Chen Jixian Zhou Mingzhi Dong Yujiang Wang Dongsheng Li, Wenyi Fang Yuanyi Lin FAN WU Li Shang · 2026

Read Dispelling the Curse of Singularities in Neural Network Optimizations by Hengjie Cao Mengyi Chen Yifeng Yang Fang Dong Ruijun Huang, Anrui Chen Jixian Zhou Mingzhi Dong Yujiang Wang Dongsheng Li, Wenyi Fang Yuanyi Lin FAN WU Li Shang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01308.txt.

Description

Hengjie Cao and co-authors blame deep-net train wobbles on growing singularities in weight space. Singular spikes grow with each slope step and sync ever tighter with layer outputs, loosening slope-norm caps and raising the odds of sharp loss blowups. They dub that feedback loop the curse of singularities. Their Parametric Singularity Smoothing plan irons out weight singular spectra in a light, bendy way. Tests across data sets, net forms, and tuners show it calms trains, revives dead trains, and lifts both speed and broad scores.

Catalog record

CREATORHengjie Cao Mengyi Chen Yifeng Yang Fang Dong Ruijun Huang, Anrui Chen Jixian Zhou Mingzhi Dong Yujiang Wang Dongsheng Li, Wenyi Fang Yuanyi Lin FAN WU Li Shang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE98192
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.01308

Browse the full library · About the Tower of Babel