Depth, Not Data: An Analysis of Hessian Spectral Bifurcation
Shenyang DengDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected], Boyao LiaoDepartment of MathematicsUniversity of BirminghamBirmingham, UK [email protected], Zhuoli OuyangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected] Tianyu PangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected] Yaoqing YangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected] · 2026
Read Depth, Not Data: An Analysis of Hessian Spectral Bifurcation by Shenyang DengDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected], Boyao LiaoDepartment of MathematicsUniversity of BirminghamBirmingham, UK [email protected], Zhuoli OuyangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected] Tianyu PangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected] Yaoqing YangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-00545.txt.
Description
Shenyang Deng and colleagues challenge the view that the bulk-and-spike Hessian spectrum of deep networks comes from imbalanced data covariance. In a deep linear network with perfectly balanced data covariance, the Hessian still splits into a dominant cluster and a bulk cluster. The ratio between dominant and bulk eigenvalues grows linearly with network depth, showing architecture, not data alone, shapes the spectral gap. Architecture and data characteristics should therefore both inform optimisation algorithm design for deep networks.
Catalog record
| CREATOR | Shenyang DengDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected], Boyao LiaoDepartment of MathematicsUniversity of BirminghamBirmingham, UK [email protected], Zhuoli OuyangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected] Tianyu PangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected] Yaoqing YangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 97717 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602.00545