Depth, Not Data: An Analysis of Hessian Spectral Bifurcation

Shenyang DengDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected], Boyao LiaoDepartment of MathematicsUniversity of BirminghamBirmingham, UK [email protected], Zhuoli OuyangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected] Tianyu PangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected] Yaoqing YangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected] · 2026

Read Depth, Not Data: An Analysis of Hessian Spectral Bifurcation by Shenyang DengDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected], Boyao LiaoDepartment of MathematicsUniversity of BirminghamBirmingham, UK [email protected], Zhuoli OuyangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected] Tianyu PangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected] Yaoqing YangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-00545.txt.

Description

Shenyang Deng and colleagues challenge the view that the bulk-and-spike Hessian spectrum of deep networks comes from imbalanced data covariance. In a deep linear network with perfectly balanced data covariance, the Hessian still splits into a dominant cluster and a bulk cluster. The ratio between dominant and bulk eigenvalues grows linearly with network depth, showing architecture, not data alone, shapes the spectral gap. Architecture and data characteristics should therefore both inform optimisation algorithm design for deep networks.

Catalog record

CREATORShenyang DengDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected], Boyao LiaoDepartment of MathematicsUniversity of BirminghamBirmingham, UK [email protected], Zhuoli OuyangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected] Tianyu PangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected] Yaoqing YangDepartment of Computer ScienceDartmouth CollegeHanover, NH, USA [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97717
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.00545

Browse the full library · About the Tower of Babel