Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics

Connall Garrod, Jonathan P. Keating, Christos Thrampoulidis · 2025

Read Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics by Connall Garrod, Jonathan P. Keating, Christos Thrampoulidis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-04006.txt.

Introduction

Cross-entropy loss runs nearly all of deep learning, yet theory usually swaps it for squared error or studies only convex models, missing how it truly behaves. Connall Garrod, Jonathan P. Keating, and Christos Thrampoulidis close part of that gap in a 2025 machine learning preprint. Their testbed is the simplest non-convex case: a two-layer linear network fed standard basis inputs.

A quiet discovery powers the analysis. Initialising the network with a Hadamard pattern diagonalises the softmax operator. The weight matrices keep their singular vectors fixed, so only singular values evolve. With that reduction, the authors build a Lyapunov function and prove global convergence, despite spurious critical points in the landscape. The endpoint is the neural collapse geometry, the first proof that gradient flow on cross-entropy reaches it. The authors suggest the trick can unlock analyses well beyond their model.

Catalog record

CREATORConnall Garrod, Jonathan P. Keating, Christos Thrampoulidis
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE114 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-04006

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics Connall Garrod, Jonathan P. Keating, Christos Thrampoulidis Abstract Cross-entropy (CE) training loss dominates deep learning practice, yet existing theory often relies on simplifications—either replacing it with squared loss or restricting to convex models—that miss essential behavior. CE and squared loss generate fundamentally different dynamics, and convex linear models cannot capture the complexities of non-convex optimization. We provide an in-depth characterization of multi-class CE optimization dynamics beyond the convex regime by analyzing a canonical two-layer linear neural network with standard-basis vectors as inputs—the simplest non-convex extension for which the implicit bias remained unknown. This model coincides with the unconstrained features model used to study neural collapse, making our work the first to prove that gradient flow on CE converges to the neural collapse geometry. We construct an explicit Lyapunov function that establishes global convergence, despite the presence of spurious critical points in the non-convex landscape. A key insight underlying our analysis is an inconspicuous finding: Hadamard Initialization diagonalizes the softmax operator, freezing the singular vectors of the weight matrices and reducing the dynamics entirely to their singular values. This technique opens a pathway for analyzing CE training dynamics well beyond our specific setting considered here. 1 Introduction 1.1 Motivation Modern overparameterized classification networks exhibit a remarkable geometric structure at the end of training, known as neural collapse (NC) [ 52 ] : class features collapse to their respective means, those means form a simplex equiangular tight frame (ETF), and the classifier aligns with this frame. This phenomenon appears consistently across diverse architectures and datasets, despite the absence of any explicit mechanism in the training procedure that enforces it. This raises a central question: What implicit mechanism in gradient-based optimization produces this precise geometric configuration among the infinitely many solutions that achieve zero training loss? This question lies squarely within the implicit bias framework, which seeks to characterize the specific solutions that gradient descent selects when multiple optima exist. For logistic regression, Soudry et al. [ 60 ] established that gradient descent converges to the max-margin solution. Subsequent research extended this result in several directions, most notably to deep homogeneous networks [ 48 , 37 , 23 ] . These works prove convergence of gradient descent—or its infinitesimal limit, gradient flow—to KKT points of corresponding non-convex max-margin problems. Yet, in contrast to the convex case, these results are comparatively less explicit: they do not specify which KKT point is reached. For NC, Ji et al. [ 34 ] and Thrampoulidis et al. [ 68 ] applied this framework to the unconstrained features model (UFM), a theoretical proxy for studying representation geometries in deep networks. They showed that cross-entropy (CE) gradient flow converges to a KKT point of a max-margin problem whose global minimizer corresponds to NC. Critical gaps remain, however: these results guarantee only convergence to some KKT point—which may be a saddle rather than NC—and provide no characterization of the training trajectory or the convergence rate. More broadly, existing implicit bias results remain restricted to three narrow regimes: (i) logistic regression, which lacks depth; (ii) binary classification, which omits multi-class interactions; and (iii) KKT analyses, which cannot distinguish between NC and other critical points. This gap is particularly significant for unregularized training, where the global minimum is attained only at infinity: analyses based solely on asymptotic limits may overlook crucial finite-time dynamics, such as trajectories remaining near non-NC saddles indefinitely or exhibiting non-monotonic behavior that initially moves away from NC. Understanding implicit bias in this setting requires analyzing the entire trajectory , not just its limit. A complementary line of work, pioneered by Saxe et al. [ 58 , 59 ] and extended by numerous follow-ups, provides tools for such trajectory-level analysis, but only for mean squared error (MSE) loss. Under special initializations that restrict dynamics to singular values, they derived exact solutions for linear networks, showing that singular values evolve independently along sigmoidal curves. This framework was recently applied to the UFM with MSE loss [ 81 ] . However, NC is observed in classification networks that use CE loss, which introduces a softmax nonlinearity and couples singular value dynamics in ways that prevent exact solutions. Whether the insights and techniques from MSE can extend to CE—and whether they can rigorously characterize the implicit bias toward NC—has remained an open question. 1.2 Contributions We provide the first rigorous proof of convergence to NC under multi-class CE training in a non-convex setting. Our key technical innovation is showing that Hadamard Initialization diagonalizes the softmax operator in the UFM. Concretely, for a normalized Sylvester-Hadamard matrix U and any vector a , we prove that: P(U\operatorname{diag}\left(a\right)U^{T})=U\operatorname{diag}\left(UP(Ua)\right)U^{T}\,, where P(\cdot) is the column-wise softmax map. This property enables us to extend the Saxe et al. framework from MSE to CE, reducing its gradient flow to a vector ODE over the K-1 non-trivial singular values a of the logit matrix: \frac{da}{dt}=a\odot\big(1-\tilde{U}^{T}P(\tilde{U}a)\big)\,, where \odot is the entrywise product and \tilde{U} represents the Hadamard modes excluding the trivial constant mode. Unlike the MSE setting, this reduced ODE retains the complex couplings of the softmax: singular values do not evolve independently. Nevertheless, we show the dynamics are sufficiently structured to allow rigorous implicit bias analysis. Non-monotonic convergence. Prior work has numerically observed monotonic convergence to NC under CE, but we show that this is not true in general: convergence can be non-monotonic, and the behavior depends sensitively on the number of classes K . For K\in\{2,4\} , natural distance metrics decrease monotonically, whereas for K\geq 8 we construct explicit initializations where these metrics initially increase before eventually decreasing. Even more strikingly, we construct metrics that are monotonic even for K=8 but break down for K\geq 16 . These phenomena are invisible to KKT-based analyses and reveal a subtle, previously unrecognized K -dependence in multi-class dynamics. This complexity is missed by prior implicit bias work, which has focused almost exclusively on the binary case. Guaranteed convergence to NC via Lyapunov function. Despite the non-monotonicity of natural metrics and the presence of competing saddle points, we prove that gradient flow always converges to NC by constructing an explicit Lyapunov function—the KL divergence—that decreases monotonically throughout training. This provides the first rigorous characterization of implicit bias toward NC beyond binary classification. Our results demonstrate that NC is not merely an empirical observation but the inevitable outcome of gradient-based optimization in this setting. Exponentially slow convergence. As the logit norm grows, the CE dynamics slow down exponentially, causing trajectories with unfavorable initializations to linger near non-NC saddle points for arbitrarily long periods. This plateau behavior is practically significant yet invisible to purely asymptotic analyses. Role of initialization. To reconcile our non-monotonic examples with the monotonic behavior observed in practice, we show that standard near-zero random initialization schemes naturally bias the dynamics toward the simplex ETF subspace, steering trajectories away from problematic saddle regions. This explains why non-monotonic behavior, while theoretically possible, is rarely observed. MSE vs CE comparison. To highlight the impact of softmax nonlinearity, we show that MSE and CE dynamics coincide only under restrictive conditions: either in the linearized regime (small norms) or under direct logit optimization (removing depth). Outside these settings, CE exhibits coupled dynamics, non-monotonicity, and prolonged saddle plateaus that

Read the complete text