Embedding principle of homogeneous neural network for classification problem

Jiahan Zhang, Yaoyu Zhang, Tao Luo · 2025

Read Embedding principle of homogeneous neural network for classification problem by Jiahan Zhang, Yaoyu Zhang, Tao Luo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12419.txt.

Introduction

Wide nets and thin nets solve the same sort task in like ways. Jiahan Zhang, Yaoyu Zhang, and Tao Luo ask in 2025 if their best points can be mapped to each other. The nets in view are smooth, in that each part scales in step when its weights scale.

The team sets a plant rule for such best points. A best point of a thin net can be set into the best points of a wide net by a fixed, length true map. They prove it for full nets by split of a nerve cell, and for coil nets by split of a path. Train flows that start mapped stay mapped as they run. Tests back the claim. The rule ties width, spare parts, and what train finds.

Catalog record

CREATORJiahan Zhang, Yaoyu Zhang, Tao Luo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-12419

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Embedding principle of homogeneous neural network for classification problem Jiahan Zhang, Yaoyu Zhang, Tao Luo Abstract In this paper, we study the Karush-Kuhn-Tucker (KKT) points of the associated maximum-margin problem in homogeneous neural networks, including fully-connected and convolutional neural networks. In particular, We investigates the relationship between such KKT points across networks of different widths generated. We introduce and formalize the KKT point embedding principle , establishing that KKT points of a homogeneous network’s max-margin problem ( P Φ P_{\Phi} ) can be embedded into the KKT points of a larger network’s problem ( P Φ ~ P_{\tilde{\Phi}} ) via specific linear isometric transformations. We rigorously prove this principle holds for neuron splitting in fully-connected networks and channel splitting in convolutional neural networks. Furthermore, we connect this static embedding to the dynamics of gradient flow training with smooth losses. We demonstrate that trajectories initiated from appropriately mapped points remain mapped throughout training and that the resulting ω \omega -limit sets of directions are correspondingly mapped, thereby preserving the alignment with KKT directions dynamically when directional convergence occurs. We conduct several experiments to justify that trajectories are preserved. Our findings offer insights into the effects of network width, parameter redundancy, and the structural connections between solutions found via optimization in homogeneous networks of varying sizes. 1 Introduction The optimization of neural networks remains a central challenge, with significant research dedicated to understanding the training dynamics and the properties of converged solutions. For homogeneous networks, such as those using ReLU-like activations without biases, a particularly fruitful line of inquiry connects optimization algorithms like gradient descent to the concept of margin maximization in classification tasks ( Lyu and Li, 2019 ; Ji and Telgarsky, 2020 ) . In these settings, the Karush-Kuhn-Tucker (KKT) conditions associated with a minimum-norm maximum-margin problem provide a theoretical characterization of optimal parameter configurations ( Gunasekar et al., 2018 ; Nacson et al., 2019b ) . While much work focuses on the implicit bias and convergence properties within a single network architecture, a fundamental question arises regarding the relationship between solutions found in networks of different sizes. Specifically, how do the optimal solutions (characterized by KKT points) of a smaller homogeneous network relate to those of a larger network, particularly one derived by increasing width through operations like neuron splitting? Understanding this relationship is crucial for insights into the effects of overparameterization, the structure of the solution space, and the mechanisms underlying implicit regularization. To address this question, this paper introduces the KKT Point Embedding Principle . The core idea is that under specific, structure-preserving transformations corresponding to neuron splitting, the KKT points of the max-margin problem associated with a smaller homogeneous network ( \Phi ) can be precisely mapped, via a linear isometry T , to KKT points of the analogous problem for a larger network ( \tilde{\Phi} ). This principle provides a concrete link between the optimization landscapes and solution sets of related networks. Specifically, our contributions include: 1. We formalize the KKT Point Embedding Principle (Theorem 4.2 ) and establishing conditions under which a linear transformation preserves KKT points between the max-margin problems P_{\Phi} and P_{\tilde{\Phi}} (Theorem 4.5 ). 2. We provide explicit constructions and rigorous proofs demonstrating that this principle holds for neuron splitting in full-connected networks (Theorem 4.8 ) and channel splitting in convolutional neural networks(Theorem 4.11 ), showing these transformations satisfy the required conditions and are isometric. 3. We connect our static KKT embedding to the dynamics of training by proving a strong trajectory preservation principle. Specifically, we show that the entire gradient flow path of the wider network is a linear mapping of its narrower counterpart ( \boldsymbol{\eta}(t)=T\boldsymbol{\theta}(t) ) (Theorem 5.2 ). This dynamic correspondence directly implies that the asymptotic behavior is also preserved, ensuring the set of directional limit points is mapped accordingly ( T(L(\boldsymbol{\theta}(0)))=L(\boldsymbol{\eta}(0)) ) and preserving the convergence towards KKT-aligned solutions (Theorem 5.4 , Corollary 5.5 ). The remainder of this paper is organized as follows. Section 2 discusses related work on implicit bias, network embeddings, and KKT conditions. Section 3 introduces necessary notations and definitions. Section 4 develops the static KKT Point Embedding Principle and applies it to neuron splitting and channl. Section 5 connects the static principle to gradient flow dynamics. Finally, Section 6 concludes the paper. An overview of our theoretical contributions and their relationships is provided in Figure 1 . Figure 1 : Overview of Theoretical Contributions. This figure illustrates the logical flow of our paper’s theoretical framework. The analysis begins with the static principles (top section), which are then shown to apply to specific network architectures. These static results form the basis for the dynamic analysis (bottom row), which shows the preservation of training trajectories and their limit sets. KKT Point Embedding Principle (Thm. 4.2 ) Equivalence Conditions (Thm. 4.5 ) Full-connected Network neuron Splitting (Thm. 4.8 ) CNN Channel Splitting (Thm. 4.11 ) \omega -limit Set Mapping (Thm. 5.4 ) Trajectory Preservation (Thm. 5.2 ) Limit Direction Embedding (Cor. 5.5 ) Characterizes Special case Special case Preserves Preserves Implies Special case Static Analysis: KKT points of smaller network embed into larger network Dynamic Analysis: Training trajectories preserve embedding structure 2 Related work Our work builds upon and contributes to several lines of research concerning neural network optimization, implicit bias, network structure, and optimality conditions. Optimization dynamics and implicit bias A significant body of research investigates the implicit bias of gradient-based optimization in neural networks. Seminal work by Soudry et al. (2018) showed that for linear logistic regression on separable data, gradient descent (GD) converges in direction to the L^{2} -max-margin solution. This finding was extended to various settings, including stochastic GD ( Nacson et al., 2019c ) , and generalized to deep linear networks ( Ji and Telgarsky, 2018 ) . For non-linear homogeneous neural networks (e.g., ReLU networks without biases), studies confirm that GD/GF with exponential-tailed losses also typically steer parameters towards margin maximization ( Lyu and Li, 2019 ; Ji and Telgarsky, 2020 ; Nacson et al., 2019a ; Wei et al., 2019 ) . The resulting solutions are frequently characterized as KKT points of an associated max-margin problem ( Gunasekar et al., 2018 ; Nacson et al., 2019c ) . More recently, this line of inquiry has been extended to analyze the margin in non-homogeneous models ( Kunin et al., 2023 ; Cai et al., 2023 ) and to connect KKT conditions with the phenomenon of benign overfitting ( Frei et al., 2023 ) . These analyses, however, primarily focus on the dynamics within a single, fixed-width network, with less exploration of how such KKT-characterized solutions relate across networks of varying widths, such as those generated by neuron splitting. Network embedding principles The idea that smaller network functionalities can be embedded within larger ones has been explored, notably by Zhang et al. (2021) who proposed an “Embedding Principle.” This concept is related to foundational ideas on hierarchical structures and singularities in the parameter space ( Fukumizu and Amari, 2000 ) , as well as more recent work exploring symmetries, such as permutation invariance, that connect different global minima ( Simsek et al., 2021 ) . While general embedding principles focus on preserving the loss landscape, their specific application to the

Read the complete text