Asymptotic Smoothing of the Lipschitz Loss Landscape in Overparameterized One-Hidden-Layer ReLU Networks Saveliy Baturin saveliy.xo@gmail.com Abstract We study the topology of the loss landscape of one-hidden-layer ReLU networks under overparameterization. On the theory side, we (i) prove that for convex L L -Lipschitz losses with an ℓ 1 \ell_{1} -regularized second layer, every pair of models at the same loss level can be connected by a continuous path within an arbitrarily small loss increase ϵ \epsilon (extending a known result for the quadratic loss); (ii) obtain an asymptotic upper bound on the energy gap ϵ \epsilon between local and global minima that vanishes as the width m m grows, implying that the landscape flattens and sublevel sets become connected in the limit. Empirically, on a synthetic Moons dataset and on the Wisconsin Breast Cancer dataset, we measure pairwise energy gaps via Dynamic String Sampling (DSS) ( 6 ) and find that wider networks exhibit smaller gaps; in particular, a permutation test on the maximum gap yields p perm = 0 p_{\mathrm{perm}}=0 , indicating a clear reduction in the barrier height. † † shortheadings: Lipschitz Loss Topology Smoothing / Baturin † † firstpage: 1 † † editor: ¡Handling Editor¿ keywords ReLU networks, overparameterization, loss landscape, sublevel set connectivity, geodesic paths, Lipschitz losses 1 Introduction Overparameterized neural networks achieve remarkable performance in practice, yet understanding why standard optimization methods find good solutions in highly nonconvex landscapes remains an open theoretical challenge. Early foundational results established that neural networks can approximate broad function classes ( 9 ; 5 ; 13 ; 7 ; 2 ) , and characterized their expressive power (for example, counting linear regions for ReLU networks) ( 4 ; 10 ; 11 ) . Subsequent work showed that certain architectures have benign optimization landscapes: for example, deep linear networks have no bad local minima ( 8 ) , and adding skip connections in a shallow network removes any local minima worse than those of a linear predictor ( 12 ) . Increasing the number of hidden units (width) can also dramatically affect the loss topology. Theoretical evidence by 6 showed that in one-hidden-layer ReLU networks, wider layers lead to fewer local minima and a flatter quadratic-loss landscape, with geodesic paths connecting different minima on the same sublevel sets. We formalize and extend this picture to a broader class of losses in the one-hidden-layer setting. Contributions. Our contributions are both theoretical and practical: 1. Sublevel connectivity and vanishing energy gap. We prove that for a one-hidden-layer ReLU network, any two models with loss below a given level can be connected by a continuous path within a small loss increase \epsilon . In particular, for quadratic loss (mean squared error) we outline the classical argument by 6 that constructs such a path, and we extend the construction to any convex loss that is L -Lipschitz in the predictions (with an \ell_{1} regularization on the output layer). Moreover, we show that the “energy gap” \epsilon —the worst-case loss barrier along the connecting path—can be made arbitrarily small by increasing the width m . In fact, \epsilon=O(m^{-\zeta}) for some \zeta>0 , implying that in the infinite-width limit all loss sublevel sets become connected. 2. Experiments. We present pairwise energy-gap experiments on a synthetic Moons dataset (regression with mean squared error) and the Wisconsin Breast Cancer dataset ( 1 ) (binary classification with cross-entropy). For each pair we measure the gap \epsilon=\max_{t}\big[L(\gamma(t))-\max\{L(\theta_{A}),L(\theta_{B})\}\big] along DSS paths. Wider networks exhibit smaller gaps; a permutation test on the maximum gap yields p_{\mathrm{perm}}=0 , which we treat as the primary indicator of barrier reduction. 2 Preliminary Network model. We consider a one-hidden-layer neural network with Rectified Linear Unit activation. Formally, for an input x\in\mathbb{R}^{n} the network output is \Phi(x;W^{1},\theta)=\theta^{\top}Z(W^{1}x), (1) where W^{1}\in\mathbb{R}^{m\times n} is the weight matrix of the hidden layer ( m neurons) and \theta\in\mathbb{R}^{m} is the vector of output layer weights. The activation Z(\cdot) applies the ReLU function elementwise: Z(t)=\max\{0,t\} . We assume without loss of generality that each row w_{i} of W^{1} is an \ell_{2} unit vector (using the homogeneity Z(c\,w_{i}x)=c\,Z(w_{i}x) to normalize, as is standard in analysis of ReLU networks). Loss function and regularization. Let (X,Y)\sim P be a data distribution (the empirical case will be addressed in experiments). We write \hat{Y}=\Phi(X;W^{1},\theta) for the network output (logits). Throughout, we restrict attention to convex Lipschitz losses of the logits . This covers standard logistic objectives used in practice: • For binary classification with y\in\{\pm 1\} and logit z=\hat{Y} , the logistic loss \ell(z,y)=\log(1+e^{-yz}) is convex and globally 1 -Lipschitz in z under \ell_{2} . • For multiclass classification with logits z\in\mathbb{R}^{K} and label y\in\{1,\dots,K\} , the cross-entropy loss \ell(z,y)=-\log\tfrac{e^{z_{y}}}{\sum_{j}e^{z_{j}}} is convex and globally \sqrt{2} -Lipschitz in z under \ell_{2} . The regularized expected risk is F_{o}(W^{1},\theta)=\mathbb{E}_{(X,Y)\sim P}\,\mathcal{L}\!\left(Y,\Phi(X;W^{1},\theta)\right)+\kappa\,\|\theta\|_{1}, (2) with \kappa>0 a regularization coefficient (we include an \ell_{1} penalty on the second-layer weights, which will be convenient in our theoretical arguments for Lipschitz losses). The empirical counterpart F_{e} is defined analogously as a finite-sample average. We will sometimes write \theta for (W^{1},\theta) when no confusion arises. Loss sublevel sets. For a given loss value \lambda , the sublevel set is \Omega_{F}(\lambda)=\{\theta:F_{o}(\theta)\leq\lambda\} . If a sublevel set \Omega_{F}(\lambda) is connected for every \lambda , then clearly every local minimum of F_{o} must also be a global minimum (since any two minima at value \lambda can be connected within \Omega_{F}(\lambda) , contradicting the existence of an isolated strict local minimum at higher loss). We will show that in overparameterized networks, even though sublevel sets might not be strictly connected due to a small energy barrier, they become arbitrarily close to connected as width grows (Theorem 3 ). Robust compressibility. For a hidden layer \bar{W}\in\mathbb{R}^{n\times m} : \delta_{\bar{W}}(l,\alpha;m)=\min_{\begin{subarray}{c}\|\gamma\|_{0}\leq l,\\ \sup_{i}\angle(\hat{w}_{i},w_{i})\leq\alpha\end{subarray}}\mathbb{E}\!\left[\,|Y-\gamma Z(\bar{W})|^{2}\,\right]+\kappa\|\gamma\|_{1}, (3) where \gamma is a sparse coefficient vector ( \|\gamma\|_{0}\leq l ), and \angle(\hat{w}_{i},w_{i}) denotes the angle between the original weight w_{i} and its perturbed version \hat{w}_{i} . This quantity measures how well the hidden-layer representation can be compressed while retaining only l neurons and allowing small perturbations of magnitude \alpha . It is used to analyze the connectivity of sublevel sets in the loss landscape. Notation. We use \langle a,b\rangle for the standard inner product and \|v\| for the Euclidean norm. We let \Sigma_{X}=\mathbb{E}[XX^{\top}] denote the second-moment (covariance) operator of the input distribution. For a vector w , we write z(w)=Z(wX)=\max\{0,\,w^{\top}X\} to denote the random output of a ReLU neuron with weight w . Finally, \|\Sigma_{X}\| is the spectral norm of the covariance matrix (the largest eigenvalue), and S^{n-1} denotes the unit sphere in \mathbb{R}^{n} . 3 Lipschitz Convex Loss: Connectivity and Path Construction We extend the connectivity result to a much broader class of loss functions. Specifically, let \mathcal{L}(Y,\hat{Y}) be any loss that is convex in the prediction \hat{Y} and L -Lipschitz continuous in \hat{Y} . For instance, the logistic loss and binary cross-entropy are convex in \hat{Y} and Lipschitz on any compact domain (in classification tasks \hat{Y} often lies in [0,1] after a sigmoid). We assume an \ell_{1} regularization \kappa\|\theta\|_{1} on the second-layer weights, with \kappa>0 (note this is usually present or can be added without hurting performance). Before stating the main theorem, we prove a useful lemma that bounds the size of the optimal second-layer weights in terms of the Lipschitz constant L and \kappa . This will ensure that neurons cannot have arbitrarily large influence, which helps control the loss when we perturb or remove neurons. Lemma 1 (Control of \ell_{1} -norm) . Let \mathcal{L}(Y,\hat{Y}) be convex in \hat{Y} and L -Lipschitz, and let \kappa>0 . For any fixed first-layer W^{1} , consider the optimization problem \theta^{*}\in\mathop{\mathrm{arg\,min}}_{\theta\in\mathbb{R}^{m}}\Big\{\,\mathbb{E}[\mathcal{L}(Y,\Phi(X;W^{1},\theta))]\,+\,\kappa\|\theta\|_{1}\,\Big\}. Then the optimal solution \theta^{*} satisfies \|\theta^{*}\|_{1}\leq L/\kappa . In fact, if \kappa\geq L then \theta^{*}=0 (that is, the minimum is achieved by the zero output weights). Proof. Let F_{o}(\theta)=\mathbb{E}[\mathcal{L}(Y,\Phi(X;W^{1},\theta))]+\kappa\|\theta\|_{1} be the convex objective as a function of \theta . Since F_{o} is convex and differentiable in \theta almost everywhere, at the optimum \theta^{*} we have the first-order (subgradient) optimality condition 0\in\partial F_{o}(\theta^{*})=\nabla_{\theta}\mathbb{E}[\mathcal{L}]\,\big|_{\theta^{*}}+\kappa\,\partial\|\theta^{*}\|_{1} . This implies: • For any coordinate i such that \theta^{*}_{i}\neq 0 , we must have \nabla_{\theta_{i}}\mathbb{E}[\mathcal{L}]=-\kappa\,\mathrm{sign}(\theta^{*}_{i}) . In particular, |\nabla_{\theta_{i}}\mathbb{E}[\mathcal{L}]|=\kappa . • For any coordinate i such that \theta^{*}_{i}=0 , we have \nabla_{\theta_{i}}\mathbb{E}[\mathcal{L}]\in[-\kappa,\,+\kappa] (the subgradient of |\theta_{i}| at 0 is any value in [-1,1] ). Next, Lipschitz continuity of \mathcal{L} in \hat{Y} implies a bound on the gradient norms: since |\mathcal{L}(y,\hat{y}_{1})-\mathcal{L}(y,\hat{y}_{2})|\leq L|\hat{y}_{1}-\hat{y}_{2}| , we have \|\nabla_{\theta}\mathbb{E}[\mathcal{L}]\|_{\infty}\leq L . In particular |\nabla_{\theta_{i}}\mathbb{E}[\mathcal{L}]|\leq L for every i . Combining this with the first optimality condition above for coordinates where \theta^{*}_{i}\neq 0 , we get \kappa=|\nabla_{\theta_{i}}\mathbb{E}[\mathcal{L}]|\leq L . Thus a necessary condition for \theta^{*}_{i} to be nonzero is \kappa\leq L . Equivalently, if \kappa>L then the only feasible solution is \theta^{*}=0 (because any nonzero coefficient would violate the condition). This proves the second statement. Finally, to bound \|\theta^{*}\|_{1} when \kappa\leq L , we can use convexity of \mathbb{E}[\mathcal{L}] : for any \theta , \mathbb{E}[\mathcal{L}(Y,\Phi(X;W^{1},\theta))]\geq\mathbb{E}[\mathcal{L}(Y,\Phi(X;W^{1},\theta^{*}))]+\langle\nabla_{\theta}\mathbb{E}[\mathcal{L}]|_{\theta^{*}},\theta-\theta^{*}\rangle\,. Plugging in \theta=0 and using \nabla_{\theta}\mathbb{E}[\mathcal{L}]|_{\theta^{*}}=-\kappa\,\mathrm{sign}(\theta^{*}) (for each component where \theta^{*} is nonzero, and bounded in [-\kappa,\kappa] where \theta^{*} is zero), we get \begin{gathered}\mathbb{E}[\mathcal{L}(Y,\Phi(X;W^{1},0))]\geq\mathbb{E}[\mathcal{L}(Y,\Phi(X;W^{1},\theta^{*}))]-\kappa\,\langle\mathrm{sign}(\theta^{*}),\,\theta^{*}\rangle=\\ =\mathbb{E}[\mathcal{L}(Y,\Phi(X;W^{1},\theta^{*}))]+\kappa\,\|\theta^{*}\|_{1}\,,\end{gathered} (4) since \langle\mathrm{sign}(\theta^{*}),\theta^{*}\rangle=\|\theta^{*}\|_{1} . On the other hand, by L -Lipschitz continuity, \mathbb{E}[\mathcal{L}(Y,\Phi(X;W^{1},0))]-\mathbb{E}[\mathcal{L}(Y,\Phi(X;W^{1},\theta^{*}))]\leq L\,\|\theta^{*}\|_{1}\,, because \Phi(X;W^{1},0)=0 and \Phi(X;W^{1},\theta^{*})=\sum_{i=1}^{m}\theta^{*}_{i}\,z(w^{*}_{i}) differs by at most |\theta^{*}_{i}| per active neuron. Combining the two inequalities yields \kappa\,\|\theta^{*}\|_{1}\leq L\,\|\theta^{*}\|_{1}\,, and since \kappa>0 we can cancel \|\theta^{*}\|_{1} to obtain \|\theta^{*}\|_{1}\leq L/\kappa (and this inequality trivially holds if \theta^{*}=0 as well). ∎ Lemma 1 implies that throughout the training landscape (and in particular for any local or global minimum), the sum of the absolute output weights is bounded by L/\kappa . In the special case \kappa\geq L , the only optimum is the trivial zero network—which in practice one would avoid by choosing \kappa