Ubiquitous Symmetry at Critical Points Across Diverse Optimization Landscapes
Irmi Schneider · 2025
Read Ubiquitous Symmetry at Critical Points Across Diverse Optimization Landscapes by Irmi Schneider (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01959.txt.
Introduction
Irmi Schneider shows that balance points in many math landscapes always seem to carry hidden symmetry, in a 2025 paper. The puzzle began with neural networks. Their loss surfaces repeat under swaps of rows and columns of weights. Every low point found so far respects that symmetry.
Schneider tests four more settings beyond networks. They cover a projective space over a finite field, an eight sided graph, perfect matchings, and particles that attract each other. In every case, the resting points found had real symmetry. A new symmetry measure then revealed still more pattern the old one missed. The result hints at a broad rule about how balanced solutions must look.
Catalog record
| CREATOR | Irmi Schneider |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01959
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Ubiquitous Symmetry at Critical Points Across Diverse Optimization Landscapes Irmi Schneider Abstract Symmetry plays a crucial role in understanding the properties of mathematical structures and optimization problems. Recent work has explored this phenomenon in the context of neural networks, where the loss function is invariant under column and row permutations of the network weights. It has been observed that local minima exhibit significant symmetry with respect to the network weights (invariance to row and column permutations). And moreover no critical point was found that lacked symmetry. We extend this line of inquiry by investigating symmetry phenomena in real-valued loss functions defined on a broader class of spaces. We will introduce four more cases: the projective case over a finite field, the octahedral graph case, the perfect matching case, and the particle attraction case. We show that as in the neural network case, all the critical points observed have non-trivial symmetry. Finally we introduce a new measure of symmetry in the system and show that it reveals additional symmetry structures not captured by the previous measure. 1 Introduction The optimization of neural networks is a central challenge in machine learning. Recent research has emphasized the importance of symmetry in understanding the optimization landscape, particularly for shallow ReLU networks. In this context, Arjevani and Field 2019 consider the population loss minimization problem \mathcal{L}(W)\coloneqq\frac{1}{2}\,\mathbb{E}_{x\sim\mathcal{N}(0,I_{d})}\Bigl[\Bigl(1_{d}^{\top}\phi(Wx)-1_{d}^{\top}\phi(Vx)\Bigr)^{2}\Bigr], (1) where: • W\in M(d,d) is the weight matrix, • V\in M(d,d) is the target weight matrix, fixed as V=I_{d} , • \phi\colon\mathbb{R}\to\mathbb{R} is the ReLU activation function, defined by \phi(z)\coloneqq\max\{0,z\} , • 1_{d}\in\mathbb{R}^{d} is the vector of all ones. Similarly, Arjevani et al. 2021 investigate the symmetry of local minima in symmetric tensor decomposition. They study the optimization problem \min_{W\in M(d,d)}L_{\star}(W)\coloneqq\left\|\sum_{i=1}^{d}w_{i}^{\otimes n}-T\right\|^{2}_{\star}, (2) where: • w_{i} denotes the i th row of W , • \otimes n represents the n -fold tensor product, • T is a target symmetric tensor, typically expressed as T=\sum_{i=1}^{d}v_{i}^{\otimes n}, with v_{i} being the i th row of V=I_{d} , • \|\cdot\|_{\star} denotes a norm (e.g., the Frobenius norm or a Gaussian norm). Invariance Under Permutations In both settings, the loss function is invariant under the action of the group S_{d}\times S_{d}, which acts on a d\times d weight matrix W by independently permuting its rows and columns. Formally, for any (\pi_{1},\pi_{2})\in S_{d}\times S_{d} with corresponding permutation matrices P_{\pi_{1}} and P_{\pi_{2}} , the action is defined as (\pi_{1},\pi_{2})\cdot W=P_{\pi_{1}}\,W\,P_{\pi_{2}}^{-1}. (3) The stabilizer (or isotropy group) of a weight matrix W is the subgroup of S_{d}\times S_{d} that leaves W invariant. Empirical Observations Empirical studies reveal that: • ReLU Networks: For networks with an identity target matrix (which has full isotropy group \Delta S_{d}=\{(\pi,\pi)\mid\pi\in S_{d}\} ), local minima consistently exhibit nontrivial stabilizers. In particular, the observed stabilizers are often conjugate to subgroups such as \Delta S_{d-1}\times\Delta S_{1}\quad\text{or}\quad\Delta S_{d-2}\times\Delta S_{2}, which are maximal proper subgroups of \Delta S_{d} . See Figure 1 for an illustration. • Tensor Decomposition: For the tensor decomposition problem with the target tensor T=\sum_{i=1}^{d}v_{i}^{\otimes n}, where the v_{i} ’s are the rows of V=I_{d} , all local minima are observed to have stabilizers conjugate to subgroups such as \Delta(S_{d-1}\times S_{1})\quad\text{or}\quad\Delta(S_{d-3}\times S_{2}\times S_{1}), which are significant subgroups of the isotropy group of V . Figure 1 : Typical minima observed for ReLU networks with an identity target matrix. The stabilizer under the S_{d}\times S_{d} action is conjugate to \Delta S_{d-1} . These findings suggest that optimization landscapes adhere to a principle of least symmetry breaking —while spurious local minima may break the symmetry of the global minimum, they do so minimally. Indeed, after numerous optimization runs (using methods such as gradient descent or Adam), every critical point identified exhibits substantial, nontrivial symmetry. Kernel Reformulation Both optimization problems admit an elegant reformulation in terms of kernel functions. Let \kappa\colon\mathbb{R}^{d}\times\mathbb{R}^{d}\to\mathbb{R} be a kernel function. Then, the loss functions can be expressed as L(W)=\sum_{i,j=1}^{d}\kappa(w_{i},w_{j})-2\sum_{i,j=1}^{d}\kappa(w_{i},v_{j})+\sum_{i,j=1}^{d}\kappa(v_{i},v_{j}), (4) where w_{i} and v_{i} denote the i th rows of W and V , respectively. For the ReLU network case with V=I_{d} , the corresponding kernel is \kappa_{\text{ReLU}}(w,v)=\mathbb{E}_{x\sim\mathcal{N}(0,I_{d})}\Bigl[\phi(w\cdot x)\,\phi(v\cdot x)\Bigr]. (5) For the tensor decomposition problem under the Frobenius norm, the kernel is given by \kappa_{\text{tensor}}(w,v)=\langle w^{\otimes n},v^{\otimes n}\rangle_{F}=\langle w,v\rangle^{n}, (6) where \langle\cdot,\cdot\rangle_{F} denotes the Frobenius inner product and \langle\cdot,\cdot\rangle is the standard Euclidean inner product. 1.1 Main results of this paper Previous research has examined the loss function on shallow ReLU network and tensor decomposition optimization problem. Both are invariant under the S_{d}\times S_{d} group action. A key finding was that for these loss functions, every observed local minimum exhibits substantial, nontrivial symmetry through its stabilizer group. In this work, we extend this investigation to several new entirely different symmetry groups. Projective Space Case We begin by examining the projective linear group over a finite field \text{PGL}(n+1,\mathbb{F}_{q}) , where n is the dimension of the projective space and q is a prime number. This group acts naturally on the projective space \mathbb{P}^{n}(\mathbb{F}_{q}) . We assign a real-valued function to the projective space and define a loss function that remains invariant under the induced action of \text{PGL}(n+1,\mathbb{F}_{q}) on the space of real-valued functions. The loss function is constructed by summing pairwise interactions (kernel function) between restrictions of the real-valued function on pairs of projective hyperplanes. We initialize the system and optimize the loss function. After optimization, we analyze the stabilizer (denoted as I_{V} ) of the resulting configuration to understand which symmetries are preserved. Our experiments across a wide range of dimensions n , polynomial degree d of the kernel function, and prime numbers q demonstrate that local minima of this loss function consistently possess non-trivial stabilizer subgroups of \text{PGL}(n+1,\mathbb{F}_{q}) . Graph-Based Cases We investigate two specific graph structures: • Octahedral Graph : A 6-vertex, 12-edge undirected graph representing an octahedron • Perfect Matching Graph : An undirected perfect matching graph Though these graphs are undirected, we treat them as directed by assigning two directed edges (i,j) and (j,i) for each undirected edge \{i,j\}\in E . The symmetry group acting on each graph consists of its automorphisms, i.e. bijections of the vertex set that preserve the edge structure. For each vertex i we assign a real value v_{i} , then for each directed edge e=(i_{1},i_{2}) , we define an edge vector (v_{i_{1}},v_{i_{2}}) . We then construct the loss function as the sum of a kernel function evaluated on all pairs of these edge vectors. This kernel function represents the interaction between edges. We initialize the system and optimize the loss function. After optimization, we analyze the stabilizer (denoted as I_{V} ) of the resulting configuration to understand which symmetries are preserved. Notably, we show that in both graph cases, every observed critical point demonstrates non-trivial, substantial symmetry. Particle Attraction Case Finally, we explore a system of n particles in \mathbb{R}^{2} . In this model, a kernel function defines the attraction and repulsion between particle pairs, and the overall loss aggregates all pairwise interactions. This loss function is invariant under the action of the symmetric group S_{n} on particle positions. Our experiments show that almost all critical points possess non-trivial symmetry. New Measure of Symmetry To capture more subtle symmetry properties, we introduce a new measure called the edge isotropy group ( I_{E} ). This group generalizes the vertex isotropy group