The Empirical Impact of Reducing Symmetries on the Performance of Deep Ensembles and MoE
Andrei Chernov, Oleg Novitskij · 2025
Read The Empirical Impact of Reducing Symmetries on the Performance of Deep Ensembles and MoE by Andrei Chernov, Oleg Novitskij (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17391.txt.
Introduction
Andrei Chernov and Oleg Novitskij tested how cutting symmetries in neural networks affects ensembles in a 2025 paper. Prior work showed that networks with fewer symmetries connect to each other more smoothly, so blended networks perform better. But blending is rare in practice; teams instead average the answers of separate networks in an ensemble.
The authors ask whether the symmetry trick helps those practical setups, including mixture-of-experts models. Their experiments measure accuracy changes when symmetry is reduced before ensembling. The answer guides a cheap design choice: whether to constrain networks at birth for later teamwork. Results speak to anyone who ships model ensembles and wants free gains from structure rather than more compute.
Catalog record
| CREATOR | Andrei Chernov, Oleg Novitskij |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-17391
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Empirical Impact of Reducing Symmetries on the Performance of Deep Ensembles and MoE Andrei Chernov, Oleg Novitskij Abstract Recent studies have shown that reducing symmetries in neural networks enhances linear mode connectivity between networks without requiring parameter space alignment, leading to improved performance in linearly interpolated neural networks. However, in practical applications, neural network interpolation is rarely used; instead, ensembles of networks are more common. In this paper, we empirically investigate the impact of reducing symmetries on the performance of deep ensembles and Mixture of Experts (MoE) across five datasets. Additionally, to explore deeper linear mode connectivity, we introduce the Mixture of Interpolated Experts (MoIE). Our results show that deep ensembles built on asymmetric neural networks achieve significantly better performance as ensemble size increases compared to their symmetric counterparts. In contrast, our experiments do not provide conclusive evidence on whether reducing symmetries affects both MoE and MoIE architectures. The code is available on GitHub 1 1 1 https://github.com/krds00/asym_ensembles/ . 1 Introduction In the last decade, neural networks have proven to be one of the most important algorithms in the field of machine learning. Despite their undeniable empirical success, many fundamental questions remain unanswered. One such question concerns parameter space symmetries: for any given set of neural network parameters, there exist numerous ‘twins’ that produce exactly the same output for every input while having different parameter values. There are multiple sources of symmetry in neural network architectures. One prominent example is permutation symmetry in fully connected layers. Consider a standard multi-layer perceptron (MLP). If we swap two neurons in a hidden layer along with their incoming and outgoing weights, the network will produce the exact same output. As a result, any hidden layer of size n has n! different sets of parameters that yield identical outputs. Activation functions such as ReLU Nair & Hinton 2010 can also produce symmetries Wiese et al. 2023 . The effects of parameter symmetries have been studied in various areas, including neuron interpretability Godfrey et al. 2022 , optimization Neyshabur et al. 2015 , and Bayesian deep learning Kurle et al. 2022 . In this work, we primarily focus on the impact of symmetries on model accuracy. It has been shown in Lim et al. 2024 that eliminating redundant parameters in neural networks improves linear mode connectivity, thereby enhancing the performance of networks whose parameters are obtained by interpolating between two trained models. In this study, we present the Mixture of Interpolated Experts model (see Section 4.3 for details) and investigate how parameter symmetries influence its performance. However, the practical application of interpolated neural networks remains controversial, as such architectures usually do not provide a performance boost. The most common approach to leveraging multiple neural networks is through ensembles, where outputs from several models are aggregated to form a final prediction—examples include Deep Ensembles Lakshminarayanan et al. 2017 and Mixture of Experts (MoE) Fedus et al. 2022 . In this paper, we empirically investigate how reducing symmetry impacts the performance of Deep Ensembles and MoE on 5 datasets. 2 Related Work 2.1 W-Asymmetric MLP Various methods for breaking parameter symmetries in neural networks have been studied, including approaches to removing permutation symmetries Pourzanjani et al. 2017 ; Pittorino et al. 2022 , scaling symmetries Badrinarayanan et al. 2015 , and sign symmetries Wiese et al. 2023 . However, in most of these approaches, the neural network architectures or training processes deviate from standard practices, making them difficult to apply in practice. In this work, we fully adopt the approach from Lim et al. 2024 to break symmetries in neural networks. This method randomly freezes a portion of the neural network’s weights before training, keeping them unchanged throughout training (see Section 4.1 for details). Notably, it does not require any special modifications to the training process. Authors of Lim et al. 2024 showed that breaking symmetries improves linear mode connectivity between two independently trained neural networks. In this paper, we investigate the empirical impact of reducing symmetries on the performance of Deep Ensembles and Mixture of Experts. 2.2 Neural Network Ensembles In this study, we employ two different approaches for ensembling neural networks. The first approach, known as Deep Ensembles Lakshminarayanan et al. 2017 , trains k neural networks independently and averages their outputs to obtain the final prediction. The second approach is the Mixture of Experts (MoE) Yuksel et al. 2012 , which consists of two main components: experts and a gating network. Each expert generates an output, but unlike Deep Ensembles, the final prediction is obtained through a weighted average of the experts’ outputs. The weights for each expert are dynamically predicted by the gating network rather than being fixed. Recently, MoE architectures utilizing MLP models as experts have gained popularity Fedus et al. 2022 especially in NLP Du et al. 2022 and CV domains Puigcerver et al. 2023 ; Riquelme et al. 2021 . In this work, we adapt MoE architectures for tabular data from Chernov 2025 . We cover it in detail in Section 4.2 . 3 Datasets For our work, we selected five datasets to cover different problems: • Regression: California Housing Prices dataset Pace & Barry 1997 . • Binary classification: Churn Modeling 2 2 2 https://www.kaggle.com/shrutimechlearn/churn-modelling and Adult Income Kohavi et al. 1996 . • Multi-class classification: MNIST Deng 2012 and Otto Group Product 3 3 3 https://www.kaggle.com/c/otto-group-product-classification-challenge/data . Appendix A.1 summarizes the key attributes of these datasets. To ensure consistency, we applied a standardized preprocessing pipeline. Each dataset was split into training, validation, and testing sets with an overall partitioning of 64% for training, 16% for validation, and 20% for testing. Real-valued features were scaled using a StandardScaler , and for classification tasks, the splits were stratified by the target variable. Additional preprocessing steps were applied to each dataset as follows: • Churn Modeling dataset: Non-informative columns such as RowNumber , CustomerId , and Surname were removed. • Otto Group Product dataset: The id column was dropped, and the target variable was encoded using a LabelEncoder . • Adult Income dataset: The target variable was transformed by mapping <=$50K to 0 and >$50K to 1 . Categorical features were processed using a OneHotEncoder , with missing values imputed as MissingValue , while numerical missing values were filled with 0 . • California Housing Prices dataset: Since this dataset contains no missing values, its numerical features were simply scaled. • MNIST dataset: Grayscale images were preprocessed by normalizing and centering pixel values. . 4 Models 4.1 W-Asymmetric MLP In this paper, we fully adopt the implementation of W-Asymmetric MLP (WMLP) from Lim et al. 2024 , where it was theoretically proven that this approach significantly reduces parameter symmetries. This is achieved by freezing a small portion of the weights, approximately \mathcal{O}(n^{1/4}) for details see Algorithm 1 It is important to emphasize that in ensemble networks utilizing different WMLP models, the frozen neurons—both in value and position—remain identical across all instances. For the hidden layers, we use the GeLU activation function from Hendrycks & Gimpel 2016 in both MLP and WMLP. Algorithm 1 WMLP Weight and Bias Initialization with Masking 1: Number of layers L=4 , hidden dimension d\in\{64,128,256\} , and mask seeding parameter \mathtt{mask\_num} . 2: Define fixed weights