Noisy Deep Ensemble: Accelerating Deep Ensemble Learning via Noise Injection
Shunsuke Sakai, Shunsuke Tsuge, Tatsuhito Hasegawa · 2025
Read Noisy Deep Ensemble: Accelerating Deep Ensemble Learning via Noise Injection by Shunsuke Sakai, Shunsuke Tsuge, Tatsuhito Hasegawa (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05677.txt.
Introduction
Shunsuke Sakai and co-authors speed up neural ensembles with noise in a 2025 paper. Ensembles train many nets and average their calls, which lifts accuracy but costs time in step with team size. Their swap trains one parent net to full fit, then shakes its weights many ways to spawn child nets.
Each child starts near a good spot yet can settle into a distinct local dip, which keeps the team diverse at a slice of the cost. Tests on two image sets with varied net shapes beat past fast ensemble tricks and matched plain ensembles on test accuracy. Code is shared in public for reuse.
Catalog record
| CREATOR | Shunsuke Sakai, Shunsuke Tsuge, Tatsuhito Hasegawa |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-05677
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Noisy Deep Ensemble: Accelerating Deep Ensemble Learning via Noise Injection Shunsuke Sakai, Shunsuke Tsuge, Tatsuhito Hasegawa Abstract Neural network ensembles is a simple yet effective approach for enhancing generalization capabilities. The most common method involves independently training multiple neural networks initialized with different weights and then averaging their predictions during inference. However, this approach increases training time linearly with the number of ensemble members. To address this issue, we propose the novel “ Noisy Deep Ensemble ” method, significantly reducing the training time required for neural network ensembles. In this method, a parent model is trained until convergence, and then the weights of the parent model are perturbed in various ways to construct multiple child models . This perturbation of the parent model weights facilitates the exploration of different local minima while significantly reducing the training time for each ensemble member. We evaluated our method using diverse CNN architectures on CIFAR-10 and CIFAR-100 datasets, surpassing conventional efficient ensemble methods and achieving test accuracy comparable to standard ensembles. Code is available at https://github.com/TSTB-dev/NoisyDeepEnsemble 1 Introduction Deep neural networks have achieved remarkable results in various fields, such as image recognition, natural language processing, and speech recognition. Their success can be attributed to their exceptionally high representation capacity. However, deep neural networks often lead to overfitting, particularly when the training data size is small. Consequently, the assessment of deep neural networks primarily focuses on their capability to effectively predict outcomes on new, unseen data. When multiple neural networks are trained independently, the randomness in the initialization of weights and the selection of mini-batches in SGD leads each network to learn different feature representations. Ensemble learning involves combining the predictions of multiple models to produce a more accurate prediction during inference. By integrating the predictions of multiple models, ensemble learning improves accuracy and uncertainty estimation performance compared to using a single model [ 5 , 17 , 19 ] . Ensemble learning is known to be more effective when each ensemble member possesses comparably high accuracy and makes mistakes on different samples [ 12 ] . Ensemble learning is a simple yet effective approach for improving generalization performance, but it faces the issue of training time increasing linearly with the number of ensemble members. This issue is especially significant for deep neural networks, as training just one model can require weeks to months, which restricts the practical use of ensemble learning. Additionally, there are reported scaling laws indicating that the performance of neural networks improves with increased data size, model size, and computation [ 14 ] . Furthermore, a phenomenon known as “Grokking”, where continued training beyond the point of sufficient loss reduction can enhance generalization performance, has been observed in specific problems [ 13 , 25 ] . These factors contribute to the increased training time for a single model, making realizing their ensembles more challenging. Several methods have been proposed to reduce the training time of ensemble learning [ 12 , 3 , 32 , 28 , 31 , 2 ] . These methods improve the efficiency of ensemble learning by sharing parameters among ensemble members [ 3 , 31 , 28 ] , constructing ensembles from the training process of a single model [ 12 ] , and speeding up training through the pre-training of a base model [ 20 , 32 ] . However, these methods have lower test accuracy than standard ensembles, resulting in significant performance differences. In typical neural network training, randomness is generally limited to weight initialization and mini-batch selection in Stochastic Gradient Descent (SGD). However, additional perturbations can also be introduced. A notable example includes the addition of noise to inputs, which can enhance model regularization and robustness on out-of-distribution data [ 1 , 26 , 6 ] . There is also extensive research on perturbing the weights of neural networks [ 1 , 35 , 2 , 34 , 36 , 29 , 23 ] . Weight perturbation can lead to effects such as regularization, exploration of parameter space, and robustness against adversarial perturbations. These approaches of introducing perturbations during neural network training are called Noise Injection. In this study, we specifically focus on perturbations to the weights. We aim to reduce ensemble members’ training time while achieving performance comparable to that of standard ensembles. In this study, we propose the Noisy Deep Ensemble, which utilizes noise injection to achieve this goal. In this method, a single model ( parent model ) is trained until it converges, then perturbs its weights to construct multiple ensemble members ( child models ). Since each ensemble member starts with reasonably good weights, they converge to nearby local minima after a short training period. Each ensemble member converges to different local minima compared to the original model, and their predictions exhibit diversity. Therefore, an ensemble of these models can achieve performance comparable to those trained independently at inference time. Fig. 1 shows the differences in the learning process between the proposed method and the existing method (Snapshot Ensemble [ 12 ] ). The Snapshot Ensemble converges a single model to multiple local solutions by resetting to a high learning rate after convergence. On the other hand, the proposed method adds perturbations to the initial convergence point and explores a wider range of the parameter space, not limited to optimization by SGD. Figure 1 : Difference in the learning process between Noisy Deep Ensemble and the existing method (Snapshot Ensemble [ 12 ] ). Noisy Deep Ensemble promotes the exploration of wider parameter space by not being limited to the optimization path of SGD through Noise Injection. Snapshot Ensemble consists of \mathcal{M}=\{\theta_{2},\theta_{3}\} , while Noisy Deep Ensemble consists of \mathcal{M}=\{\theta_{2}^{\prime},\theta_{3}^{\prime}\} . The contributions of this study are outlined as follows: 1. We developed a novel ensemble method called Noisy Deep Ensemble, which significantly reduces the training time of ensembles by perturbing the weights of neural networks. 2. We evaluated the proposed method with various CNN architectures on CIFAR-10 and CIFAR-100 datasets. Noisy Deep Ensemble achieved higher test accuracies than the traditional efficient ensemble method, Snapshot Ensemble, across various CNN architectures. 3. We conducted comprehensive experiments to investigate the diversity of predictions of each ensemble member and the impact of hyperparameters in the proposed method. 2 Background 2.1 Ensemble Learning Here, we describe ensemble learning for neural networks in a standard class classification setting. Consider a given training set \mathcal{D}=\{(\boldsymbol{x}_{i},y_{i})\}_{i=1}^{N} , where (\boldsymbol{x}_{i},y_{i}) represents the pair of the i -th data point and its corresponding label. N denotes the size of the training data, and y belongs to \{1,2,\dots,C\} , with C being the number of classes. Let f_{\boldsymbol{\theta}} represent a neural network with parameters \boldsymbol{\theta} . The optimization problem for a single neural network is formulated as follows: \boldsymbol{\theta}^{*}=\underset{\boldsymbol{\theta}}{\operatorname{argmin}}\,\mathbb{E}_{(\boldsymbol{x},y)\sim\mathcal{D}}\left[\mathcal{L}(f_{\boldsymbol{\theta}}(\boldsymbol{x}),y)\right] (1) Here, \mathcal{L} is the loss function given the neural network’s predicted probability distribution f_{\boldsymbol{\theta}}(\boldsymbol{x})\in\mathbb{R}^{C} and the correct label y , generally employing the cross-entropy loss function. In many cases, optimization, as depicted in Eq. ( 1 ), utilizes SGD. In ensemble learning, M neural networks which have different initial weights, f_{\boldsymbol{\theta}^{(1)}},f_{\boldsymbol{\theta}^{(2)}},\dots,f_{\boldsymbol{\theta}^{(M)}} , are independently optimized based on Eq. ( 1 ). At inference time, the ensemble prediction for a given test sample \boldsymbol{x} is determined by the average of the