On Defining Neural Averaging

Su Hyeong Lee, Richard Ngo · 2025

Read On Defining Neural Averaging by Su Hyeong Lee, Richard Ngo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-14832.txt.

Introduction

Su Hyeong Lee and Richard Ngo ask what it means to average neural networks in this 2025 paper. One popular trick, called model soup, simply averages the weights of several trained models and often works better than any single one. The authors want a principled account of why, and a more flexible method.

They recast soup as one case of a broader scheme that needs no training data, only final weights. In their scheme, the gaps between models act like update steps that steer a merged model. Tests show the averaged models beat both single experts and plain soup, especially on data unlike the training sets. The work offers a general recipe for merging models after training.

Catalog record

CREATORSu Hyeong Lee, Richard Ngo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-14832

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On Defining Neural Averaging Su Hyeong Lee, Richard Ngo Abstract What does it even mean to average neural networks? We investigate the problem of synthesizing a single neural network from a collection of pretrained models, each trained on disjoint data shards, using only their final weights and no access to training data. In forming a definition of neural averaging, we take insight from model soup, which appears to aggregate multiple models into a singular model while enhancing generalization performance. In this work, we reinterpret model souping as a special case of a broader framework: Amortized Model Ensembling (AME) for neural averaging, a data-free meta-optimization approach that treats model differences as pseudogradients to guide neural weight updates. We show that this perspective not only recovers model soup but enables more expressive and adaptive ensembling strategies. Empirically, AME produces averaged neural solutions that outperform both individual experts and model soup baselines, especially in out-of-distribution settings. Our results suggest a principled and generalizable notion of data-free model weight aggregation and defines, in one sense, how to perform neural averaging. ${}^{1}$ ${}^{1}$ footnotetext: University of Chicago, Correspondence to: [email protected] ${}^{2}$ ${}^{2}$ footnotetext: Independent 1 Introduction Consider the following problem. Suppose we have N neural networks that are domain experts with differing capabilities, perhaps due to being trained on different shards of a larger data stream. Therefore, each expert is knowledgeable about its own shard, but is largely oblivious or out-of-domain on others. Now, remove all access to the data. We are only given the model weights and nothing else. In this setup, can we synthesize a single model that preserves the knowledge embedded in each expert, using only these final weights and without performing any additional data-driven training? At a high level, our goal is to construct a form of “average” over these domain experts–a model that aggregates their capabilities and ideally retains their individual performance. But what does it even mean to perform neural averaging? What properties should such an average of neural nets satisfy? In this work, we propose a framework for answering these questions. We begin by identifying a characteristic for neural network averaging. Neural nets are often highly specialized: they memorize the data they are trained on and tend to generalize less well outside of it [ 1 , 2 ] . If we succeed in constructing an averaged model, then, even without access to any training data, we expect it to retain the memory of each expert. That is, such a model should perform better on each expert’s original training distribution than it would on a random validation set, despite never having seen any of that data itself. Is such a notion of neural averaging even realizable? A naive approach might be to create a rudimentary mixture model of neural nets: at inference time, sampling from a multinomial distribution over the N experts and using the selected model to make predictions. In expectation, this creates a kind of functional ensemble. However, this strategy is deeply unsatisfying. It requires constantly maintaining access to all N models, which is inefficient, and does nothing to address the fact that each expert is blind to most of the overall data distribution. Deploying any single expert is thus inherently brittle. A more promising direction comes from the model soup literature [ 3 , 4 ] . These works show that averaging the weights of multiple independently trained models, e.g., trained with different data orderings or hyperparameters, can yield a new model with stronger generalization. Not only does this suggest that weight-space averaging can be meaningful, but also that it can outperform the individual models being averaged. This empirical success motivates our central question: can we extend model soup to a useful notion of neural averaging? Our approach is guided by the following intuition. During training, each minibatch gradient update pushes a model through a high-dimensional, non-convex optimization landscape, nudging its weights to encode the statistical features of the data [ 5 , 6 ] . After enough iterations, the model converges to a local minimum that reflects the distribution it was trained on. We hypothesize that these converged weights implicitly contain enough directional signal that, by computing distance vectors in weight space, we can construct pseudogradients that continue to point toward the expert’s learned minima. If this is true, then we may be able to define a meta-optimization algorithm, operating directly on the expert weights, that mimics the effect of data-driven training. Crucially, this approach would require no actual access to data, relying only on the geometry of the converged models. Moreover, if we employ adaptive optimizers known to accelerate convergence in non-convex settings [ 7 , 8 , 9 ] , we may be able to reach high-quality solutions more efficiently [ 10 , 11 ] . In this paper, we introduce a novel ensembling algorithm that aggregates multiple pretrained neural networks into a single model, without using any data or retraining. Our method constructs updates in weight space by treating model differences as pseudogradients, enabling optimization toward a fused model that integrates the knowledge of each expert. We show that the state-of-the-art model soup strategy [ 3 , 4 ] emerges as a special case of our framework, corresponding to a single-step gradient descent with a fixed step size. By reframing ensembling as data-free meta-optimization, we offer a new perspective on what it means to “average” in neural network space, that is, to perform neural averaging. 2 Related Literature 2.1 Ensembling Methods in Machine Learning Traditional ensembling techniques seek to combine multiple models to enhance overall performance, leveraging the strengths of each individual model. Bootstrap Aggregating [ 12 ] trains multiple instances of the same model on different subsets of the training data created through random sampling with replacement. This approach can help reduce variance and mitigate overfitting [ 13 , 14 ] . Boosting [ 15 , 16 ] , on the other hand, builds models sequentially, where each new model attempts to correct the errors of its predecessors. Voting [ 17 ] combines the predictions of multiple models by taking a majority vote for classification tasks or averaging for regression tasks. Although these methods are effective in improving model robustness and predictive accuracy, bagging and boosting are expensive to implement for DNN architectures, and voting requires the maintenance of all selected workers in parallel, resulting in an n -fold increase in run-time compute for n ensembled learners during deployment [ 3 ] . In contrast, our approach consolidates all n sub-optimal learners into a single, powerful learner. 2.2 Model Ensembling in Deep Learning & Model Soup The idea of combining multiple neural network weights, either through weighted linear averaging or exponential moving averaging, is a well-studied approach [ 18 , 19 , 20 , 21 , 22 , 23 , 24 ] . For instance, Snapshot Ensembling [ 25 ] captures multiple local minima in the optimization landscape over a single training run, by encouraging transitory local convergence via a periodic warm-up-ramp-down learning rate schedule. The state dictionary at each minima is saved, and the last m snapshots are linearly averaged to form a robust predictive model, improving generalization without additional training cost. Additional works have shown that when averaging

Read the complete text