Beyond Mixtures and Products for Ensemble Aggregation: A Likelihood Perspective on Generalized Means
Raphaël Razafindralambo, Rémy Sun, Frédéric Precioso, Damien Garreau, Pierre-Alexandre Mattei · 2026
Read Beyond Mixtures and Products for Ensemble Aggregation: A Likelihood Perspective on Generalized Means by Raphaël Razafindralambo, Rémy Sun, Frédéric Precioso, Damien Garreau, Pierre-Alexandre Mattei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04204.txt.
Introduction
Razafindralambo and others go beyond mixtures and products for ensemble aggregation from a likelihood perspective. They show the regime r between zero and one is the only range guaranteeing systematic gains relative to individual spreads. This justifies the reliability and broad real use of linear and shape pooling. In contrast, aggregation rules outside that range can fail to deliver steady gains, with stated counterexamples.
Catalog record
| CREATOR | Raphaël Razafindralambo, Rémy Sun, Frédéric Precioso, Damien Garreau, Pierre-Alexandre Mattei |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58367 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.04204
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Mixtures and Products for Ensemble Aggregation: A Likelihood Perspective on Generalized Means Raphaël Razafindralambo, Rémy Sun, Frédéric Precioso, Damien Garreau, Pierre-Alexandre Mattei Abstract Density aggregation is a central problem in machine learning, for instance when combining predictions from a Deep Ensemble. The choice of aggregation remains an open question with two commonly proposed approaches being linear pooling (probability averaging) and geometric pooling (logit averaging). In this work, we address this question by studying the normalized generalized mean of order r ∈ ℝ ∪ { − ∞ , + ∞ } r\in\mathbb{R}\cup\{-\infty,+\infty\} through the lens of log-likelihood, the standard evaluation criterion in machine learning. This provides a unifying aggregation formalism and shows different optimal configurations for different situations. We show that the regime r ∈ [ 0 , 1 ] r\in[0,1] is the only range ensuring systematic improvements relative to individual distributions, thereby providing a principled justification for the reliability and widespread practical use of linear ( r = 1 r=1 ) and geometric ( r = 0 r=0 ) pooling. In contrast, we show that aggregation rules with r ∉ [ 0 , 1 ] r\notin[0,1] may fail to provide consistent gains with explicit counterexamples. Finally, we corroborate our theoretical findings with empirical evaluations using Deep Ensembles on image and text classification benchmarks. 1 Introduction Figure 1 : Only intermediate powers r , particularly r\in[0,1] , yield consistent NLL improvements, whereas extreme values can degrade performance. Top: aggregated densities \overset{\circ}{p}_{2} obtained from two Gaussian experts p^{(1)} and p^{(2)} for various r . Small r concentrates mass in a single region, while large positive r preserves the bimodal structure of the experts. Bottom: NLL evaluated on samples y\sim\mathcal{N}(0,4^{2}) (purple ticks indicate test samples). In this setting, negative r (min-type behavior) underperforms the average individual NLL (dashed line), whereas positive r achieves the lowest values. Our theory ( Theorem 3.1 ) identifies r\in[0,1] as the provably reliable interval. See Appendix B for different behaviors. Modern machine learning (ML) increasingly relies on collections of probabilistic models rather than a single monolithic predictor ( 32 ; 33 ) . This naturally raises the question of how to integrate several probability distributions into a single coherent distribution ( 19 ; 21 ) . Distribution integration arises in many settings in ML, including ensembling classifier outputs ( 30 , Section 5) , Mixup training techniques ( 16 ) , Bayesian inference ( 8 ) , the aggregation of Gaussian process experts ( 34 ; 11 ) , mixture-of-experts architectures in large language models ( 33 ; 6 ) , and generative modeling ( 35 ; 39 ) . Two canonical approaches are particularly widespread. The first consists in forming a mixture of distributions ( 27 ; 28 ) , corresponding to an arithmetic average of densities. The second relies on multiplicative aggregation, where distributions are combined through a normalized product ( 26 ) . These methods offer contrasting properties ( 46 ) : mixtures act as a logical ‘OR’, capturing heterogeneity by pooling the supports, whereas multiplicative models act as a logical ‘AND’, sharpening the density where components overlap and thus favoring regions of consensus. This viewpoint has is recurrent in compositional generative modeling ( 15 ; 35 ) . More generally in ML, the choice between both has been a fundamental and practical question; for instance, 5 empirically compare logit and probability averaging in multi-class classification. A key theoretical motivation for mixtures lies in their variance-reduction effect, allowing the collective evaluation to outperform the average individual ones ( 24 , Chapter 15; 37 ), a phenomenon related to the broader “wisdom of crowds” principle ( 42 ) . When predictors are diverse, individual errors tend to compensate for one another, improving collective performance. Beyond the classical arithmetic and geometric means, several works have sought to go beyond and generalize the notion of averaging probability distributions ( 21 ; 2 ) to reveal new aggregation properties and fundamental trade-offs underlying different pooling rules ( 17 ) . These efforts have led to continuous families of pooling operators that extend standard averages and include a broad spectrum of aggregation rules, ranging from harmonic and arithmetic means to more extreme operators such as max- or min-based pooling. In particular, 2 introduces an \alpha -family of operators parameterized by a free parameter \alpha , and establishes optimality properties with respect to \alpha -divergences for each member of this family. We examine whether density aggregation through generalized means of order r\in\overline{\mathbb{R}}\coloneqq\mathbb{R}\cup\{-\infty,+\infty\} ( 23 ) provides improved aggregation rules for model ensembling. A natural criterion in ML to assess a aggregated distributions is the data log-likelihood, as it directly measures how well an aggregated model explains observed data. This objective is central to modern learning procedures, as it underlies cross-entropy loss minimization in supervised classification and KL-based objectives in broader settings such as generative modeling and variational inference ( 4 , Chapter 10; 43 ; 22 , Chapter 5). This perspective therefore motivates both a theoretical and empirical investigation into whether variance reduction extends to generalized pooling operators and improve log-likelihood over individual models. In this work, we show that our defined normalized generalized mean of order r yields a reliable aggregation rule in terms of log-likelihood, with theoretical guarantees and empirical evidence that gains over individual distributions are consistently observed for r\in[0,1] . The idea is summarized in Figure 1 , which illustrates how the shape of normalized generalized mean density varies with r along with the unreliable behavior of negative log-likelihood induced by negative r values. Our main contribution lies in analyzing, through the lens of log-likelihood, the generalized mean of order r\in\overline{\mathbb{R}} which defines valid densities ( Section 3.1 ) with properties determined by r ( Sections 2 and 3 ). More precisely, this translates into the following results: • We identify in Section 3.2 a theoretical regime r\in[0,1] ensuring consistent log-likelihood benefits for any set of densities, uniformly across all data points. This regime includes the arithmetic and geometric means as boundary cases, explaining their practical successes. • We characterize in Section 3.3 failure points outside this interval based on agreement and disagreement of density values, highlighting distinct behaviors for r<0 and r>1 . This explains the lack of reliability outside [0,1] compared to the guarantees within the interval, as observed in Figure 1 . • We empirically corroborate the theoretical findings through Deep Ensemble experiments on classification tasks on vision and sentiment analysis ( Section 4 ). We also observe that the optimal r is non trivial but close to the [0,1] range. 2 Background and notations on generalized means We consider a data space \mathcal{X} equipped with a reference measure \mathrm{d}\mathbf{x} . We focus on probability distributions that admit densities with respect to \mathrm{d}\mathbf{x} . We denote by \mathcal{P} the set of probability densities p:\mathcal{X}\to\mathbb{R}_{\geq 0} satisfying \int_{\mathcal{X}}p(\mathbf{x})\,\mathrm{d}\mathbf{x}=1 . Given k\geq 1 densities p^{(1)},\dots,p^{(k)}\in\mathcal{P} , this work studies the construction of an aggregated density \overset{\circ}{p} obtained by combining these distributions through generalized averaging operators, followed in most of the cases by a normalization step to ensure that \overset{\circ}{p}\in\mathcal{P} . We first introduce formal notation for the mixture and multiplicative aggregation, as they constitute the most popular reference aggregations for our results. We then turn