Divergent Ensemble Networks: Enhancing Uncertainty Estimation with Shared Representations and Independent Branching

A. Chandorkar * and A. Kharbanda * · 2024

Read Divergent Ensemble Networks: Enhancing Uncertainty Estimation with Shared Representations and Independent Branching by A. Chandorkar * and A. Kharbanda * (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01193.txt.

Introduction

Arnav Kharbanda and Advait Chandorkar present a 2024 paper on a cheaper way to build ensembles of neural networks. Ensembles, groups of models whose answers are combined, improve accuracy and give a sense of uncertainty. Training many full networks separately wastes effort, since each relearns the same basics. The pair proposes the Divergent Ensemble Network, which shares early layers across all branches.

After the shared start, each branch splits off and trains on its own, keeping the variety that makes ensembles work. The shared-to-branching design cuts repeated parameters while keeping the spread of views. That makes the ensemble leaner and easier to scale. The design targets tasks where knowing uncertainty matters, such as medical or safety uses. The paper sets out the design and its rationale for later testing.

Catalog record

CREATORA. Chandorkar * and A. Kharbanda *
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01193

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Divergent Ensemble Networks: Enhancing Uncertainty Estimation with Shared Representations and Independent Branching A. Chandorkar * and A. Kharbanda * Abstract Ensemble learning has proven effective in improving predictive performance and estimating uncertainty in neural networks. However, conventional ensemble methods often suffer from redundant parameter usage and computational inefficiencies due to entirely independent network training. To address these challenges, we propose the Divergent Ensemble Network (DEN), a novel architecture that combines shared representation learning with independent branching. DEN employs a shared input layer to capture common features across all branches, followed by divergent, independently trainable layers that form an ensemble. This shared-to-branching structure reduces parameter redundancy while maintaining ensemble diversity, enabling efficient and scalable learning. † † footnotetext: * These authors have contributed equally. 1 Introduction In controlled conditions and simulations, contemporary deep-learning networks have shown remarkable performance. However, when confronted with unexplored data, unforeseen fluctuations, or noisy inputs, its reliability frequently declines in real-world circumstances, potentially resulting in predictions that are not accurate. This raises concerns about the dependability of these networks in practical applications. [ 6 ] . To address this challenge, leveraging uncertainty estimation was proposed as a solution to enhance their robustness and ensure real-world applicability. Neural networks struggle with quantifying predictive uncertainty and often generate overly confident predictions. Such overconfident incorrect predictions can lead to detrimental results [ 8 ] . In deep learning, there are two primary types of uncertainties: aleatoric uncertainty and epistemic uncertainty . • Aleatoric uncertainty originates from the inherent noise or randomness in the data. This type of uncertainty cannot be reduced by altering the model, as it is an intrinsic characteristic of the data itself [ 5 ] . • Epistemic uncertainty arises from a lack of knowledge or information regarding the model. Unlike aleatoric uncertainty, epistemic uncertainty can be mitigated by improving the model design or gathering additional data [ 5 ] . For aleatoric uncertainty , one approach to quantifying it is through the entropy of the predictive distribution. This allows us to measure the uncertainty inherent in the data [ 13 ] . On the other hand, epistemic uncertainty is typically quantified by assessing the variability between different models, which reflects how much our model predictions vary due to a lack of knowledge. Several approaches have been proposed to handle these uncertainties: • Bayesian Neural Networks (BNNs) treat the network weights as random variables, thereby modeling uncertainty by learning distributions over the weights [ 10 ] . • Monte Carlo Dropout (MC Dropout) approximates the posterior distribution by applying dropout during inference, enabling uncertainty estimation without explicitly Bayesian training [ 3 ] . • Ensemble Methods , such as Bagging, reduce uncertainty by combining predictions from multiple models. The diversity of the models helps in reducing the overall uncertainty [ 1 ] . • Bootstrapping involves training multiple models on resampled subsets of the data. This method estimates uncertainty by evaluating the variance between models trained on different data subsets [ 7 ] . • Deep Ensembles combine the predictions of multiple models to estimate uncertainty by analyzing the variance of their predictions [ 4 ] . These methods offer distinct approaches to quantifying and reducing uncertainty in deep learning models, helping to enhance model reliability and performance. The ensemble’s biggest challenge is reducing redundancy while preserving its diversity. While techniques like Monte Carlo Dropout (MC Dropout) approximate ensembles by stochastic regularization, they may compromise predictions’ independence [ 14 ] . Similarly, approaches like Batch Ensemble aim to reduce computation but need help with flexibility in capturing model variance. This trade-off between computational efficiency and one of the primary obstacles to ensemble learning is still ensemble diversity. To address these challenges, we propose the Divergent Ensemble Network (DEN), a novel architecture that combines the benefits of shared representation learning and ensemble diversity. DEN features a shared input layer that captures a common representation across all ensemble members, followed by independent branching layers that train separately to maintain prediction diversity. This architecture balances computational efficiency with predictive robustness, making it suitable for scenarios where both accuracy and uncertainty estimation are critical. In this study, we assess DEN across various tasks, emphasizing its ability to estimate uncertainty while preserving predictive accuracy. Our findings indicate that DEN surpasses conventional ensembles in computational efficiency and achieves similar or superior uncertainty estimation. Additionally, its scalable nature allows for adaptability in a wide range of applications, including those that require real-time predictions. The remainder of this paper is structured as follows: Section 2 provides an overview of the DEN architecture and its training approach. Section 3 showcases the findings, while Section 4 explores the applications and constraints, and Section 5 offers concluding remarks. 2 Architecture 2.1 Problem setup and High-level summary We assume that the training dataset \mathcal{D} consists of N Independent and Identically Distributed (i.i.d) data points. For classification problems, the label is assumed to be one of K classes, that is y\in\{1,\dots,K\} . For regression problems, the label is assumed to be real-valued, that is y\in\mathbb{R} . Given the input features x , we use a neural network to model the probabilistic predictive distribution p_{\theta}(y|x) over the labels, where \theta are the parameters of the neural network. The Divergent Ensemble Network (DEN) uses a shared-to-branching architecture for ensemble learning. A common input layer processes shared features, followed by a single shared representation layer that extracts fundamental features. After this, the network diverges into multiple independent branches, each trained with distinct parameters to have independent predictions. These branches operate on the shared features and produce independent outputs. This model thus provides independent predictions while also giving faster predictions. Figure 1: Visual representation of the proposed neural network architecture. The number of branches and the nodes in each branch are hyper-parameters and varies from problem to problem. The shared layer in DEN ensures efficient parameter usage by processing input features once, reducing redundancy. Meanwhile, the divergent branches maintain the independence necessary for effective ensemble learning. By decoupling shared representation from individual predictions, DEN achieves a unique balance between computational efficiency and predictive variance. 2.2 Shared Input and Common Representation Layer The architecture is divided into several independent networks after the shared layers. Each branch processes the shared representation further, utilizing distinct weights and biases to enable independent predictions. These branches act as independent ensemble members while benefiting from the standard input processing. Each branch consists of multiple fully connected layers, with the activation functions being themselves considered as a hyperparameter. Dropout layers are also incorporated, as they can improve accuracy by reducing overfitting. However, increasing the number of branches introduces trade-offs in terms of computational resources. While adding more branches can potentially improve model performance by enhancing ensemble diversity, it also requires more memory and computational power, as each additional branch adds extra parameters and layers to the model. This results in higher processing time during both training and inference stages, which can be particularly costly in resource-constrained environments. Therefore, a balance must be struck between the number of branches and available computational resources to ensure efficient use of the system. 2.3 Divergent Branches for Independent Predictions The architecture is divided into several independent networks after the shared layers. Each branch processes the

Read the complete text