Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
Ngoc-Quan Pham, Tuan Truong, Quyen Tran, Tan Nguyen, Dinh Phung, Trung Le, Ngoc-Quan Pham, Tuan Truong, Quyen Tran, Tan Nguyen, Dinh Phung, Trung Le · 2025
Read Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models by Ngoc-Quan Pham, Tuan Truong, Quyen Tran, Tan Nguyen, Dinh Phung, Trung Le, Ngoc-Quan Pham, Tuan Truong, Quyen Tran, Tan Nguyen, Dinh Phung, Trung Le (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07247.txt.
Introduction
Ngoc-Quan Pham, Tuan Truong, Quyen Tran, Tan Nguyen, Dinh Phung, and Trung Le fine-tune foundation models with ensembles whose members are pushed to disagree usefully, in a 2025 paper. An ensemble of models beats a single one when its members err differently; sameness wastes the trick. Standard Bayesian fine-tuning lets its sampled models huddle together.
Their framework, IBDR, adds interactions between the sampled models — the particles — so diversity is rewarded during training, while a robustness term guards against data shift. On a thousand-example adaptation benchmark and a reasoning task, IBDR beat the rival methods tested. Diverse ensembles that stay robust could make small-data fine-tuning of giant models safer for medicine and law.
Catalog record
| CREATOR | Ngoc-Quan Pham, Tuan Truong, Quyen Tran, Tan Nguyen, Dinh Phung, Trung Le, Ngoc-Quan Pham, Tuan Truong, Quyen Tran, Tan Nguyen, Dinh Phung, Trung Le |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-07247
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models Ngoc-Quan Pham, Tuan Truong, Quyen Tran, Tan Nguyen, Dinh Phung, Trung Le, Ngoc-Quan Pham, Tuan Truong, Quyen Tran, Tan Nguyen, Dinh Phung, Trung Le Abstract We introduce Interactive Bayesian Distributional Robustness (IBDR), a novel Bayesian inference framework that allows modeling the interactions between particles, thereby enhancing ensemble quality through increased particle diversity. IBDR is grounded in a generalized theoretical framework that connects the distributional population loss with the approximate posterior, motivating a practical dual optimization procedure that enforces distributional robustness while fostering particle diversity. We evaluate IBDR’s performance against various baseline methods using the VTAB-1K benchmark and the common reasoning language task. The results consistently show that IBDR outperforms these baselines, underscoring its effectiveness in real-world applications. † † affiliationnotice: Equal contribution 1 Introduction Tackling uncertainty remains one of the most challenging problems in deep learning. This uncertainty arises from the real world’s inherent randomness and noisy, complex data. To address this challenge, Bayesian inference offers a powerful solution by providing a probabilistic framework that enables reasoning under uncertainty. A particularly practical approach within Bayesian inference involves particle sampling techniques, which are well-suited for scenarios requiring multiple models. Notable methods include Stochastic Gradient Langevin Dynamics (SGLD) ( Welling & Teh 2011a ) , Hamiltonian Monte Carlo (HMC) ( Neal 1996 ) , Stochastic Gradient HMC (SGHMC) ( Chen et al. 2014 ) , and Stein Variational Gradient Descent (SVGD) ( Liu & Wang 2016 ) . However, a key limitation of these methods is the computational and storage overhead associated with maintaining multiple models, especially for large-scale architectures. To mitigate this issue, variational inference (VI) techniques have been developed to approximate the true posterior distribution with tractable approximations known as variational posteriors. These methods optimize a variational lower bound, significantly reducing computational costs while effectively capturing uncertainty. Key contributions in this domain include Kingma & Welling 2013 ; Kingma et al. 2015 , and Blundell et al. 2015 , who extended Gaussian variational posterior approximations to neural networks. Additionally, Gupta & Nagar 2018 introduced greater flexibility in posterior approximations, enhancing the applicability of VI techniques. Traditional Bayesian inference faces a limitation: the independent sampling of model particles from the posterior often fails to capture interactions and can lead to particle collapse into a single mode. To overcome this, we introduce a novel framework, Interactive Bayesian Distributional Robustness (IBDR), which establishes a joint distribution over independent posteriors and integrates a divergence loss to model this interaction and encourage particle diversity. To further enhance robustness, we leverage Wasserstein-based distributional robustness optimization ( Gao & Kleywegt 2023 ; Blanchet et al. 2019 ; Sinha et al. 2018 ) , as formalized in Theorem 4.1 . This extension generalizes distributional robustness optimization (DRO) to accommodate more general risk functions and product joint distributions, ensuring a balance between robustness and diversity. The resulting framework improves ensemble performance by preventing mode collapse while maintaining computational efficiency. To demonstrate the effectiveness and versatility of our framework, we conduct experiments on the image classification task with ViT ( Dosovitskiy et al. 2020 ) and the commonsense reasoning task on LLaMA-2 ( Touvron et al. 2023 ) , demonstrating significant improvements over baseline methods. Our contributions are as follows: (i) We introduce a novel Bayesian framework that explicitly models interactions between model particles during training . Leveraging distributional robustness, we provide a theoretical analysis of this interactive framework, generalizing existing results to product distribution spaces. (ii) Building on this analysis, we propose a practical framework that ensures both ensemble diversity and distributional robustness . We validate our approach by fine-tuning ViT ( Dosovitskiy et al. 2020 ) on the VTAB-1K classification benchmark and LLaMA-2 ( Touvron et al. 2023 ) on a commonsense reasoning task, demonstrating significant performance improvements. 2 Related Works 2.1 Bayesian Neural Networks Variational Inference. This approach estimates the posterior distribution by selecting a specific approximation and refining a variational lower bound. Graves 2011 introduced a Gaussian variational posterior for neural network weights, further developed in Kingma & Welling 2013 ; Kingma et al. 2015 ; Blundell et al. 2015 using the reparameterization trick for deep latent variable models. Extensions to improve posterior flexibility include Caterini et al. 2021 , who used normalizing flows, and Louizos & Welling 2017 and Gupta & Nagar 2018 , who employed a matrix-variate Gaussian. Other studies have explored structured variational Gaussian posteriors, such as Kronecker-factored approximations ( Rossi et al. 2020 ; Eschenhagen et al. 2023 ) , Gaussian score matching ( Modi et al. 2023 ) , and non-centered or rank-1 parameterizations ( Ghosh et al. 2018 ; Dusenberry et al. 2020 ) . Hybrid approaches combining Variational Inference and Markov Chain Monte Carlo (MCMC) have gained popularity, such as Alexos et al. 2022 , which used latent variable averaging to speed up mixing. Applications of Variational Inference to modern architectures, like Vision Transformers ( Zhang et al. 2021 ) , highlight its role in uncertainty-aware deep learning. Markov Chain Monte Carlo (MCMC). This approach allows sampling multiple models from the posterior distribution, commonly used for neural network inference via Hamiltonian Monte Carlo (HMC) ( Neal 1996 ) . However, HMC requires full gradient estimation, which is computationally expensive. Stochastic Gradient HMC (SGHMC) ( Chen et al. 2014 ) uses stochastic gradients for scalability and solution exploration. Alternatively, Stochastic Gradient Langevin Dynamics (SGLD) ( Welling & Teh 2011b ) applies Langevin dynamics in the stochastic gradient setting. Stein Variational Gradient Descent (SVGD) ( Liu & Wang 2016 ) uses particles to approach the posterior distribution. MCMC methods, unlike variational methods, can be costly due to the need to store multiple models. Nonetheless, SGHMC, SGLD, and SVGD asymptotically sample from the posterior with infinitely small step sizes. 2.2 Flat Minimizers Flat minimizers improve neural network generalization by helping models find broader local minima, making them more robust to training-test set differences ( Jiang et al. 2020 ; Petzka et al. 2021 ; Nguyen et al. 2023a ) . The link between generalization and minimum width has been explored both theoretically and empirically ( Hochreiter & Schmidhuber 1994 ; Neyshabur et al. 2017 ; Dinh et al. 2017 ; Fort & Ganguli 2019 ) . Various methods for finding flat minima have been proposed ( Pereyra et al. 2017 ; Izmailov et al. 2018 ; Foret et al. 2021 ; Nguyen et al. 2023a ) . Studies by Keskar et al. 2017 , Jastrzebski et al. 2017 , and Wei et al. 2020 examine how batch size, learning rate, gradient covariance, and dropout affect flatness. Some methods also use regularization terms in the loss function to promote wide local minima ( Pereyra et al. 2017 ; Zhang et al. 2018 ; Zhang et al. 2019 ) , such as softmax entropy penalties ( Pereyra et al. 2017 ) and distillation losses ( Zhang et al. 2018 ; Zhang et al. 2019 ) . Among the flat minimizers, Sharpness-Aware Minimization (SAM) ( Foret et al. 2021 ) has gained attention for its effectiveness and scalability across tasks like domain generalization ( Cha et al. 2021 ; Wang et al. 2023 ; Zhang et al. 2023 ) , federated learning (