Soft Specialists: α \alpha -Rényi Ensembles for Uncertainty-Aware LLM Post-Training

Paula Cordero-Encinar, Georgy Tyukin, Andrew B. Duncan · 2026

Read Soft Specialists: α \alpha -Rényi Ensembles for Uncertainty-Aware LLM Post-Training by Paula Cordero-Encinar, Georgy Tyukin, Andrew B. Duncan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27747.txt.

Introduction

Paula Cordero-Encinar, Georgy Tyukin, and Andrew Duncan propose soft specialists for uncertainty-aware language model post-training. Their Renyi variational system learns spreads over post-training weights, weighing sound solo models against partner specialist teams. Training cases route softly across team members, building deep skills while giving usable uncertainty scores across tasks.

Catalog record

CREATORPaula Cordero-Encinar, Georgy Tyukin, Andrew B. Duncan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE109979
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27747

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Soft Specialists: α \alpha -Rényi Ensembles for Uncertainty-Aware LLM Post-Training Paula Cordero-Encinar, Georgy Tyukin, Andrew B. Duncan Abstract Existing training approaches for large language models learn a single set of parameters, based on large volumes of data, which is typically heterogeneous, conflicting and often outright contradictory. As a result, the model is forced to compress conflicting goals, and inherent uncertainties into a single, averaged pattern of behaviour. We propose an α \alpha -Rényi variational framework for learning distributions over post-training parameters, offering an uncertainty-aware alternative to deep ensemble approaches. The resulting variational objective interpolates between classical variational Bayes and predictively oriented posterior learning, balancing between globally plausible individual models against systems of complementary specialists. We identify local stability criteria, demonstrating how model misspecification can make non-degenerate posterior spread locally favourable, manifesting contradictory or conflicting data as epistemic uncertainty. We apply our framework to LLM post-training, learning an ensemble of LoRA adapters attached to a shared, frozen base model, providing a scalable training procedure for both supervised fine-tuning and preference optimisation. Our approach enables training examples to be softly routed across ensemble members, promoting model specialisation and providing actionable uncertainty estimates across different tasks. 1 Introduction Large language models are typically adapted to downstream tasks, safety requirements and user preferences through post-training techniques such as Supervised Fine-Tuning (SFT) qi2024fine , Direct Preference Optimisation (DPO), rafailov2023direct or Reinforcement Learning from Human Feedback (RLHF) bai2022training ; dai2024safe ; ouyang2022training . In all these approaches, the output is a single adapted parameter vector, which must compress any uncertainty, ambiguity as well as and potentially conflicting preferences. This forces the learned parameter vector to absorb competing pressures that may not admit a satisfactory joint resolution. This issue has implications for alignment and high-stakes deployment. Ideally, a single post-trained model is expected to preserve useful base capabilities, remaining helpful under ambiguous requests while avoiding harmful behaviour anwar2024foundational ; ganguli2022red . However, these goals need not be simultaneously well represented by a single set of parameters: a model pushed toward safer operation may over-refuse benign requests or degrade capability, while a model tuned to remain helpful may become brittle under ambiguous or malicious prompts rottger2023xstest ; cui2025or . In essence, a single adapter post-training pipeline has no explicit mechanism for representing unresolved uncertainty. Bayesian methods offer one potential approach through their ability to quantify epistemic uncertainty. Bayesian neural networks, variational inference, Laplace approximations, dropout-based approximations, and sampling-based methods have all been used to represent epistemic uncertainty in deep learning ( arbel2026primer ; kingma2013auto ; blundell2015weight ; kingma2015variational ; mackay1992practical ; yang2024bayesian ; neal2012bayesian ; chen2014stochastic ; gal2016dropout ) . Bayesian methods have been deployed in the context of Language model safety, hu2026adaptive . However, the posterior distribution is intended to represent epistemic uncertainty over parameter values that are individually plausible explanations of the entirety of the observed data under the assumed model and prior. A ramification of this is that even mild forms of model misspecification can cause Bayesian posterior predictive distributions in performing arbitrarily badly ( grunwald2017inconsistency ) . Deep ensembles ( fort2019deep ; lakshminarayanan2017simple ) offer a practical second approach. By combining independently trained models at inference time, they often yield useful heuristic uncertainty, particularly when combined with post-hoc calibration strategies angelopoulos2025learn ; rivera2024conformal . As their diversity is usually induced through random initialisation, data order and optimisation noise, ensemble members are not coordinated as samples of a posterior distribution. However, introducing appropriate coordination during training allows these ensembles to effectively approximate a Bayesian posterior d2021repulsive ; wild2023rigorous . A third approach arises from generalised Bayesian methods grunwald2007suboptimal ; zellner1988optimal ; aitchison1975goodness ; bissiri2016general ; Knoblauch2022RoT , designed to better handle misspecification. Methods such as Gibbs posteriors ( jiang2008gibbs ; martin2022direct ) , tempered or fractional posteriors ( bhattacharya2019bayesian ; wenzel2020good ) , and safe-Bayesian approaches ( grunwald2012safe ) adapt posterior concentration to mitigate brittleness arising from the mismatch between the (unseen) data-generating distribution and the parametrised model family. These methods are highly relevant to training large-scale deep learning models, where the distribution is rarely clean or homogeneous. Nevertheless, they retain the same basic structure where posterior mass is assigned according to the averaged loss of individual parameter choices over the full dataset. Recent work on predictively oriented posteriors (PrO) ( McLatchie2025PrO ) builds upon the generalised Bayesian approach, by evaluating a distribution Q through the predictive quality of the mixture it induces. This is effective under heterogeneity or misspecification: no single parameter may adequately explain the data, but a mixture of complementary predictors might. However, the predictively oriented objective lies at the opposite end of the spectrum from generalised Bayesian inference. In practice, one may want both: individually meaningful and regularised parameter settings, but also enough predictive cooperation to represent conflicting or heterogeneous data. This motivates the need for a principled approach to interpolating between classical and prediction oriented posteriors. In this paper, we introduce an \alpha -Rényi variational framework for learning distributions over post-training parameters. For a distribution Q over parameters and a labelled example (x,y) , we define the per-example loss \ell_{\alpha}(Q;x,y)=-\frac{1}{\alpha}\log\int_{\Theta}p_{\theta}(y\mid x)^{\alpha}\,Q(d\theta),\qquad\alpha\in(0,1], where p_{\theta}(y\,|\,x) is the conditional likelihood assigned by the model with parameters \theta . As \alpha\to 0 , this recovers the classical variational Bayes data-fit term -\mathbb{E}_{\theta\sim Q}\big[\log p_{\theta}(y\mid x)\big] , while at \alpha=1 it becomes the negative log-likelihood of the predictive mixture. The parameter \alpha directly influences the geometry of the variational objective: small \alpha favours globally plausible individual models, while larger \alpha increasingly rewards complementary specialists whose utility emerges through aggregation. Our main application is distributional LLM post-training: instead of compressing heterogeneous or conflicting supervision into a single adapter, we learn a structured ensemble of LoRA adapters hu2022lora whose diversity represents unresolved epistemic uncertainty. Our parameter space \Theta characterises low rank perturbations of a common frozen base model. The resulting objective induces soft responsibilities across particles where each training example is routed towards those adapters that currently explain it well. In this way, specialisation emerges naturally from the variational objective itself. Figure 1 : \alpha -Rényi flow training for LLM ensembles. A single frozen base model W_{0} is shared across M trainable LoRA particles. For each minibatch example, the particles produce sequence log-likelihoods s_{i,b} which are coupled through the objective. The resulting responsibilities w_{i,b}^{(\alpha)} softly route examples towards particles that explain them well, inducing specialisation for \alpha>0 . The final predictor is the induced mixture p_{Q^{M}} . Our contributions are: 1. We introduce an \alpha -indexed variational objective over distributions of post-training parameters, interpolating between classical variational Bayes and predictively oriented posterior learning. 2. We analyse the local stability of Dirac posteriors and show how misspecification can make non-degenerate posterior spread favourable. This behaviour is governed by an information-geometric matrix that underpins the variational objective. 3. We derive finite-particle training objectives for supervised fine-tuning and preference optimisation, yielding soft responsibility-based routing across LoRA adapters in LLMs, see Figure 1 . 4. We demonstrate how the resulting ensemble can represent localised epistemic uncertainty under misspecification and contamination. We provide a more in-depth discussion of related work in App. E . 2 Setup We begin by recalling the variational formulations underlying classical Bayesian inference and predictively oriented posteriors, and then introduce the \alpha -indexed family of objectives studied in

Read the complete text