Making Foundation Models Probabilistic via Singular Value Ensembles

Mehmet Ozgur Turkoglu, Dominik J. Mühlematter, Alexander Becker, Konrad Schindler, Helge Aasen · 2026

Read Making Foundation Models Probabilistic via Singular Value Ensembles by Mehmet Ozgur Turkoglu, Dominik J. Mühlematter, Alexander Becker, Konrad Schindler, Helge Aasen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22068.txt.

Introduction

Mehmet Ozgur Turkoglu and co-authors make foundation models less cocky. These large pretrained models perform remarkably across tasks, but their predictions are often overconfident and poorly calibrated. The standard cure, training an ensemble of independent models, costs far too much at foundation scale.

The team's 2026 preprint builds ensembles from singular value pieces of a single model, creating varied members without full retraining. Spread among those members gives an estimate of epistemic uncertainty, the model's own ignorance. Calibrated uncertainty matters wherever a model's confidence drives action, from medical triage to financial risk. Because a system that knows when it does not know can hand off to a human in time.

Catalog record

CREATORMehmet Ozgur Turkoglu, Dominik J. Mühlematter, Alexander Becker, Konrad Schindler, Helge Aasen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22068

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Making Foundation Models Probabilistic via Singular Value Ensembles Mehmet Ozgur Turkoglu, Dominik J. Mühlematter, Alexander Becker, Konrad Schindler, Helge Aasen Abstract Foundation models have become a dominant paradigm in machine learning, achieving remarkable performance across diverse tasks through large-scale pretraining. However, these models often yield overconfident, uncalibrated predictions. The standard approach to quantifying epistemic uncertainty, training an ensemble of independent models, incurs prohibitive computational costs that scale linearly with ensemble size, making it impractical for large foundation models. We propose Singular Value Ensemble (SVE), a parameter-efficient implicit ensemble method that builds on a simple, but powerful core assumption: namely, that the singular vectors of the weight matrices constitute meaningful subspaces of the model’s knowledge. Pretrained foundation models encode rich, transferable information in their weight matrices. If the singular vectors are indeed meaningful (orthogonal) ”knowledge directions”. To obtain a model ensemble, we modulate only how strongly each direction contributes to the output. Rather than learning entirely new parameters, we freeze the singular vectors and only train per-member singular values that rescale the contribution of each direction in that shared knowledge basis. Ensemble diversity emerges naturally as stochastic initialization and random sampling of mini-batches during joint training cause different members to converge to different combinations of the same underlying knowledge. SVE achieves uncertainty quantification comparable to explicit deep ensembles while increasing the parameter count of the base model by less than 1%, making principled uncertainty estimation accessible in resource-constrained settings. We validate SVE on NLP and vision tasks with various different backbones and show that it improves calibration while maintaining predictive accuracy. 1 Introduction Machine learning models are increasingly deployed in high-stakes domains where incorrect predictions can have severe consequences, e.g., medical diagnosis, autonomous driving, financial risk assessment, and agricultural decision support. In such applications, a model must not only be accurate but also know when it does not know . A prediction delivered with undue confidence can be more dangerous than no prediction at all: an overconfident misdiagnosis may delay treatment, while an overconfident crop disease prediction may lead to unnecessary pesticide application or, worse, untreated outbreaks that devastate yields. Uncertainty in predictions arises from two fundamentally different sources. Aleatoric uncertainty reflects irreducible noise inherent in the data, e.g., measurement error, label ambiguity, or stochastic processes. Epistemic uncertainty , in contrast, stems from the model’s limited knowledge: regions of input space where training data is sparse, or where multiple hypotheses are consistent with observations. Epistemic uncertainty is reducible in principle; it quantifies what the model does not know and signals when predictions should not be trusted ( 29 ) . Modern deep neural networks, despite their remarkable predictive power, are notoriously poor at quantifying their own uncertainty. Trained with maximum likelihood objectives, they tend to produce overconfident predictions, assigning near-certain probabilities even to inputs far from the training distribution ( 18 ) . The dominant approach to epistemic uncertainty estimation remains to train an ensemble of independently initialized models ( 32 ) . The principle is intuitive: near observed training samples, ensemble members agree; far from the data, they diverge. This disagreement serves as a proxy for epistemic uncertainty. Deep ensembles consistently outperform other Bayesian approximations such as MC-Dropout ( 15 ) , both in terms of calibration and for out-of-distribution detection. However, deep ensembles suffer from a fundamental limitation: their computational cost scales linearly with the number of members. For foundation models with billions of parameters, maintaining even a modest ensemble of four members quadruples memory consumption and training time. This cost is tolerable for well-resourced institutions but excludes precisely the practitioners who have embraced efficient fine-tuning: engineers who have to adapt models with few (often a single) GPU, researchers who deploy models on edge devices, organizations that do not have access to large compute clusters. In an era where parameter-efficient adaptation has made foundation models accessible to all, uncertainty quantification remains a privilege of the few. The machine learning landscape has shifted rapidly towards foundation models as the default starting point. Pre-trained models continue to improve at a rapid pace, LLaMA, Mistral, and Qwen in language; DINOv3, and CLIP in vision, each generation offering stronger representations learned from ever-larger corpora. Concurrently, the community has embraced parameter-efficient fine-tuning (PEFT) as the default paradigm for adaptation: methods such as LoRA ( 27 ) , adapters ( 26 ) , and prompt tuning ( 33 ) enable practitioners to specialize billion-parameter models with modest hardware. As pre-training continues to improve, the representations encoded in these models become increasingly valuable, and the case for preserving them during fine-tuning grows stronger. Recent advances in parameter-efficient fine-tuning offer a promising direction. A growing body of work has shown that the knowledge encoded in foundation models is organized along linear subspaces : the singular vectors of weight matrices correspond to semantically meaningful directions, while singular values encode their relative importance ( 39 ; 14 ) . 1 demonstrated that fine-tuning operates in a low-dimensional subspace, with the intrinsic dimension decreasing as models scale. Building on this insight, 52 introduced Singular Value Fine-tuning (SVF), which decomposes pre-trained weights via SVD and trains only the singular values while freezing the singular vectors. Subsequent work has extended this paradigm: SVFit ( 51 ) scales singular value fine-tuning to large language models and vision transformers, manipulating 16 \times fewer parameters than LoRA. These developments share a common insight: the singular vectors learned during pre-training encode valuable knowledge that should be preserved, while singular values provide a compact parameterization for task-specific adaptation . We hypothesize that this same principle can be leveraged to build model ensembles. If different rescalings of singular values yield functionally distinct models, then training per-member singular values should produce diverse ensemble members while sharing the same underlying knowledge basis. We introduce Singular Value Ensemble (SV-Ensemble, SVE), a parameter-efficient implicit ensemble method that implements this insight. SVE decomposes pre-trained weight matrices via Singular Value Decomposition, freezes the singular vectors (the shared knowledge basis), and trains only per-member singular values. Diversity among ensemble members emerges naturally from two sources: (i) stochastic initialization of singular values with small perturbations, and (ii) mini-batch sampling during joint training, which causes members to converge to distinct combinations of the same underlying subspaces. This design yields extreme parameter efficiency. Where low-rank adaptation methods add matrices of dimension d\times r for each ensemble member ( 40 ) , SVE adds only a vector of \min(m,n) singular values per target matrix, a reduction from \mathcal{O}(dr) to \mathcal{O}(\min(d,k)) parameters per layer. This makes SVE significantly more efficient than prior implicit ensemble methods, e.g., LoRA-Ensemble ( 40 ) , and orders of magnitude more efficient than explicit deep ensembles. SVE thus brings principled uncertainty quantification to the same resource-constrained settings where parameter-efficient fine-tuning has already democratized large, foundational models. Our contributions: • We introduce Singular Value Ensemble (SVE), a parameter-efficient implicit ensemble method that creates diverse ensemble members by learning per-member singular values while sharing pre-trained singular vectors, the knowledge basis of a foundation model. • We establish SVE as a new strong baseline for uncertainty quantification in foundation models, achieving state-of-the-art calibration while maintaining strong accuracy. • We provide extensive validation across vision (Flowers102, CIFAR-100, DTD, Oxford Pets, OOD detection, CIFAR-100-C corruptions) and NLP (ARC-Easy, SST-2) benchmarks

Read the complete text