TwinTURBO: Semi-Supervised Fine-Tuning of Foundation Models via Mutual Information Decompositions for Downstream Task and Latent Spaces

Guillaume QuétantDept. of Computer ScienceUniversity of GenevaCarouge, SwitzerlandPavlo MolchanovNVIDIA, Slava VoloshynovskiyDept. of Computer ScienceUniversity of GenevaCarouge, Switzerland · 2025

Read TwinTURBO: Semi-Supervised Fine-Tuning of Foundation Models via Mutual Information Decompositions for Downstream Task and Latent Spaces by Guillaume QuétantDept. of Computer ScienceUniversity of GenevaCarouge, SwitzerlandPavlo MolchanovNVIDIA, Slava VoloshynovskiyDept. of Computer ScienceUniversity of GenevaCarouge, Switzerland (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07851.txt.

Introduction

Fine-tuning a foundation model usually demands labeled data, which is exactly what many projects lack. Guillaume Quétant, Pavlo Molchanov, and Slava Voloshynovskiy present TwinTURBO, a semi-supervised method that learns from mostly unlabeled examples. It splits the learning signal into two parts: one bound guides the model's answers on the task itself, and another shapes the hidden representation so it stays orderly and aligned.

Crucially, the pretrained model itself is left untouched. All the adaptation happens in a small added module, a compact transformer with a token-merging step. Across several datasets, the method improved classification sharply when labels were extremely scarce, by putting the unlabeled data to work. The authors offer it as a cheap way to specialize giant models without disturbing what they already know.

Catalog record

CREATORGuillaume QuétantDept. of Computer ScienceUniversity of GenevaCarouge, SwitzerlandPavlo MolchanovNVIDIA, Slava VoloshynovskiyDept. of Computer ScienceUniversity of GenevaCarouge, Switzerland
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-07851

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TwinTURBO: Semi-Supervised Fine-Tuning of Foundation Models via Mutual Information Decompositions for Downstream Task and Latent Spaces Guillaume QuétantDept. of Computer ScienceUniversity of GenevaCarouge, SwitzerlandPavlo MolchanovNVIDIA, Slava VoloshynovskiyDept. of Computer ScienceUniversity of GenevaCarouge, Switzerland Abstract We present a semi-supervised fine-tuning framework for foundation models that utilises mutual information decomposition to address the challenges of training for a limited amount of labelled data. Our approach derives two distinct lower bounds: i) for the downstream task space, such as classification, optimised using conditional and marginal cross-entropy alongside Kullback-Leibler divergence, and ii) for the latent space representation, regularised and aligned using a contrastive-like decomposition. This fine-tuning strategy retains the pre-trained structure of the foundation model, modifying only a specialised projector module comprising a small transformer and a token aggregation technique. Experiments on several datasets demonstrate significant improvements in classification tasks under extremely low-labelled conditions by effectively leveraging unlabelled data. \svgsetup inkscapepath=svgsubdir 1 Introduction Foundation models are large-scale neural networks pre-trained on diverse data to learn general-purpose representations that can be fine-tuned for specific downstream tasks. This poses significant challenges, especially in the case of low-labelled data, a semi-supervised learning setting where only a small fraction of the data samples are labelled, while the majority remain unlabelled. While foundation models are pre-trained on large datasets in a self-supervised manner, their deployment often requires fine-tuning on new datasets with limited labelled samples and potential distribution shifts. Furthermore, the downstream tasks frequently differ from the pre-training objectives, complicating the adaptation process. Existing semi-supervised approaches, such as pseudo-labelling, rely heavily on assumptions about data distributions or task-specific tuning, limiting their generalisability. Addressing these challenges is essential to fully exploit the potential of foundation models and ensure their adaptability and scalability in diverse applications. The main contributions of this study are: • A new framework for foundation models fine-tuning: We introduces a fine-tuning strategy based on mutual information decomposition. This framework guides the adaptation of foundation models to downstream tasks and latent space representations, providing a theoretically sound and efficient alternative to traditional supervised learning. • Novel theoretical insights: The framework derives two distinct mutual information bounds: one for optimising downstream task performance and another for aligning latent space representations. These insights deepen our understanding of how information can be effectively utilised in semi-supervised learning. • Practical validation: Extensive experiments on classification tasks under extremely low-labelled data conditions demonstrate significant performance improvements. These results highlight the effectiveness of focusing on theoretical decompositions over heuristic-based approaches, and reveal previously underexplored opportunities in semi-supervised fine-tuning. TwinTURBO builds on and extends existing work in semi-supervised learning and mutual information-based optimisation Voloshynovskiy et al. 2020 ; Quétant et al. 2023 ; Drozdova et al. 2024 . Traditional approaches often emphasise pseudo-label generation or global fine-tuning strategies, which can be computationally expensive and less adaptable to diverse tasks. Our formalism provides a lightweight and modular framework that combines pre-trained vision foundation models with a specialised adapter module. This approach integrates with state-of-the-art methods such as InfoNCE-based contrastive learning van den Oord et al. 2018 ; Chen et al. 2020 ; Caron et al. 2021 ; Radford et al. 2021 , while addressing their limitations by providing a unified perspective on labelled and unlabelled semi-supervised fine-tuning. 2 Related work Semi-Supervised Machine Learning. Semi-supervised learning methods aim to exploit large amounts of unlabelled data to improve performance when labelled data is scarce. Traditional approaches include pseudo-labelling Lee 2013 , consistency regularisation Sajjadi et al. 2016 ; Laine and Aila 2017 , and entropy minimisation Grandvalet and Bengio 2005 . In the context of semi-supervised learning for foundation models, recent works explore the fine-tuning of pre-trained models with minimal labelled data while maximising the use of unlabelled samples. Methods such as FixMatch Sohn et al. 2020 and FlexMatch Zhang et al. 2021 refine pseudo-labelling by enforcing consistency under data augmentations. These approaches aim to minimise the reliance on large labelled datasets while maintaining strong generalisation capabilities. Fine-Tuning of Foundation Models. Full-model fine-tuning of large-scale models has been shown to be effective, but computationally expensive. To mitigate the computational cost of full-model fine-tuning, several works introduce adapter-based methods where additional small modules are trained while the pre-trained backbone is frozen. LoRA Hu et al. 2021 , VeRA Kopiczko et al. 2023 , DoRA Mao et al. 2024 and others introduce low-rank adaptation layers that allow efficient task-specific tuning without updating the entire model. Other studies, such as Compacter Mahabadi et al. 2021 and Prompt Tuning Lester et al. 2021 have demonstrated that tuning a fraction of the parameters can achieve competitive results compared to full-model fine-tuning, especially in low-data regimes. We also refer the reader to Drozdova et al. 2024 for a recent study on fine-tuning of vision foundation models. Mutual Information for Machine Learning. Mutual information bounds play a crucial role in modern machine learning by formalising the relationship between observed data, latent representations and downstream task labels. Classical approaches, such as the information bottleneck Tishby et al. 2000 , propose to minimise mutual information between inputs and latent variables while maximising mutual information between latent representations and outputs. More recent works decompose mutual information for machine learning applications. The study presented in Barber and Agakov 2004 introduces a variational lower bound on mutual information. Other information-theoretic studies, such as the variational information bottleneck Alemi et al. 2016 , the exploration of several bounds Poole et al. 2019 ; Tschannen et al. 2019 and Deep InfoMax Hjelm et al. 2018 apply mutual information maximisation to learn robust representations. More recently, TURBO Quétant et al. 2023 also uses mutual information maximisation to give a meaningful interpretation of a wide family of deep learning models. InfoNCE van den Oord et al. 2018 and its extensions provide tractable mutual information estimates via contrastive learning that is a common basis for modern unimodal self-supervised methods such as SimCLR Chen et al. 2020 and DINO Caron et al. 2021 and multimodal ones such as CLIP Radford et al. 2021 . 3 TwinTURBO TwinTURBO is a framework for guiding semi-supervised fine-tuning of foundation models using mutual information decompositions. It adapts models to downstream tasks or latent spaces by leveraging data and statistical structures, bridging theoretical insights with practical applications, while connecting to and advancing state-of-the-art approaches. 3.1 Decomposition principle We consider the problem of predicting, from a random variable \mathbf{X} , the random variable \mathbf{Y}=f(\mathbf{X}) , for a given function f . 1 1 1 Capital letters are reserved for random variables and small letters for their realisations. This function can usually be stochastic and is thought as any process that represent a physical or virtual relation between the two modalities \mathbf{X} and \mathbf{Y} . Once the function parameterised by a neural network f_{\theta} , where \theta denotes the parameters of the network, we look for a meaningful description of the training procedures utilised in modern machine learning. A natural consideration is to express the mutual information between \mathbf{X} and \mathbf{Y} \displaystyle I(\mathbf{X};\mathbf{Y})=\mathbb{E}_{p(\mathbf{x},\mathbf{y})}\left[\log\frac{p(\mathbf{y}|\mathbf{x})}{p(\mathbf{y})}\right], (1) and take it as a maximum objective towards which we want to push our neural network. To achieve this, we inject a parametric conditional density p_{\theta}(\mathbf{y}|\mathbf{x}) , characterised by the network outputs, and use the bound proposed in Barber and Agakov 2004 \displaystyle\begin{split}I(\mathbf{X};\mathbf{Y})&=\mathbb{E}_{p(\mathbf{x},\mathbf{y})}\left[\log\frac{p_{\theta}(\mathbf{y}|\mathbf{x})}{p(\mathbf{y})}\right]+\mathbb{E}_{p(\mathbf{x})}\left[D_{\mathrm{KL}}\left(p(\mathbf{y}|\mathbf{x})\,||\,p_{\theta}(\mathbf{y}|\mathbf{x})\vphantom{\left[\frac{1}{1}\right]}\right)\right]\\ &\geq\mathbb{E}_{p(\mathbf{x},\mathbf{y})}\left[\log\frac{p_{\theta}(\mathbf{y}|\mathbf{x})}{p(\mathbf{y})}\right],\end{split} (2) in order to get

Read the complete text