Bayesian Natural Gradient Fine-Tuning of CLIP Models via Kalman Filtering
Hossein Abdi, Mingfei Sun, Wei Pan · 2025
Read Bayesian Natural Gradient Fine-Tuning of CLIP Models via Kalman Filtering by Hossein Abdi, Mingfei Sun, Wei Pan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01694.txt.
Introduction
Hossein Abdi, Mingfei Sun, and Wei Pan fine-tune CLIP, the vision-language model. They work in the hard regime. Labeled examples are scarce there, and test data may drift from training data. First-order optimizers converge slowly there and are touchy about step size, while CLIP's loss landscape is non-convex with sharp critical points, exactly where curvature information should help.
Natural gradient descent uses that curvature by scaling the gradient with the inverse Fisher information matrix, but the matrix is far too expensive for large models. The authors approximate the natural gradient in a Bayesian way, using a Kalman filter, which also yields uncertainty estimates as a byproduct. Experiments on diverse image datasets test whether the second-order, uncertainty-aware updates generalize better both in-distribution and under distribution shift.
Catalog record
| CREATOR | Hossein Abdi, Mingfei Sun, Wei Pan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-01694
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bayesian Natural Gradient Fine-Tuning of CLIP Models via Kalman Filtering Hossein Abdi, Mingfei Sun, Wei Pan Abstract Vision-language pre-trained models, such as CLIP, have established new benchmarks in multimodal data mining. In such models, few-shot fine-tuning is a major challenge to achieve optimal performance on both in-distribution (ID) and out-of-distribution (OOD) datasets, especially when labeled data is scarce. Most existing fine-tuning approaches rely on first-order gradient-based optimizers, which typically suffer from slow convergence, sensitivity to step-size hyperparameters, and poor generalization in OOD settings. In contrast, second-order methods utilize local curvature information of the loss landscape to adjust the update step size. This is particularly beneficial for CLIP models, whose non-convex loss functions often contain sharp critical points. In such cases, natural gradient direction can offer more substantial and efficient per-iteration updates when fine-tuning with limited data. Natural Gradient Descent (NGD) is obtained by preconditioning the standard gradient with the inverse Fisher Information Matrix (FIM), which is computationally expensive for large models. To address this, we propose a Bayesian approximation of NGD using a Kalman filter for CLIP models. Our method combines the benefits of second-order optimization with Bayesian inference, which enhances generalization while providing uncertainty quantification. Extensive experiments conducted on diverse image classification datasets demonstrate that our algorithm consistently achieves superior–or comparable–ID performance and improved OOD robustness compared to state-of-the-art baselines. To the best of our knowledge, this work represents the first successful application of Kalman filtering to fine-tuning CLIP-based models, which enables more robust and efficient learning in vision-language tasks. I Introduction Pre-trained vision-language models, particularly CLIP ( radford2021learning ) , have demonstrated remarkable performance in zero-shot and few-shot multimodal learning tasks. However, optimal performance typically requires further adaptation to specific tasks. The effectiveness of transferring knowledge from such pre-trained models is highly sensitive to the distributional alignment between the pre-training (source) data and the task-specific (target) data. In real-world deployments, significant domain shifts can lead to substantial performance degradation. Fig. 1: We employ a Kalman-based adapter to fine-tune the CLIP models. Kalman-based optimization algorithm closely approximates the natural gradient direction within a Bayesian framework. While natural gradient facilitates improved ID performance, Bayesian formulation inherently enables uncertainty quantification, which leads to improvement in OOD generalization. CLIP-based fine-tuning approaches have been proposed to improve the In-Distribution (ID) performance and enhance the Out-of-Distribution (OOD) generalization. However, identifying an optimal strategy to balance ID performance with OOD generalization remains an open and critical research question ( kumar2022fine ) . Most of the existing methods predominantly rely on first-order gradient-based optimizers such as SGD, Adam, and their variants, which may result in (sub-)optimal convergence in non-convex loss landscapes, particularly under distribution shifts ( reddi2019convergence ; foret2020sharpness ) . To address these challenges, second-order optimization methods have emerged as sophisticated alternatives to first-order approaches. These methods exploit curvature information from the loss manifold, typically via the Hessian matrix (as in Newton’s method or quasi-Newton methods ( nocedal1999numerical ) ) or the Fisher information matrix (as in Natural Gradient Descent ( amari1998natural ) ). By incorporating curvature, second-order optimizers dynamically adjust updates based on the local geometry of the loss landscape. This curvature-aware approach is particularly effective in few-shot learning, where sharp minima in the loss landscape are common. Second-order methods navigate these regions more effectively and boost ID performance ( foret2020sharpness ; keskar2016large ) . Furthermore, Bayesian methods have shown superior effectiveness in managing uncertainty and enhancing the robustness of OOD compared to gradient-based techniques ( wilson2020bayesian ; ovadia2019can ) . These methods are highly effective at capturing model uncertainty and incorporating prior knowledge, which are critical to improving generalization to unseen data. Among Bayesian approaches, Kalman filtering, a step-size-independent and free from gradients algorithm, has shown considerable potential in machine learning optimization. In particular, it can operate as a straightforward second-order optimizer within a Bayesian framework ( ollivier2018online ) , which is expected to provide effective convergence while maintaining robustness to distributional shifts. Building on these foundations, we propose a robust fine-tuning algorithm based on Bayesian inference with the Kalman filtering method to simultaneously improve the ID performance and enhance the OOD robustness. Our contributions can be summarized as follows: • We introduce a robust adapter for fine-tuning CLIP-based vision-language models by leveraging Bayesian inference with Kalman filtering. This approach achieves consistent improvements in ID performance while improving the generalization of OOD. • To the best of our knowledge, this work represents the first successful application of the Kalman filter algorithm for fine-tuning CLIP-based vision-language models. Our method delivers robust and effective performance across both ID and OOD scenarios. • We conducted extensive experiments in multimodal classification tasks to demonstrate that our algorithm consistently achieves superior ID performance and OOD generalization compared to existing baselines. II Related Work Few-Shot Fine-Tuning Methods for CLIP To overcome the limitations of zero-shot learning, numerous studies have explored few-shot learning with CLIP. CoOp (Context Optimization) ( zhou2022learning ) introduces learnable vectors as text prompts to fine-tune CLIP for specific tasks. The CLIP-Adapter ( gao2024clip ) enhances the performance by employing an adapter-based fine-tuning approach, which adapts the model to downstream tasks with minimal task-specific data. Tip-Adapter-F ( zhang2022tipadaptertrainingfreeadaptionclip ) utilizes a key-value cache model from a few-shot training set to update CLIP’s prior knowledge through feature retrieval. ClipFit ( li2024visionlanguagemodelfinetuningsimple ) fine-tunes specific parameters, such as bias terms and normalization layers, to improve zero-shot performance while avoiding catastrophic forgetting. SAFT (Sparse Adaptation for Fine-Tuning) ( nguyen2024saftoutofdistributiongeneralizationfinetuning ) updates only a small subset of parameters with large gradient magnitudes, which preserves general knowledge and improves OOD generalization. SAFE ( zhu2024enhancing ) fine-tunes the attention pooling layer of CLIP’s visual encoder to improve performance in few-shot scenarios by focusing on task-specific semantics. The mask-aware fine-tuning method (MAFT) ( jiao2023learningmaskawarecliprepresentations ) addresses CLIP’s insensitivity to mask proposals in zero-shot segmentation by fine-tuning a specialized encoder while maintaining transferability. Domain-Aligned CLIP (DAC) ( gondal2024domain ) enhances both intra-modal and inter-modal alignment without modifying CLIP’s parameters, while GLCM-Adapter ( wang2024glcm ) improves few-shot learning by considering both global and local views of the input image for more robust recognition. Despite these advancements, the development of a fine-tuning method to balance ID performance with OOD generalization remains an open research area. Algorithm 1 Our Kalman-based Algorithm 1: Initialization: 2: p(\boldsymbol{\theta}_{0})=\mathcal{N}(\boldsymbol{\mu}_{0},\mathbf{\Sigma}_{0}) , \mathbf{R}_{0}=\mathbf{O}_{m\times m}+\epsilon\mathbb{I} 3: 4: for k=1,2,... do 5: Prediction: 6: \boldsymbol{\mu}_{k|k-1}=\boldsymbol{\mu}_{k-1} 7: \mathbf{\Sigma}_{k|k-1}=\mathbf{\Sigma}_{k-1}+\mathbf{Q}_{k} 8: 9: Pre-Updating: 10: \hat{\mathbf{y}}_{k}=h(\mathcal{B}_{k},\boldsymbol{\mu}_{k\mid k-1}) 11: \boldsymbol{H}_{k}=\nabla_{\boldsymbol{\theta}}h|_{(\mathcal{B}_{k},\boldsymbol{\mu}_{k\mid k-1})} 12: d_{M}=\sqrt{\left(\mathbf{y}_{k}-\hat{\mathbf{y}}_{k}\right)\boldsymbol{R}_{k-1}^{-1}\left(\mathbf{y}_{k}-\hat{\mathbf{y}}_{k}\right)^{\top}} 13: \lambda=\mathbf{e}^{-\alpha d_{M}} 14: \hat{\mathbf{R}}_{k}=\left(\mathbf{y}_{k}-\hat{\mathbf{y}}_{k}\right)\left(\mathbf{y}_{k}-\hat{\mathbf{y}}_{k}\right)^{\top}+\mathbf{H}_{k}\mathbf{\Sigma}_{k|k-1}\mathbf{H}_{k}^{\top} 15: \boldsymbol{R}_{k}=\beta\boldsymbol{R}_{k-1}+\lambda(1-\beta)\hat{\boldsymbol{R}}_{k} 16: 17: Updating: 18: \mathbf{K}_{k}=\mathbf{\Sigma}_{k|k-1}\mathbf{H}_{k}^{\top}\left(\mathbf{H}_{k}\mathbf{\Sigma}_{k|k-1}\mathbf{H}_{k}^{\top}+\mathbf{R}_{k}\right)^{-1} 19: \boldsymbol{\mu}_{k}=\boldsymbol{\mu}_{k|k-1}+\lambda\mathbf{K}_{k}(\mathbf{y}_{k}-\hat{\mathbf{y}}_{k}) 20: \mathbf{\Sigma}_{k}=\mathbf{\Sigma}_{k|k-1}-\mathbf{K}_{k}\mathbf{H}_{k}\mathbf{\Sigma}_{k|k-1} 21: 22: Output: 23: Posterior: p(\boldsymbol{\theta}_{k}\mid\mathcal{B}_{1:k})=\mathcal{N}(\boldsymbol{\mu}_{k},\mathbf{\Sigma}_{k}) 24: end for Kalman Filter for Optimizing Neural Networks The concept of employing the Kalman filter for parameter optimization in deep learning originates from the work of Singhal ( singhal1988training ) , who demonstrated that training neural networks can be framed as a system identification problem for nonlinear dynamic systems. This insight led to the use of the Extended Kalman Filter (EKF) to train neural network parameters. The superior performance of Kalman-based training algorithms over traditional backpropagation methods sparked significant interest in exploring the connections between these two classical approaches ( ruck1992comparative ; ollivier2018online ) . To enhance the applicability of the Kalman filter to large-scale models, several studies have focused on reducing its computational complexity.