Sequential Function-Space Variational Inference via Gaussian Mixture Approximation
Menghao Waiyan William Zhu, Pengcheng Hao, Ercan Engin Kuruoğlu · 2025
Read Sequential Function-Space Variational Inference via Gaussian Mixture Approximation by Menghao Waiyan William Zhu, Pengcheng Hao, Ercan Engin Kuruoğlu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07114.txt.
Introduction
Zhu, Hao, and Kuruoğlu tackled forgetting in neural networks in this paper from twenty twenty-five. Networks that learn tasks in sequence often lose the old tasks. One family of methods guards old knowledge by tracking a spread of likely network outputs. A single bell curve, though, can match only one peak of that spread.
The authors swap the single curve for a mixture of curves, which can hug several peaks at once. They test the idea in two setups: training only the last layer, or training every layer. The mixture version, tuned to favor likelihood, gave the best final average accuracy. The edge was largest when all layers were trained. The work offers a steadier way to learn continually.
Catalog record
| CREATOR | Menghao Waiyan William Zhu, Pengcheng Hao, Ercan Engin Kuruoğlu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-07114
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sequential Function-Space Variational Inference via Gaussian Mixture Approximation Thanks: This paper has been submitted to the IEEE Transactions on Artificial Intelligence for possible publication. Menghao Waiyan William Zhu, Pengcheng Hao, Ercan Engin Kuruoğlu Abstract Continual learning in neural networks aims to learn new tasks without forgetting old tasks. Sequential function-space variational inference (SFSVI) uses a Gaussian variational distribution to approximate the distribution of the outputs of the neural network corresponding to a finite number of selected inducing points. Since the posterior distribution of a neural network is multi-modal, a Gaussian distribution could only match one mode of the posterior distribution, and a Gaussian mixture distribution could be used to better approximate the posterior distribution. We propose an SFSVI method based on a Gaussian mixture variational distribution. We also compare different types of variational inference methods with a fixed pre-trained feature extractor (where continual learning is performed on the final layer) and without a fixed pre-trained feature extractor (where continual learning is performed on all layers). We find that in terms of final average accuracy, likelihood-focused Gaussian mixture SFSVI outperforms other sequential variational inference methods, especially in the latter case. † † impactstatement: This paper presents work whose goal is to advance the field of continual learning. The ability to continually learn new knowledge without forgetting old knowledge has a great impact on society, especially in healthcare, where, for example, we would like to be able to learn new diseases without forgetting old diseases, or diagnosis and treatment is made as signs and symptoms progress over the course of time. Since continual learning methods inherently have some form of memory, it is important that they are designed to preserve the privacy of people. I Introduction When neural networks learn new tasks, they lose predictive performance on old tasks. This is known as catastrophic forgetting [ 1 ] . It can be prevented by training using all the data, which we refer to as joint maximum a posteriori (MAP) training. However, access to the previous data may be limited due to computational or privacy constraints, so the aim of continual learning is to prevent forgetting with little or no access to the previous data. For classification tasks, three types of continual learning settings are commonly studied [ 2 ] : 1. Task-incremental learning , in which task IDs are provided and the classes change between tasks 2. Domain-incremental learning , in which task IDs are not provided and the classes remain the same between tasks but the input data distribution changes between tasks 3. Class-incremental learning , in which task IDs are not provided and the classes change between tasks In task-incremental learning, a multi-headed neural network is typically used, in which there is one head per task and the task ID is used to determine the head to use for prediction. In domain- and class-incremental learning, a single-headed neural network is typically used. Task IDs make continual learning much easier by restricting the possible classes and are unlikely to be accessible in practice. We adhere to the desiderata proposed in [ 3 ] and restrict ourselves to domain- and class-incremental learning with single-headed neural networks on task sequences with more than two similar tasks and limited access to the previous data. Continual learning methods are mostly based on regularization, replay or both. Regularization constrains the parameters of the neural network during training. Parameter-space regularization constrains them by directly penalizing their change [ 4 , 5 , 6 , 7 ] , while function-space regularization constrains them by penalizing the change in the outputs [ 8 , 9 , 10 , 11 ] . Replay trains the neural network with the current data together with the previous data which may be stored or generated [ 12 , 13 ] . It has been empirically observed that replay-based methods generally perform much better than regularization-based methods [ 14 , 3 , 15 ] . A class of continual learning method is based on sequential Bayesian inference on the parameters of the neural network, in which the previous posterior distribution becomes the current prior distribution, and prediction is provided by the posterior predictive distribution. Sequential maximum a posteriori inference (SMI) performs the prediction by using only the maximum of the posterior probability density function (PDF), while sequential variational inference (SVI) finds an approximate posterior PDF called the variational PDF by minimizing its Kullback-Leibler (KL) divergence from the posterior PDF and uses it to give an approximate posterior predictive distribution. [ 15 ] describes two general ways of performing SVI. In a prior-focused approach, the previous variational distribution is used as the current prior distribution, while in a likelihood-focused approach, the prior distribution is not updated, and the previous likelihood functions are approximated by using the previous data which may be stored or generated. Thus, prior-focused approaches are based on regularization, and likelihood-focused approaches are based on replay. Neural network posteriors are known to be highly complex and often multi-modal [ 16 ] . To capture this structure, we employ a Gaussian mixture variational distribution, which provides greater flexibility than standard Gaussian approximations. This choice is further motivated by prior work showing that various heavy-tailed distributions, including \alpha -stable distributions, can be expressed as scale mixtures of Gaussians [ 17 ] , and have been used to model the distributional behavior of neural network parameters [ 18 ] . We propose a method for learning a Gaussian mixture variational distribution via a function-space approach. Our main goal is to investigate whether Gaussian mixture methods perform better than Gaussian methods. We also combine likelihood-focused and function-space approaches and compare prior-focused and likelihood-focused approaches. Finally, since a fixed pre-trained feature extractor can be used to reduce the high computational cost of function-space approaches, we also investigate the usefulness of a fixed pre-trained feature extractor for these methods. The rest of the paper of organized as follows. Section II introduces the background for sequential Bayesian inference and sequential variational inference, and Section III mentions related work. Section IV describes the proposed method, and Section V describes the experiments. Finally, Section VI provides the conclusion. Appendix A provides the experiment details. II Background We introduce sequential Bayesian inference and describe three SVI methods, which our proposed method builds upon. We use bold letters to denote random variables, and the difference between vectors and scalars should be clear from context. 1 Sequential Bayesian Inference We describe a probabilistic model for Bayesian continual learning. Let \bm{\theta} be the collection of parameters of the neural network and (\bm{x}_{t})_{t=1}^{T} and (\bm{y}_{t})_{t=1}^{T} be the inputs and outputs from time 1 to t , respectively. (\bm{x}_{t})_{t=1}^{T} are assumed to be independent, and (\bm{y}_{t})_{t=1}^{T} are assumed to be conditionally independent given \bm{\theta} and (\bm{x}_{t})_{t=1}^{T} . These assumptions are depicted in 1 . \bm{\theta} \ldots \ldots \bm{y}_{2} \bm{x}_{2} \bm{y}_{1} \bm{x}_{1} \bm{y}_{T} \bm{x}_{T} \tilde{\bm{y}} \tilde{\bm{x}} Fig. 1 : Bayesian network for continual learning. \bm{\theta} is the collection of parameters of the neural network. (\bm{x}_{t})_{t=1}^{T} are the inputs, and (\bm{y}_{t})_{t=1}^{T} are the outputs. \tilde{\bm{x}} and \tilde{\bm{y}} are the input and output for prediction, respectively. Shaded nodes are observed. Note that (\bm{x}_{t},\bm{y}_{t})_{t=1}^{T} are not necessarily identically distributed, but tasks are similar, i.e. the form of the likelihood function l_{t}(y_{t}|\theta,x_{t}) is