Bayesian Predictive Coding

Alexander Tschantz, Magnus Koudahl, Hampus Linander, Lancelot Da Costa, Conor Heins, Jeff Beck, Christopher Buckley · 2025

Read Bayesian Predictive Coding by Alexander Tschantz, Magnus Koudahl, Hampus Linander, Lancelot Da Costa, Conor Heins, Jeff Beck, Christopher Buckley (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-24016.txt.

Introduction

Alexander Tschantz and six co-authors give predictive coding a full Bayesian upgrade. Their paper from twenty twenty-five starts from a brain-inspired learning theory. Predictive coding trains by trimming variational free energy, a Bayesian score. In practice it kept only best-guess values for states and weights. Doubt about the weights was lost.

Their Bayesian predictive coding tracks a whole posterior spread over weights instead. Locality, the trait that makes the scheme brain-like, is kept. Weight updates come out in closed Hebbian form. In full-batch tests the method settles in fewer rounds than the classic form. Mini-batch runs stay on pace. Doubt gauges match known Bayesian deep learning tools. The authors pitch it as both a brain model and a practical uncertainty tool.

Catalog record

CREATORAlexander Tschantz, Magnus Koudahl, Hampus Linander, Lancelot Da Costa, Conor Heins, Jeff Beck, Christopher Buckley
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-24016

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bayesian Predictive Coding Alexander Tschantz, Magnus Koudahl, Hampus Linander, Lancelot Da Costa, Conor Heins, Jeff Beck, Christopher Buckley Abstract Predictive coding (PC) is an influential theory of information processing in the brain, providing a biologically plausible alternative to backpropagation. It is motivated in terms of Bayesian inference, as hidden states and parameters are optimised via gradient descent on variational free energy. However, implementations of PC rely on maximum a posteriori (MAP) estimates of hidden states and maximum likelihood (ML) estimates of parameters, limiting their ability to quantify epistemic uncertainty. In this work, we investigate a Bayesian extension to PC that estimates a posterior distribution over network parameters. This approach, termed Bayesian Predictive coding (BPC), preserves the locality of PC and results in closed-form Hebbian weight updates. Compared to PC, our BPC algorithm converges in fewer epochs in the full-batch setting and remains competitive in the mini-batch setting. Additionally, we demonstrate that BPC offers uncertainty quantification comparable to existing methods in Bayesian deep learning, while also improving convergence properties. Together, these results suggest that BPC provides a biologically plausible method for Bayesian learning in the brain, as well as an attractive approach to uncertainty quantification in deep learning. 1 Introduction Originating in the domain of neuroscience, the predictive coding (PC) framework [ 1 , 2 , 3 , 4 ] proposes that the function of neural plasticity is to minimize local prediction errors , which quantify the difference between estimated and observed signals. This framework has been adapted as a method for training deep neural networks using only local information, offering a biologically plausible alternative to backpropagation (BP) [ 5 , 6 , 7 , 8 ] . PC offers several benefits over BP, including improved performance in online and continual learning settings [ 9 ] , favourable optimization properties [ 10 ] , the flexibility to be used in either a generative or discriminative manner [ 11 ] and intrinsic auto-associative memory capabilities [ 12 ] . PC has traditionally been motivated through variational Bayesian inference [ 13 , 14 ] , characterising the relationships between hidden states and parameters using probability distributions. However, implementations of PC generally do not operate on distributions but instead use maximum a posteriori (MAP) estimates of hidden states and maximum likelihood (ML) estimates of parameters. This contrasts with Bayesian deep learning [ 15 ] , where the goal is to estimate posterior distributions over parameters so that epistemic and aleatoric uncertainty [ 16 ] can be quantified and subsequently used for model comparison, network pruning, or well-calibrated confidence estimation. Regardless, uncertainty quantification is essential for the robustness, reliability, and interpretability of learning systems, and how it is performed by the brain remains an open question. In this work, we propose an extension of PC that estimates approximate Bayesian posteriors over network parameters. This approach, termed Bayesian predictive coding (BPC), parameterises neural activity in a way that allows for the use of conjugate priors, enabling closed-form update rules for the weight distribution. The resulting updates are Hebbian functions of the pre- and post-synaptic activity, and hidden state updates retain their interpretation as precision-weighted prediction errors, thus preserving the locality and simplicity of the PC algorithm. Moreover, the ability to compute posterior updates in a closed form means that BPC can converge in substantially fewer iterations than gradient-based methods [ 17 ] . See Appendix E for an overview of related work. In a set of experiments, we empirically verify that BPC achieves comparable performance to PC and traditional BP in full-batch training and remains competitive in mini-batch training. Notably, in the full-batch training context, BPC converges in a remarkably few epochs. Moreover, we demonstrate that the learned posterior distribution enables robust quantification of epistemic and aleatoric uncertainties in synthetic regression tasks. We compare BPC to a popular benchmark in Bayesian deep learning and demonstrate that our method provides both improved uncertainty quantification and improved accuracy and converges in fewer iterations. Taken together, our results suggest that BPC is a viable method for training uncertainty-aware neural networks using local information, suggesting a potential mechanism for uncertainty quantification in neural systems. 2 Methods Bayesian predictive coding is an algorithm for inverting hierarchical Gaussian generative models with L layers of variables \mathbf{Z}=\left\{\mathbf{z}_{l}:l=0\dots L\right\} and parameters \bm{\Theta}=\left\{\mathbf{W}_{l},\bm{\Sigma}_{l}:l=1\dots L\right\} : \displaystyle p(\mathbf{Z},\bm{\Theta}) \displaystyle=p(\mathbf{z}_{0})p(\mathbf{W}_{0},\bm{\Sigma}_{0})\prod_{l=1}^{L}p(\mathbf{z}_{l}\mid\mathbf{z}_{l-1},\mathbf{W}_{l},\bm{\Sigma}_{l})p(\mathbf{W}_{l},\bm{\Sigma}_{l}), (1) \displaystyle p(\mathbf{z}_{l}\mid\mathbf{z}_{l-1},\mathbf{W}_{l},\bm{\Sigma}_{l}) \displaystyle=\mathcal{N}(\mathbf{z}_{l}\mid\mathbf{W}_{l}f(\mathbf{z}_{l-1}),\bm{\Sigma}_{l}), \displaystyle p(\mathbf{W}_{l},\bm{\Sigma}_{l}) \displaystyle=\mathcal{MN}(\mathbf{W}_{l}\mid\mathbf{M}_{l}^{(0)},\bm{\Sigma}^{-1}_{l},\mathbf{V}_{l}^{(0)})\ \mathcal{W}(\bm{\Sigma}_{l}^{-1}\mid\bm{\Psi}_{l}^{(0)},\nu_{l}^{(0)}), where f(\cdot) is a point-wise non-linear activation function, and p(\mathbf{z}_{0})=\mathcal{N}(\mathbf{z}_{0}\mid\bm{\mu}_{0},\bm{\Sigma}_{0}) . Notably, we have placed the parameters \mathbf{W}_{l} outside of the non-linear activation function f(\cdot) , which is essential for enabling the closed-form updates used in Equation 7 . In practice, we also use a bias term that we ignore here for simplicity. Given that the log-likelihood is quadratic in the network parameters, a Matrix Normal Wishart prior for parameters \mathbf{W}_{l} and \bm{\Sigma}_{l} is conditionally conjugate given network activity \mathbf{Z} . See Appendix A for relevant definitions. In the standard PC framework, latent variables \mathbf{Z} are represented via maximum a posteriori (MAP) estimates, and parameters \bm{\Theta} are represented via maximum likelihood (ML) estimates. In contrast, BPC augments this approach by representing an approximate posterior distribution over parameters, \displaystyle q_{\bm{\lambda}}(\bm{\Theta}) \displaystyle=\prod_{l=1}^{L}q(\mathbf{W}_{l},\bm{\Sigma}_{l}), (2) \displaystyle q_{\bm{\lambda}}(\mathbf{W}_{l},\bm{\Sigma}_{l}) \displaystyle=\mathcal{MN}(\mathbf{W}_{l}\mid\mathbf{M}_{l},\bm{\Sigma}^{-1}_{l},\mathbf{V}_{l})\ \mathcal{W}(\bm{\Sigma}^{-1}_{l}\mid\bm{\Psi}_{l},\nu_{l}). where \bm{\lambda}=\left\{\mathbf{M}_{l},\mathbf{V}_{l},\bm{\Psi}_{l},\nu_{l},:l=1\dots L\right\} are the variational parameters. Given Equation ( 1 ) and ( 2 ), we define the objective function \mathcal{E}(\mathbf{Z},\bm{\lambda}) as: \mathcal{E}(\mathbf{Z},\bm{\lambda})=\Big<\log q_{\bm{\lambda}}(\bm{\Theta})-\log p(\mathbf{Z},\bm{\Theta})\Big>_{q_{\bm{\lambda}}(\bm{\Theta})} (3) which is equivalent to the variational free energy under the assumption that q_{\lambda}(\mathbf{Z}) is a Dirac delta distribution [ 13 ] . See Appendix C comparing Equation ( 3 ) to the energy function used in PC. To estimate \mathbf{Z} and \bm{\Theta} , we use the expectation-maximisation (EM) algorithm [ 18 ] on \mathcal{E}(\mathbf{Z},\bm{\lambda}) : \mathbf{Z}^{*}=\arg\min_{\mathbf{Z}}\mathcal{E}(\mathbf{Z},\bm{\lambda}),\quad\bm{\lambda}^{*}=\arg\min_{\bm{\lambda}}\mathcal{E}(\mathbf{Z}^{*},\bm{\lambda}). (4) Following PC, we solve the first optimisation problem using gradient descent on the energy function \mathcal{E}(\mathbf{Z},\bm{\lambda}) , which can be rewritten as: \mathcal{E}(\mathbf{Z},\bm{\lambda})=\frac{1}{2}\sum_{l=1}^{L}\underbrace{\left<\left(\mathbf{z}_{l}-\mathbf{W}_{l}f(\mathbf{z}_{l-1})\right)^{\top}\bm{\Sigma}_{l}^{-1}\left(\mathbf{z}_{l}-\mathbf{W}_{l}f(\mathbf{z}_{l-1})\right)\right>_{q_{\bm{\lambda}}(\mathbf{W}_{l},\bm{\Sigma}_{l})}}_{\mathcal{E}_{l}}+C, (5) where C represents terms independent of \mathbf{Z} and \mathcal{E}_{l} is the precision-weighted prediction error for layer l . Gradient descent is performed by iteratively updating the latent variables \mathbf{Z} using the following update rule: \mathbf{z}_{l}\leftarrow\mathbf{z}_{l}-\alpha\frac{\partial\mathcal{E}}{\partial\mathbf{z}_{l}},\quad\frac{\partial\mathcal{E}}{\partial\mathbf{z}_{l}}=\frac{1}{2}\left(\frac{\partial\mathcal{E}_{l}}{\partial\mathbf{z}_{l}}+\frac{\partial\mathcal{E}_{l+1}}{\partial\mathbf{z}_{l}}\right), (6) where \alpha is the learning rate. This process is repeated until convergence or a maximum number of iterations T , and the converged latent variables are denoted as \mathbf{Z}^{\star} . See Appendix B for the closed-form expressions of the derivatives in Equation ( 6 ) as well as a method for optimally selecting \alpha . Given the converged latent variables \mathbf{Z}^{\star} , we can find a closed-form expression for \arg\min_{\bm{\lambda}}\mathcal{E}(\mathbf{Z}^{*},\bm{\lambda}) . Specifically, we update the natural parameters \bm{\eta}_{l} of the variational posterior q_{\bm{\lambda}}(\mathbf{W}_{l},\bm{\Sigma}_{l}) : \bm{\eta}_{l}^{\star}=\bm{\eta}^{(0)}_{l}+\sum_{n}\left(f(\mathbf{z}^{*n}_{l-1})f(\mathbf{z}^{*n}_{l-1})^{\top},f(\mathbf{z}^{*n}_{l-1}){\mathbf{z}^{*n}_{l}}^{\top},\mathbf{z}^{n*}_{l}{\mathbf{z}^{n*}_{l}}^{\top},\mathbf{1}\right), (7) where and \bm{\eta}^{(0)}_{l} are the natural parameters for the prior distribution defined in Equation ( 1 ), and \mathbf{1} denotes a vector of ones whose dimension matches the number of data points n . Equation 7 has a natural interpretation as a Hebbian function of pre- and post-synaptic activity. See Appendix D for a proof that Equation 7 minimises \mathcal{E}(\mathbf{Z}^{*},\bm{\lambda}) . The natural parameters \bm{\eta}_{l} relate to the parameters of Equation ( 2 ) via: \bm{\eta}_{l}=\left(\mathbf{V}_{l}^{-1},\,\mathbf{M}_{l}\mathbf{V}_{l}^{-1},\,\mathbf{\Psi}_{l}^{-1}+\mathbf{M}_{l}\mathbf{V}_{l}^{-1}\mathbf{M}_{l}^{\top},\,\nu_{l}-d_{y_{l}}+d_{x_{l}}-1\right). (8) Equation 7 provides an exact solution to the optimisation problem when the latent variables \mathbf{Z}^{\star} are estimated for the entire dataset. In practice, mini-batching is often used, in which case we introduce a learning rate \kappa and

Read the complete text