μ \mu PC: Scaling Predictive Coding to 100+ Layer Networks

Francesco Innocenti, El Mehdi Achour, Christopher L. Buckley · 2025

Read μ \mu PC: Scaling Predictive Coding to 100+ Layer Networks by Francesco Innocenti, El Mehdi Achour, Christopher L. Buckley (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13124.txt.

Introduction

Francesco Innocenti, El Mehdi Achour, and Christopher Buckley scale a brain-like learning rule to very deep networks in a paper from 2025. Predictive coding learns from local signals only, unlike backpropagation. Such networks have long failed beyond modest depth. The authors analyze why standard versions destabilize as layers pile up. Their fix adapts a depth-aware parameterization from mainstream deep learning.

The resulting method trains residual networks beyond one hundred layers reliably. Performance on simple classification tasks rivals current benchmarks with little tuning. Learning rates transfer across widths and depths without re-searching. Code ships in an open library. The authors call it a first step toward complex architectures for local learning rules.

Catalog record

CREATORFrancesco Innocenti, El Mehdi Achour, Christopher L. Buckley
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13124

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

μ \mu PC: Scaling Predictive Coding to 100+ Layer Networks Francesco Innocenti, El Mehdi Achour, Christopher L. Buckley Abstract The biological implausibility of backpropagation (BP) has motivated many alternative, brain-inspired algorithms that attempt to rely only on local information, such as predictive coding (PC) and equilibrium propagation. However, these algorithms have notoriously struggled to train very deep networks, preventing them from competing with BP in large-scale settings. Indeed, scaling PC networks (PCNs) has recently been posed as a challenge for the community [ 48 ] . Here, we show that 100+ layer PCNs can be trained reliably using a Depth- μ \mu P parameterisation [ 72 , 3 ] which we call “ μ \mu PC”. Through an extensive analysis of the scaling behaviour of PCNs, we reveal several pathologies that make standard PCNs difficult to train at large depths. We then show that, despite addressing only some of these instabilities, μ \mu PC allows stable training of very deep (up to 128-layer) residual networks on simple classification tasks with competitive performance and little tuning compared to current benchmarks. Moreover, μ \mu PC enables zero-shot transfer of both weight and activity learning rates across widths and depths. Our results have implications for other local algorithms and could be extended to convolutional and transformer architectures. Code for μ \mu PC is made available as part of a JAX library for PCNs at https://github.com/thebuckleylab/jpc [ 22 ] . Figure 1: \mu PC enables stable training of 100+ layer ResNets with zero-shot learning rate transfer. ( Right ) Test accuracy of ReLU ResNets with depths H=\{8,16,32,64,128\} trained to classify MNIST for one epoch with standard PC, \mu PC and BP with Depth- \mu P (see A.4 for details). Solid lines and shaded regions indicate the mean and \pm 1 standard deviation across 3 different random seeds. These results hold across other activation functions (see Fig. A.16 ). See also Figs. A.17 - A.18 for asymptotic results with 128-layer ReLU networks trained for multiple epochs on both MNIST and Fashion-MNIST, respectively. ( Left ) Example of zero-shot transfer of the weight and activity learning rates from 16- to 128-layer Tanh networks. Contours were averaged over 3 different runs. See Figs. 5 & A.30 - A.31 for the complete transfer results across widths as well as depths. 1 Introduction Backpropagation (BP) is arguably the core algorithm behind the success of modern AI and deep learning [ 52 , 28 ] . Yet, it is widely believed that the brain cannot implement BP due to its non-local nature [ 33 ] , in that the update of any weight requires knowledge of all the weights deeper or further downstream in the network. This fundamental biological implausibility of BP has motivated the study of many local algorithms, including predictive coding (PC) [ 36 , 35 , 54 , 63 ] , equilibrium propagation [ 59 ] , and forward learning [ 19 ] , among others [ 32 , 43 , 8 ] . While different in many aspects, all these algorithms have Hebbian-like weight updates that depend only on neighbouring neurons. However, local learning rules have notoriously struggled to train large and especially deep models on the scale of modern AI applications. 1 1 1 One might argue that these algorithms are more suited to alternative, non-digital hardware, but the question still remains as to whether they can be scaled on standard consumer GPUs. Moreover, the issues we expose with the standard parameterisation of PCNs can be argued to be hardware-independent (§ 3.1 ). For the first time, we show that very deep (100+ layer) networks can be trained reliably using a Depth- \mu P parameterisation [ 72 , 3 ] of PC which we call “ \mu PC” (Fig. 1 ). To our knowledge, no networks of such depth have been trained before with a local algorithm . Indeed, this has recently been posed as a challenge for the PC community [ 48 ] . We first show that the standard parameterisation of PC networks (PCNs) is inherently unscalable in that (i) the inference landscape becomes increasingly ill-conditioned with network width, depth and training time, and (ii) the forward initialisation of the activities vanishes or explodes with the depth. We then show that, despite addressing only the second instability, \mu PC is capable of training up to 128-layer fully connected residual networks (ResNets) on standard classification tasks with competitive performance and little tuning compared to current benchmarks (Fig. 1 ). Moreover, \mu PC enables zero-shot transfer of the weight and activity learning rates across widths and depths (Fig. 5 ). Code for \mu PC is made available as part of a JAX library for PCNs at https://github.com/thebuckleylab/jpc [ 22 ] . The rest of the paper is structured as follows. Following a brief review of the maximal update parameterisation ( \mu P) and PCNs (§ 2 ), Section 3 exposes two distinct pathologies in standard PCNs which make training at large scale practically impossible. We then suggest a minimal set of desiderata for a scalable PCN parameterisation (§ 4 ). Section 5 presents experiments with \mu PC, and Section 6 studies a regime where \mu PC converges to BP. We conclude with the limitations of this work and promising directions for future research (§ 7 ). For space reasons, we inclulde related work and additional experiments in Appendix A , along with derivations, experimental details and supplementary figures. 1.1 Summary of contributions • We show that \mu PC, which reparameterises PCNs using Depth- \mu P [ 72 , 3 ] , allows stable training of very deep (100+ layer) ResNets on simple classification tasks with competitive performance and little tuning compared to current benchmarks [ 48 ] (Figs. 1 & A.17 - A.18 ). • \mu PC also enables zero-shot transfer of the weight and activity learning rates across widths and depths (Figs. 5 & A.30 - A.31 ). • We achieve these results by a theoretical and empirical analysis of the scaling behaviour of the inference landscape and dynamics of PCNs (§ 3 ), revealing the following two pathologies: – the inference landscape becomes increasingly ill-conditioned with network width, depth (Fig. 2 ) and training time (Fig. 3 ) (§ 3.1 ); and – the forward pass of standard PCNs vanishes or explodes with the depth (§ 3.2 ). • To address these instabilities, we propose a minimal set of desiderata that PCNs should aim to satisfy to be trainable at scale (§ 4 ), revealing an apparent trade-off between the conditioning of the inference landscape and the stability of the forward pass (Fig. 4 ). This analysis applies to other inference-based algorithms (§ A.2.5 ). • For random linear ResNets, we study a regime where the \mu PC energy at the inference equilibrium converges to the mean squared error (MSE) loss and so PC effectively implements BP (Theorem 1 , Fig. 6 ). 2 Background 2.1 The maximal update parameterisation ( \mu P) The maximal update parameterisation was first introduced by [ 70 ] to ensure that the order of the activation updates at each layer does not vary with the width N . This was motivated by the lack of feature learning

Read the complete text