Augmented Lagrangian Predictive Coding

Jeffrey Seely, Julian Gould · 2026

Read Augmented Lagrangian Predictive Coding by Jeffrey Seely, Julian Gould (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31022.txt.

Introduction

Seely and Gould offer PC-ALM, a rival to backpropagation. Predictive coding trains networks via local energy dynamics rather than a backward pass. Their method keeps predictive coding's budget but aligns weight updates toward BP by adding layer errors into a layer-local Lagrange term. In linear networks, it settles to a balance with BP gradients via layer-local updates. In nonlinear networks up to depth 128, it matches BP scores, notably in deep narrow networks. Its flow shows fast credit flow, with credit evenly spread, versus PC's slow spread.

Catalog record

CREATORJeffrey Seely, Julian Gould
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64364
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.31022

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Augmented Lagrangian Predictive Coding Jeffrey Seely, Julian Gould Abstract Predictive coding (PC) is a local-learning alternative to backpropagation (BP), training deep networks via local energy-minimization dynamics rather than a global backward pass. We introduce Augmented Lagrangian Predictive Coding (PC-ALM), which maintains PC’s inference budget but aligns each weight update toward BP by accumulating per-layer constraint errors into a layer-local Lagrange multiplier. In linear PC networks, PC-ALM converges to an equilibrium with exact BP gradients distributed across the network via only layer-local updates. We analyze PC-ALM in nonlinear PC networks up to depth 128 and show that it matches BP performance across all width-depth regimes, notably in deep narrow networks where PC underperforms. PC-ALM introduces recurrent dynamics in each layer’s activations. Compared to PC’s heat flow on a scalar energy, PC-ALM dynamics are driven by dual ascent on the augmented Lagrangian. We observe “ballistic” credit propagation across very deep networks, with credit signals evenly distributed across layers, compared to PC’s slow, diffusive credit propagation. Beyond the algorithm itself, the augmented Lagrangian framework offers a generalization of PC, and may yield insights into how distributed systems could compute and propagate BP-like credit signals through purely local dynamics. 1 Introduction Modern deep learning relies on backpropagation (BP). Yet biological circuits cannot implement BP, at least not exactly ( Lillicrap et al. 2020 ) . One reason is that BP is a global algorithm: even though BP’s chain rule spatially localizes updates, the network must still create error variables in parallel with the matching forward activations, then gate weight updates in a precisely orchestrated sequence of steps. This gap has motivated research into local alternatives to BP, where state, error, and weight variables evolve as a single dynamical system that requires only layer-local coupling. Predictive coding (PC) is one of the more actively researched layer-local alternatives to BP ( Whittington and Bogacz 2017 ; Millidge et al. 2021 ; Salvatori et al. 2026 ) . In a Predictive Coding Network (PCN), a mean-squared error term is introduced between each layer, the sum of which is the PC energy. Gradient descent on this energy yields a layer-local dynamical system for updating the activation states (PC inference) and weights (PC learning). At fixed weights, the activation equilibrium yields a different activation pattern than one would get from a typical forward pass, resulting in a corresponding weight update that is aligned with, but not equal to, the BP gradient. In this paper, we frame PC within a broader class of local learning methods involving the augmented Lagrangian (AL). Dual ascent on the AL—known as the method of multipliers (MM) ( Hestenes 1969 ) —yields a local algorithm in certain cases and has been extensively used in decentralized approaches to network training (e.g., Taylor et al. 2016 ). We show that MM can be used as a replacement for PC’s inference step, but with a substantial difference: at equilibrium, the Lagrange multipliers converge to exactly the gradient signal of the supervised loss. This connection is not entirely new. LeCun 1988 observed that the multipliers of the standard Lagrangian are the BP adjoints at feasibility. Interestingly, the augmented Lagrangian is essentially the standard Lagrangian plus the PC energy (up to the shared supervised-loss term), suggesting that there is a close relationship between PC and augmented Lagrangian methods. Contribution: PC-ALM. The caveat of MM is that exact BP computation requires convergence to equilibrium in its inner solve, which is not guaranteed for general nonlinear networks ( Wang et al. 2019 ) . However, we note that this difficulty is shared by PC. Nevertheless, the PC literature handles this empirically, fixing a finite inference budget and assessing performance directly ( Innocenti et al. 2025 ; Innocenti et al. 2026 ; Pinchetti et al. 2025 ) , often justified by using good initializations that are close to equilibrium. Motivated by this, we develop a finite-inference variant of MM, called PC-ALM (for Augmented Lagrangian Method), which operates on the same inference iteration budget as PC. Mechanistically, PC-ALM modifies PC inference with a dual variable (i.e. Lagrange multiplier) per layer that integrates prediction errors and feeds back into the local activations. After a fixed number of inference steps, the weight update acts on a composite signal of the prediction error shifted by the dual. The recurrence introduces stable oscillatory transients on an intermediate time scale between PC’s fast inference and slow weight updates. We first show that in linear PC networks, PC-ALM converges to a solution with exact BP gradients. Our core experimental finding is that PC-ALM closes the PC-BP gap at matched inference budget in nonlinear networks. Following Innocenti et al. 2026 , we test PC-ALM at depths up to 128 on image classification. PC-ALM matches BP across all width-depth regimes, including the deep narrow networks where PC significantly underperforms BP. We further observe experimentally that PC-ALM exhibits surprising dynamical properties. PC networks are known to concentrate credit signals near the supervised loss ( Pinchetti et al. 2025 ) . In contrast, PC-ALM distributes credit near-uniformly across the network. Finally, we observe that the primal-dual dynamics of PC-ALM accelerate the credit-propagation wavefront in extremely deep networks, where standard PC stalls even at equilibrium. Figure 1: PC-ALM credit alignment in a scalar two-layer network ( y=w_{2}w_{1}x ; h=w_{1}x ; x{=}y{=}1 ). (A) BP loss landscape with BP and equilibrated-PC iso-loss curves. (B) Arrows are negative weight gradients for BP, equilibrated PC, and PC-ALM at different iterations. (C) Inference trace. PC settles h at h_{\mathrm{eq}} ; PC-ALM returns h to h_{0}=w_{1}x while \lambda converges to the BP adjoint \lambda_{\infty}=(y-w_{2}h_{0})\,w_{2} . Here, we take 10 primal steps before each dual step to emphasize the role of the dual as a lagged integrator of error; in Algorithm 1 we alternate between one primal and one dual step. 2 Related Work Our current work draws on an under-explored connection between the biologically plausible deep learning literature and the distributed optimization literature. Predictive coding. In biologically plausible deep learning, one core research direction has been to make learning more local. Predictive coding originated in computational neuroscience as a model of cortical inference ( Rao and Ballard 1999 ; Friston and Kiebel 2009 ) and has seen ongoing development ( Millidge et al. 2021 ; Salvatori et al. 2026 ) as a layer-local approximation to BP using only Hebbian plasticity. Millidge et al. 2022a established a unified inference-and-learning framework, Salvatori et al. 2022 extended PC to arbitrary graph topologies, and Alonso et al. 2024 characterized the optimization difficulty of PC inference and proposed accelerated solvers. The current empirical state of the field is set by Pinchetti et al. 2025 , who find that PC can match BP on small-to-medium image-classification tasks but lags substantially for deep networks. Innocenti et al. 2024 show that the equilibrated PC energy contains only strict saddles, and Innocenti et al. 2025 introduce a \mu P-style parameterization ( \mu PC) which enables stable training for deep PC networks; follow-up work ( Innocenti et al. 2026 ) characterizes the PC-BP gap as a function of network width and depth. Augmented Lagrangian and ADMM for network training. A separate community has trained networks by alternating direction method of multipliers (ADMM) or augmented-Lagrangian methods, often for scalability, conditioning, or

Read the complete text