Bridging Predictive Coding and MDL: A Two-Part Code Framework for Deep Learning

Benjamin Prada, Shion Matsumoto, Abdul Malik Zekri, Ankur Mali · 2025

Read Bridging Predictive Coding and MDL: A Two-Part Code Framework for Deep Learning by Benjamin Prada, Shion Matsumoto, Abdul Malik Zekri, Ankur Mali (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14635.txt.

Introduction

Benjamin Prada and co-authors join two learning theories in one frame in a 2025 paper. Predictive coding is a brain-inspired rule where each layer learns from its own errors. Minimum description length instead scores a model by how briefly it can be written plus its mistakes.

The team proves layerwise predictive coding descends step by step on that two-part code. Each sweep provably shortens the total code, they show. A fresh risk bound follows, trading fit against code length in a clean ratio. Repeated sweeps settle at a stable point of the shared objective. Biology-flavored learning gains the guarantees of compression theory.

Catalog record

CREATORBenjamin Prada, Shion Matsumoto, Abdul Malik Zekri, Ankur Mali
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14635

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bridging Predictive Coding and MDL: A Two-Part Code Framework for Deep Learning Benjamin Prada, Shion Matsumoto, Abdul Malik Zekri, Ankur Mali Abstract We present the first theoretical framework that connects predictive coding (PC), a biologically inspired local learning rule, with the minimum description length (MDL) principle in deep networks. We prove that layerwise PC performs block-coordinate descent on the MDL two-part code objective, thereby jointly minimizing empirical risk and model complexity. Using Bernstein’s inequality and a prefix-code prior, we derive a novel generalization bound of the form R ​ ( θ ) ≤ R ^ ​ ( θ ) + L ⁡ ( θ ) N R(\theta)\leq\hat{R}(\theta)+\frac{L(\theta)}{N} , capturing the tradeoff between fit and compression. We further prove that each PC sweep monotonically decreases the empirical two-part codelength, yielding tighter high-probability risk bounds than unconstrained gradient descent. Finally, we show that repeated PC updates converge to a block-coordinate stationary point, providing an approximate MDL-optimal solution. To our knowledge, this is the first result offering formal generalization and convergence guarantees for PC-trained deep models, positioning PC as a theoretically grounded and biologically plausible alternative to backpropagation. 1 Introduction Over the past decade, deep learning has achieved remarkable empirical success across a wide array of applications [ 13 , 10 , 39 , 6 , 11 , 27 , 19 ] . Yet, a foundational question remains unresolved: why do highly overparameterized neural networks generalize well, despite lacking explicit regularization or capacity control? While backpropagation (BP) remains the dominant training algorithm, it optimizes purely for loss minimization and offers no direct mechanism for balancing model complexity with data fit. This has motivated the exploration of alternative learning frameworks that offer both theoretical insight and practical efficiency. One principled approach to understanding generalization is rooted in information theory. The Minimum Description Length (MDL) principle formalizes model selection as a coding problem: the best model is the one that minimizes the total length of a two-part code, with one part encoding the model and the other encoding the data given the model. Though MDL provides a compelling lens on generalization, it is rarely used to guide or analyze the training dynamics of modern deep networks—particularly those trained via biologically inspired algorithms. In parallel, predictive coding (PC) has re-emerged as a compelling alternative to BP. Originally proposed as a model of hierarchical processing in the neocortex [ 29 , 7 ] , PC minimizes a layerwise energy functional by iteratively reducing prediction errors through local feedback and feedforward interactions [ 31 ] . Its simple, biologically plausible dynamics have enabled effective applications in image classification and generation [ 21 , 26 ] , continual learning [ 24 ] , associative memory [ 33 , 37 ] , and reinforcement learning [ 28 , 23 ] . PC has also been shown to scale effectively to deep networks and neuromorphic hardware implementations [ 12 , 20 ] . From Dynamics to Generalization: An Open Gap. Recent theoretical work has clarified many aspects of PC’s learning dynamics, particularly in relation to BP. Milledge et al. [ 17 ] show that PC networks trained via prospective configuration converge to BP-critical points and approximate target propagation under certain inference equilibria. Mali et al. [ 14 ] approach PC through dynamical systems theory, proving Lyapunov stability, robustness to perturbations, and quasi-Newton behavior via higher-order curvature analysis. Other works have shown how PC approximates BP under assumptions like weight symmetry or linearization [ 2 , 36 , 35 , 34 , 16 , 15 ] . While these studies provide valuable insight into the stability and efficiency of PC, they do not offer independent generalization guarantees —that is, guarantees that do not rely on BP approximation. Moreover, none of these works establishes a connection between PC’s energy-based updates and formal learning-theoretic principles such as MDL or PAC-Bayes. Developing such guarantees is critical for explaining why PC-based models have recently demonstrated strong performance in few-shot reconstruction [ 22 , 24 ] , adversarial robustness [ 33 ] , and generalization under limited supervision [ 25 , 32 ] . In this paper, we close this gap by providing the first theoretical connection between predictive coding and the MDL principle. Rather than treating PC as a proxy for BP, we reinterpret it as a standalone learning algorithm that implicitly minimizes a two-part empirical codelength objective. This reframing provides a new lens on PC’s ability to regularize model complexity and improve generalization, independent of its similarity to BP. Summary of Contributions. The following key results provide the first principled MDL-based justification for PC: Core Contributions (Summary) \star Universal Occam–style Bound (Algorithm–Agnostic). For every parameter realization \theta , sample size N , and confidence \delta\!\in\!(0,1) we have \boxed{{\color[rgb]{0,0.3984,0.8008}R(\theta)\;\leq\;\hat{R}(\theta)\;+\;\frac{L(\theta)}{N}\;+\;\frac{\ln(1/\delta)}{N}}}\quad\text{with}\quad L(\theta)=-\log p(\theta). This inequality holds independently of whether the learner uses BP, PC, or any other training rule. \star PC Minimizes Empirical Codelength: Each PC sweep performs blockwise descent on the total description length: \boxed{{\color[rgb]{0,0.6016,0.1992}\hat{C}(\theta)=\hat{R}(\theta)+\frac{1}{N}L(\theta)}} \star Convergence Guarantee: We prove that repeated PC updates yield a bounded, monotonically decreasing sequence of codelengths that converges (up to subsequences) to a blockwise stationary point of \hat{C}(\theta) . This provides the first convergence result for PC framed in terms of information-theoretic optimality. These results position PC not merely as a biologically inspired approximation to BP, but as a distinct, theoretically grounded learning algorithm—one that achieves generalization by minimizing description length. Our work bridges two historically disconnected perspectives—energy-based local learning and MDL-based generalization—and opens new directions for designing efficient, compression-driven learning systems beyond gradient BP. 2 Background and Motivation Denote a model by \theta\in\Theta (e.g., the parameters of a neural network). Minimum Description Length (MDL). The MDL principle [ 30 ] posits that the best model is the one that most compresses the data. This is formalized as: C(D)=\min_{\theta\in\Theta}\left[C(D\mid\theta)+C(\theta)\right], where C(D\mid\theta) is the data fit under the model and C(\theta) is the model complexity. Predictive Coding Networks (PCNs). PC [ 29 , 7 ] models cortical computation as hierarchical error minimization. PCNs consist of layers that predict the activity of the next and refine internal states through local feedback. For nonlinear activations f , PC minimizes the energy: \mathcal{F}(\theta,x)=\sum_{l=1}^{L}\left\|x_{l}-f(\theta_{l}x_{l-1})\right\|^{2}, with x_{0} clamped to the input and x_{L} to the label in supervised tasks. Predictive Coding Updates \Delta x_{l}=-\epsilon_{l}+\theta_{l+1}^{\top}(\epsilon_{l+1}\odot f^{\prime}(\theta_{l+1}x_{l}))\quad\text{(Inference)} \Delta\theta_{l}=-\eta(\epsilon_{l}\odot f^{\prime}(\theta_{l}x_{l-1}))x_{l-1}^{\top}\quad\text{(Learning)} Here, \epsilon_{l}=x_{l}-f(\theta_{l}x_{l-1}) denotes the local prediction error, and \eta is the learning rate. These updates are layer-local and biologically plausible, depending only on pre-/post-synaptic activity and adjacent errors. Motivation. While much prior work has viewed PC as an approximation to BP, we take a different perspective: PC is a principled learning algorithm that implicitly minimizes a two-part codelength. We show that each PC sweep performs block-coordinate descent on an MDL objective of the form: \hat{C}(\theta)=\underbrace{\hat{R}(\theta)}_{\text{data fit}}+\underbrace{\frac{1}{N}L(\theta)}_{\text{complexity}}. This insight enables us to derive generalization guarantees and convergence results, establishing PC as a compression-driven alternative to BP with provable learning-theoretic foundations. 3 Theoretical Foundation: Generalization via Codelength Bounds To understand how PCNs generalize, we begin by revisiting foundational results from statistical learning theory that relate empirical performance to population-level risk. Specifically, we ground our analysis in an information-theoretic framework in which model complexity is encoded via a prefix code prior over parameters. This naturally leads to high-probability generalization guarantees based on the

Read the complete text