Sheaf Cohomology of Linear Predictive Coding Networks
Jeffrey Seely · 2025
Read Sheaf Cohomology of Linear Predictive Coding Networks by Jeffrey Seely (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11092.txt.
Introduction
Jeffrey Seely recasts linear predictive coding networks as cellular sheaves, a math frame that attaches data to the cells of a network and tracks how local values must agree. Predictive coding replaces one global training pass with local updates on weights and activity, which makes it attractive for brain-like computing. In the sheaf view, the sheaf map turns activity into edge-wise prediction errors.
Inference then acts like diffusion under the sheaf Laplacian, a spread operator on that frame. Sheaf cohomology, which counts patterns that cannot be smoothed away, marks the error patterns inference can never remove. Feedback loops can create such clashes on their own, and the frame gives tools to spot risky layouts.
Catalog record
| CREATOR | Jeffrey Seely |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-11092
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sheaf Cohomology of Linear Predictive Coding Networks Jeffrey Seely Abstract Predictive coding (PC) replaces global backpropagation with local optimization over weights and activations. We show that linear PC networks admit a natural formulation as cellular sheaves: the sheaf coboundary maps activations to edge-wise prediction errors, and PC inference is diffusion under the sheaf Laplacian. Sheaf cohomology then characterizes irreducible error patterns that inference cannot remove. We analyze recurrent topologies where feedback loops create internal contradictions, introducing prediction errors unrelated to supervision. Using a Hodge decomposition, we determine when these contradictions cause learning to stall. The sheaf formalism provides both diagnostic tools for identifying problematic network configurations and design principles for effective weight initialization for recurrent PC networks. † † year: 2025 † † workshop: Symmetry and Geometry in Neural Representations keywords predictive coding, cellular sheaf theory, cohomology, Hodge theory, applied topology, deep linear networks 1 Introduction Predictive coding (PC) recasts neural network training as an optimization problem over weights and activations ( 23 ) . This replaces a global loss with a series of layer-local losses. Optimizing over activations (PC inference) enables one to handle arbitrary recurrent topologies without unrolling the computational graph or relying on backpropagation through time ( 24 ) . But it creates a fundamental ambiguity that, to our knowledge, has been overlooked: a node deep in a recurrent network receives error signals from all its connections—some from supervision, others from contradictory feedback loops—with no way to distinguish between them. We formalize linear PC networks as cellular sheaves ( 5 ) to analyze this problem systematically. A predictive coding sheaf assigns vector spaces to vertices (neuron activations) and edges (prediction errors) of the underlying computational graph, with restriction maps encoding the network weights. The sheaf coboundary operator \delta^{0} computes all prediction errors simultaneously. Sheaf cohomology represents the network’s capacity to resolve inconsistencies. At a high level, sheaf theory studies how local consistency patches together into global coherence ( 22 ) —detecting when this local-to-global transfer succeeds (captured by the zeroth cohomology H^{0} ) and when it fails (captured by H^{1} ). This theme matches the predictive coding ambition: how can locally informed layers coordinate to solve a global task, even amid feedback loops far from any supervision? We illustrate the correspondence between predictive coding networks and sheaf theory through simple examples ( ). When input and output nodes are clamped to data and supervision targets, a Hodge decomposition characterizes precisely how the supervision signal distributes prediction errors across edges, and how it distributes neuron activations across vertices ( 4 ). This framework reveals why certain recurrent networks fail to learn: feedback loops can create internal contradictions that dominate supervision signals. Experiments demonstrate orders-of-magnitude differences in learning convergence rates based on weight initialization ( 5 ); importantly, the stark difference is not due to weight magnitude (all our initializations are orthonormal) but due to how the orientations of the weights create tension or resonance when traced around feedback loops within the network. We focus on linear networks (where sheaf cohomology and Hodge decompositions can be readily computed), though what follows can be regarded as a linear analysis of nonlinear PC networks evaluated at equilibria. 2 Deep Linear Networks as Constraints and Penalties We will establish the connection between linear PC networks and cellular sheaves by way of example. 2.1 A three-layer warm-up Let x\in\mathbb{R}^{n_{0}} , h_{1}\in\mathbb{R}^{n_{1}} , h_{2}\in\mathbb{R}^{n_{2}} , and y\in\mathbb{R}^{n_{3}} . A three-layer deep linear network optimizes \min_{W_{1},W_{2},W_{3}}\;\tfrac{1}{2}\,\big\lVert y-W_{3}W_{2}W_{1}x\big\rVert_{2}^{2}\,. (1) Alternatively, we can write ( 1 ) as a constrained optimization problem, \displaystyle\min_{W_{1},W_{2},W_{3},h_{1},h_{2}}\;\tfrac{1}{2}\,\big\lVert y-W_{3}h_{2}\big\rVert_{2}^{2}\quad\text{s.t.}\quad h_{1}=W_{1}x,\quad h_{2}=W_{2}h_{1}, (2) whereby the key feature is that now h_{1} and h_{2} are optimization variables. When constraints are satisfied exactly, the problem is equivalent to ( 1 ). 2.2 Predictive coding as a penalty method One interpretation of predictive coding is that it is a soft relaxation of ( 2 ) via quadratic regularization terms, otherwise known as a penalty method. 1 1 1 To our knowledge, PC as a penalty method is not standard pedagogy, though we find this perspective instructive for what follows. The resulting loss function, called the total energy E_{\text{PC}} , is the sum of squared residuals (or prediction errors): \displaystyle E_{\text{PC}}(h_{1},h_{2},W_{1},W_{2},W_{3})=\tfrac{1}{2}\big(\lVert r_{1}\rVert_{2}^{2}+\lVert r_{2}\rVert_{2}^{2}+\lVert r_{3}\rVert_{2}^{2}\big), (3) \displaystyle r_{1}=h_{1}-W_{1}x,\quad r_{2}=h_{2}-W_{2}h_{1},\quad r_{3}=y-W_{3}h_{2}. (4) PC is a bilevel optimization problem: Inference fixes W_{i} s and minimizes E_{\text{PC}} over h_{i} s; learning then fixes h_{i} s and updates W_{i} s. In practice, inference amounts to multiple gradient descent steps (w.r.t. h_{i} s) and learning is a single gradient descent step (w.r.t. W_{i} s) before h_{i} s are initialized again for the next minibatch; see 25 for other update schedules. 3 Sheaf Preliminaries for Predictive Coding We formalize a linear PC network as a cellular sheaf ( 5 ) , establishing nomenclature along the way. From activations to cochains. Following the three-layer example, we organize (concatenate) activations into a 0-cochain : \displaystyle s=(x,h_{1},h_{2},y)\in C^{0}:=\mathbb{R}^{n_{0}}\oplus\mathbb{R}^{n_{1}}\oplus\mathbb{R}^{n_{2}}\oplus\mathbb{R}^{n_{3}}. (5) The vector-concatenation of prediction errors forms a 1-cochain : \displaystyle r=(r_{1},r_{2},r_{3})\in C^{1}:=\mathbb{R}^{n_{1}}\oplus\mathbb{R}^{n_{2}}\oplus\mathbb{R}^{n_{3}}. (6) The coboundary operator. The coboundary \delta^{0}:C^{0}\to C^{1} computes all prediction errors simultaneously from the activations: \displaystyle\delta^{0}s=\begin{bmatrix}-W_{1}&I_{n_{1}}&0&0\\ 0&-W_{2}&I_{n_{2}}&0\\ 0&0&-W_{3}&I_{n_{3}}\end{bmatrix}\begin{bmatrix}x\\ h_{1}\\ h_{2}\\ y\end{bmatrix}=\begin{bmatrix}h_{1}-W_{1}x\\ h_{2}-W_{2}h_{1}\\ y-W_{3}h_{2}\end{bmatrix}, (7) where we note the per-edge sign convention. PC Inference is sheaf diffusion. From ( 7 ) it is clear that PC energy ( 3 ) can be written compactly as E_{\text{PC}}(s)=\tfrac{1}{2}\|\delta^{0}s\|^{2} . Minimizing the energy E_{\text{PC}} with respect to s via gradient descent yields the gradient flow \dot{s}=-Ls (or its discrete-time variant), where L=(\delta^{0})^{\top}\delta^{0} is the sheaf Laplacian . The gradient flow is known as sheaf diffusion , as it generalizes notions of graph diffusion to sheaves ( 11 ) . Network sheaves. More generally, for an arbitrary graph G=(V,E) , a network sheaf \mathcal{F} over G assigns vector spaces \mathcal{F}(v)=\mathbb{R}^{n_{v}} to vertices and \mathcal{F}(e)=\mathbb{R}^{m_{e}} to edges. Each edge e=(u,v) carries linear restriction maps \rho_{e\leftarrow u}:\mathcal{F}(u)\to\mathcal{F}(e) and \rho_{e\leftarrow v}:\mathcal{F}(v)\to\mathcal{F}(e) . Generalizing the three-layer example from the previous section, we define a predictive coding sheaf to be a network sheaf with the following convention: e=(u\to v) denotes a forward computation s_{v}=W_{e}s_{u} , encoded via restrictions \rho_{e\leftarrow u}=W_{e} and \rho_{e\leftarrow v}=I . The coboundary then computes the prediction error at each edge: \displaystyle(\delta^{0}s)_{e}=\rho_{e\leftarrow v}s_{v}-\rho_{e\leftarrow u}s_{u}=s_{v}-W_{e}s_{u}. (8) This definition applies for a graph G with multiple edges between vertices. Formally, our graph is undirected but where we use notation e=(u\to v) to specify the placement of the W_{e} and I restriction maps. Cohomology. The sheaf coboundary induces two cohomology groups that characterize network consistency: \displaystyle H^{0}(G,\mathcal{F}) \displaystyle=\ker\delta^{0}=\{s\in C^{0}:\delta^{0}s=0\}, (9) \displaystyle H^{1}(G,\mathcal{F}) \displaystyle=C^{1}/\im\delta^{0}\cong\ker(\delta^{0})^{\top}. (10) Here H^{0} consists of activation patterns (on vertices) with zero prediction error everywhere, while H^{1} represents prediction error patterns (on edges) that cannot arise from any activation choice. In finite dimensions with inner products we can represent H^{1}(G,\mathcal{F}) by \ker(\delta^{0})^{\top} . In our setting the predictive coding sheaf \mathcal{F} depends on the current network weights, so H^{0} and H^{1} should be understood as weight-dependent linear subspaces of C^{0} and C^{1} , not as topological invariants of the underlying graph. 4 Relative Systems and Clamped Networks In predictive coding, one typically does not work with the full unconstrained system. Instead, one clamps input x to data and output y to supervision targets during network training. In sheaf terms, this corresponds to a