Cross-Entropy Is Load-Bearing: A Pre-Registered Scope Test of the K-Way Energy Probe on Bidirectional Predictive Coding
Jon-Paul Cacioli · 2026
Read Cross-Entropy Is Load-Bearing: A Pre-Registered Scope Test of the K-Way Energy Probe on Bidirectional Predictive Coding by Jon-Paul Cacioli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21286.txt.
Introduction
Jon-Paul Cacioli tests whether a K-way energy probe result survives removing cross-entropy training. Across ten seeds on CIFAR-10 with a backbone, the probe sits below softmax for standard predictive coding trained with mean squared error. On bidirectional predictive coding it exceeds softmax on all ten seeds, though a check finds no greater latent movement. Removing cross-entropy alone halves the probe-softmax gap, and about two-thirds of the gap traces to logit scale, with the rest a ranking advantage.
Catalog record
| CREATOR | Jon-Paul Cacioli |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26085 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.21286
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cross-Entropy Is Load-Bearing: A Pre-Registered Scope Test of the K-Way Energy Probe on Bidirectional Predictive Coding Jon-Paul Cacioli Abstract Cacioli 2026 showed that the K-way energy probe on standard discriminative predictive coding networks reduces approximately to a monotone function of the log-softmax margin. The reduction rests on five assumptions, including cross-entropy (CE) at the output and effectively feedforward inference dynamics. This pre-registered study tests the reduction’s sensitivity to CE removal using two conditions: standard PC trained with MSE instead of CE, and bidirectional PC (bPC; Oliviers et al. 2025 ). Across 10 seeds on CIFAR-10 with a matched 2.1M-parameter backbone, we find three results. The negative result of Cacioli 2026 replicates on standard PC: the probe sits below softmax ( Δ = − 0.082 \Delta=-0.082 , p < 10 − 6 p<10^{-6} ). On bPC the probe exceeds softmax across all 10 seeds ( Δ = + 0.008 \Delta=+0.008 , p = 0.000027 p=0.000027 ), though a pre-registered manipulation check shows that bPC does not produce materially greater latent movement than standard PC at this scale (ratio 1.6, threshold 10). Removing CE alone without changing inference dynamics halves the probe-softmax gap ( Δ MSE = − 0.037 \Delta_{\text{MSE}}=-0.037 vs Δ stdPC = − 0.082 \Delta_{\text{stdPC}}=-0.082 ). CE is a major empirically load-bearing component of the decomposition at this scale. CE training produces output logit norms approximately 15 × \times larger than MSE or bPC training. A post-hoc temperature scaling ablation decomposes the probe-softmax gap into two components: approximately 66% is attributable to logit-scale effects removable by temperature rescaling, and approximately 34% reflects a scale-invariant ranking advantage of CE-trained representations. We use “metacognitive” operationally to denote Type-2 discrimination of a readout over its own Type-1 correctness, not to imply human-like introspective access. Pre-registration: https://osf.io/2kvsp (filed prior to data collection). Code and data: https://github.com/synthiumjp/ima 1 Introduction 1.1 The decomposition and its scope Cacioli 2026 established an approximate decomposition of the K-way energy probe on discriminative predictive coding networks [ Rao and Ballard 1999 , Bogacz 2017 , Whittington and Bogacz 2017 , Millidge et al. 2022 ] . Under five assumptions (CE at the output (A1), target clamping (A2), effectively feedforward dynamics (A3), deterministic generative predictions (A4), encoder-generative consistency (A5)), the K-way energy margin decomposes as M_{k}(x)\approx[\text{log-softmax margin}]_{k}+[R_{(2)}(x)-R_{(1)}(x)], (1) where R_{k}(x) is a residual from generative-chain propagation of the clamped target. This residual is not trained to correlate with correctness. The decomposition predicts that the probe should track softmax from below. This prediction was confirmed empirically across six conditions on a single seed. Cacioli 2026 was explicit about scope. The decomposition does not apply to bidirectional PC, prospective configuration, generative PC at test time, or energy formulations without CE at the output. The present study tests the first and last of these exclusions. 1.2 Bidirectional predictive coding Oliviers et al. 2025 introduced bidirectional PC (bPC). It replaces the standard discriminative PC energy with a bidirectional MSE formulation: E_{\text{bPC}}=\sum_{l=1}^{L-1}\frac{\alpha_{\text{gen}}}{2}\overline{\|x_{l}-W_{l+1}f(x_{l+1})\|^{2}}+\sum_{l=2}^{L}\frac{\alpha_{\text{disc}}}{2}\overline{\|x_{l}-V_{l-1}f(x_{l-1})\|^{2}}. (2) There is no CE term. Both top-down (generative, W ) and bottom-up (discriminative, V ) prediction errors drive latent updates during inference. The encoder and generative pathways are co-trained under a shared bidirectional energy objective. Relative to the decomposition’s assumptions, bPC violates A1 (no CE) and A3 (genuinely iterative bidirectional dynamics), and substantially alters A5. The decomposition’s algebraic chain breaks at Step 1, where the CE term links settled energy to log-softmax. Whether the dynamics change (A3) or the energy-formulation change (A1) is the more consequential violation is the empirical question this study addresses. At the TinyConv scale tested here [ Pinchetti et al. 2024 ] , bPC’s generative chain contributes less than 1% of total energy at the calibration-selected \alpha_{\text{gen}} . This constrains the regime in which results should be interpreted. 1.3 This study We ask whether CE at the output (A1) is a load-bearing component of the decomposition. We compare three conditions on a matched backbone: standard discriminative PC with CE (baseline), standard PC with MSE replacing CE (isolating A1), and bidirectional PC (violating A1, A3, and altering A5 simultaneously). The discPC-MSE condition violates only A1 while preserving A3 and A2. If it shows a smaller probe-softmax gap than standard PC, the energy formulation is partially responsible, independent of dynamics. The bPC condition was intended to additionally test A3. As reported below, the pre-registered manipulation check for dynamics failed. bPC did not produce materially greater latent movement than standard PC at this scale. The study therefore addresses A1 directly. It addresses A3 only negatively: the dynamics did not functionally manifest. The study was pre-registered on the Open Science Framework before any training runs. All hypotheses, statistical tests, decision rules, and diagnostics were specified in advance. This paper reports the pre-registered analyses without modification. The temperature scaling ablation (§ 4.4 ) was not pre-registered and is labelled as post-hoc throughout. 2 Method 2.1 Architecture All three conditions use the same structural backbone (TinyConv, 2,144,938 parameters). The encoder/V-pathway consists of Conv 3 \to 32, BN, GELU, MaxPool, Conv 32 \to 64, BN, GELU, MaxPool, Linear 4096 \to 256 with GELU, and Linear 256 \to 10. The generative/W-pathway consists of Linear 10 \to 256, Linear 256 \to 4096 reshaped to 64 \times 8 \times 8, Interpolate(2) with Conv 64 \to 32. This matches the architecture used in Cacioli 2026 . 2.2 Conditions Condition A (stdPC-CE). Standard discriminative PC with CE at the output. T =13 inference steps for training and evaluation. Data normalised to [0, 1] with channel-wise standardisation. This replicates the baseline of Cacioli 2026 . Condition B (stdPC-MSE). Standard discriminative PC with MSE at the output instead of CE. All other settings identical to Condition A. This isolates A1 from A3. Condition C (bPC). Bidirectional PC following Oliviers et al. 2025 . Bidirectional MSE energy with no CE. T =32 for training, T =100 for evaluation. \alpha_{\text{gen}} determined by a pre-registered 5-seed calibration sweep (selected value: 10^{-5} ). \alpha_{\text{disc}}=1.0 . Data normalised to [ - 1, 1]. All conditions use AdamW (lr= 10^{-4} , weight decay= 10^{-4} ), batch size 128, 25 epochs. 2.3 Evaluation For each trained network, two readouts are evaluated on the first 1,280 CIFAR-10 test images. The K-way energy probe clamps each of K =10 candidate classes at the output, runs inference to settling, and records the per-hypothesis total energy. The structural prediction is argmin over classes. The confidence margin is E_{(2)}-E_{(1)} . The softmax baseline reads the feedforward output logits from the V-pathway with no inference loop. The confidence margin is the top-1 minus top-2 softmax probability. The primary outcome is \Delta=\text{AUROC}_{2}^{\text{probe}}-\text{AUROC}_{2}^{\text{softmax}} , computed within each network. AUROC 2 is Type-2 AUROC: the area under the ROC curve for discriminating correct from incorrect Type-1 predictions using the readout’s confidence margin [ Fleming and Lau 2014 , Maniscalco and Lau 2012 ] . 2.4 Design and hypotheses The design is paired within-seed: 10 seeds (6 to 15) \times 3 conditions = 30 training runs. All conditions share the same seed for weight initialisation and data ordering. H3 (manipulation check, evaluated first). bPC latent movement exceeds stdPC by at least 10 \times . Deterministic threshold. H2 (replication, evaluated second). Mean \Delta_{\text{stdPC}}<0 . One-sided one-sample t -test, \alpha=0.05 . H1 (primary test, evaluated third). Mean D=\Delta_{\text{bPC}}-\Delta_{\text{stdPC}}>0 . One-sided paired t