Stochastic Layer-wise Learning: Scalable and Efficient Alternative to Backpropagation

Bojian Yin, Federico Corradi · 2025

Read Stochastic Layer-wise Learning: Scalable and Efficient Alternative to Backpropagation by Bojian Yin, Federico Corradi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-05181.txt.

Introduction

Bojian Yin and Federico Corradi put forth a way in place of backpropagation in this two thousand twenty-five machine learning paper. Backpropagation trains networks but needs gradient synchronization and memory. Local learning rules cost less, yet lose the cross-layer fit that makes learning hang as one. Stochastic Layer-wise Learning threads the needle. It splits the goal into layer-local updates that stay in line through the form of the network.

The method draws on the evidence lower bound under a Markov claim across layers. A divergence term that can not be tracked is swapped for a Bhattacharyya surrogate, worked out from categorical posteriors through geometry-preserving projections. Dropout can be added for regularization. Tests ran from MNIST to ImageNet, spanning connected, convolutional, and transformer forms. The method beat local methods and matched backpropagation. Memory use stayed flat as depth grew, a win for models and neuromorphic hardware.

Catalog record

CREATORBojian Yin, Federico Corradi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-05181

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Stochastic Layer-wise Learning: Scalable and Efficient Alternative to Backpropagation Bojian Yin, Federico Corradi Abstract Backpropagation underpins modern deep learning, yet its reliance on global gradient synchronization limits scalability and incurs high memory costs. In contrast, fully local learning rules are more efficient but often struggle to maintain the cross-layer coordination needed for coherent global learning. Building on this tension, we introduce Stochastic Layer-wise Learning (SLL), a layer-wise training algorithm that decomposes the global objective into coordinated layer-local updates while preserving global representational coherence. The method is ELBO-inspired under a Markov assumption on the network, where the network-level objective decomposes into layer-wise terms and each layer optimizes a local objective via a deterministic encoder. The intractable KL in ELBO is replaced by a Bhattacharyya surrogate computed on auxiliary categorical posteriors obtained via fixed geometry-preserving random projections, with optional multiplicative dropout providing stochastic regularization. SLL optimizes locally, aligns globally, thereby eliminating cross-layer backpropagation. Experiments on MLPs, CNNs, and Vision Transformers from MNIST to ImageNet show that the approach surpasses recent local methods and matches global BP performance while memory usage invariant with depth. The results demonstrate a practical and principled path to modular and scalable local learning that couples purely local computation with globally coherent representations. K eywords Supervised Learning \cdot ELBO \cdot Layer-wise Learning 1 Introduction The success of deep learning across a wide range of domains has been substantially driven by backpropagation (BP), a foundational learning algorithm enabling hierarchical representation learning through end-to-end gradient-based optimization [ 1 , 2 ] . Despite its algorithmic clarity and practical effectiveness, BP requires the exact storage of intermediate activations and subsequent gradient computation across all layers. This mechanism facilitates global credit assignment [ 3 ] ; it also introduces a well-known bottleneck called update-locking [ 4 , 5 ] , where the weight update of a given layer must wait until both the forward pass through the entire network and the backward pass through deeper layers are complete. Consequently, this global dependency limits asynchronous updating, and imposes substantial memory and computational overhead, ultimately reducing training efficiency and scalability, especially in resource-constrained devices [ 6 , 7 , 8 ] . BP is often seen as biologically implausible and this drives efforts to discover local learning rules for credit assignment in inspired by real neural systems [ 3 , 9 , 10 ] . At the same time, neuroscience suggests that feedback connections may approximate global errors via local activity differences [ 10 , 11 ] , hinting at a bioplausible path to deep learning [ 3 , 12 ] . Yet, these approaches struggle to reconcile local updates with global learning and lack a unifying theoretical framework. Given this context, a central research question emerges: “Can we design a theoretical framework capable of decomposing deep neural network training into local (layer-wise) optimizations while retaining the benefits of hierarchical representation learning?” This question captures a fundamental conflict: while local learning encourages architectural scalability and computational parallelization, effective deep learning relies on non-linear coordination across the entire network. This disconnect often results in misaligned learning signal and suboptimal performance, challenging the network scalability of locally trained models [ 13 ] . To address this question, we ground local learning in how information propagates through deep networks: as signals traverse layers, raw inputs are progressively transformed into increasingly disentangled and class-separable representations [ 14 , 15 , 16 ] . This refinement suggests that intermediate layers perform latent inference, selectively preserving task-relevant signals while suppressing redundancy [ 17 ] . In this paper, we make this intuition precise by exhibiting a network-level ELBO that decomposes into layer-wise terms under a Markov assumption of network architecture, thereby furnishing principled local objectives while retaining an explicit link to the global goal. Building on this decomposition, we introduce Stochastic Layer-wise Learning (SLL) , a local learning framework in which each layer produces auxiliary categorical posteriors via fixed stochastic random projections, and the intractable layer-wise KL in the ELBO is replaced by a Bhattacharyya surrogate [ 18 ] computed on these induced posteriors, yielding an ELBO-inspired and numerically stable update. Here, the projections preserve minibatch geometry with high probability by the Johnson–Lindenstrauss (JL) lemma [ 19 , 15 ] , which justifies computing divergences in the compressed space; we further apply multiplicative dropout to the fixed projection, which provides stochastic regularization consistent with the dropout-as-variational-inference interpretation [ 20 ] ; we do not learn mask parameters and do not claim a variational bound over masks, and the overall objective remains ELBO-inspired at the layer level. SLL thus reconciles local optimization with hierarchical coordination, mitigating over-compression associated with direct KL minimization, maintaining global representational coherence, and enabling scalable, parallel training without full backpropagation. Figure 1: Overview of Stochastic Layer-wise Learning (SLL). (a) SLL treats each hidden activation h_{l} as a latent variable and projects it to v_{l} via a random matrix. The local ELBO comprises a log-likelihood term and a KL surrogate that promotes inter-layer consistency. Optimizing this loss yields the improved projection \tilde{v}_{l} and its corresponding activation \tilde{h}_{l} . (b) SLL optimizes each layer independently using a prediction loss \mathcal{L}_{\text{pred}}(v_{l},y) from the log-likelihood and a feature alignment loss \mathcal{L}_{\text{BC}}^{per}(v_{l},v_{l-1}^{\text{sg}}) approximating the KL term. Arrows denote forward computation (black), local updates (red), and stop-gradient paths (slashed). This work targets mathematical analysis, algorithmic development, and experimental evaluations, leading to three principal contributions: Theoretical contribution : we formally decompose the network ELBO into layer-wise terms under a Markov assumption and prove that the arithmetic mean of these layer-wise ELBOs provides a valid lower bound on the global ELBO, establishing the theoretical basis for local training. Algorithmic contribution : we proposed SLL and demonstrate its potential as a scalable and efficient alternative to BP. By integrating stochastic random projections, SLL replaces the need for a complete backward pass, thereby facilitating structured local learning. Experimental evaluations : We demonstrate that SLL scales effectively across architectures and datasets, from MLPs on MNIST to ViTs on ImageNet. Our results show that the SLL algorithm surpasses recently proposed local training methods that address the update locking problem of BP. Moreover, SLL approaches or equals the accuracy performance of BP but with a significant reduction in memory ( 4 × or more). 0 0 footnotetext: Code available at: https://github.com/byin-cwi/SSL 2 Background In supervised learning tasks, such as classification applications or regression, neural networks are designed to construct mappings between given input data X and the corresponding target label Y . Traditional feedforward neural networks have a sequential structure in which each layer processes the output of the previous layer through a parameterized function. Following the classical formulation [ 1 ] , such a L -layer neural network can be expressed as a chain of its parameterized sub-functions: f_{1:L}(x):=f(f(...f(x,\theta_{1})...,\theta_{L-1}),\theta_{L}) (1) where \theta_{i}\in\Theta represents a set of learnable parameters at layer i . This hierarchical or Markov structure introduces a sequence of hidden representations \mathcal{H}=[h_{1},h_{2},...,h_{L}] where each representation is defined recursively as h_{i}=f(h_{i-1},\theta_{i}) . Given the stacked structure of neural networks, each layer builds on the representation of the previous layer. This structure induces a hierarchical representation where higher layers encode increasingly abstract and task-relevant features. Backpropagation is the standard approach for network training, aiming to optimize the parameters \Theta of

Read the complete text