Energy-Efficient Deep Learning Without Backpropagation: A Rigorous Evaluation of Forward-Only Algorithms

Przemysław Spyra, Witold Dzwinel · 2025

Read Energy-Efficient Deep Learning Without Backpropagation: A Rigorous Evaluation of Forward-Only Algorithms by Przemysław Spyra, Witold Dzwinel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01061.txt.

Introduction

Przemyslaw Spyra and Witold Dzwinel benchmark forward only learning algorithms, which train neural networks without backpropagation. Backpropagation must store every activation and wait for a global backward sweep, so replacements promise big energy savings. The authors note those promises usually rest on operation counts, not on measurements from real hardware.

Across almost a thousand runs on four vision benchmarks, they read energy, power, and memory directly from a modern GPU. Most forward only methods used several times more energy than tuned backpropagation. One, Mono-Forward, was the exception. On the largest networks tested it used less energy and memory at similar speed. Trained to convergence, it beat backpropagation on CIFAR accuracy, though it needed more passes.

Catalog record

CREATORPrzemysław Spyra, Witold Dzwinel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-01061

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Energy-Efficient Deep Learning Without Backpropagation: A Rigorous Evaluation of Forward-Only Algorithms Przemysław Spyra, Witold Dzwinel Abstract The long-held assumption that backpropagation (BP) is essential for state-of-the-art performance is challenged by this work. We present rigorous, hardware-validated evidence that the Mono-Forward (MF) algorithm, a backpropagation-free method, consistently surpasses an optimally tuned BP baseline in classification accuracy on its native Multi-Layer Perceptron (MLP) architectures. This superior generalization is achieved with profound efficiency gains, including up to 41% less energy consumption and up to 34% faster training. Our analysis, which charts an evolutionary path from Geoffrey Hinton’s Forward-Forward (FF) to the Cascaded Forward (CaFo) and finally to MF, is grounded in a fair comparative framework using identical architectures and universal hyperparameter optimization. We further provide a critical re-evaluation of memory efficiency in BP-free methods, empirically demonstrating that practical overhead can offset theoretical gains. Ultimately, this work establishes MF as a practical, high-performance, and sustainable alternative to BP for MLPs. 1 Introduction Deep neural networks (DNNs) have achieved transformative success across AI ( Krizhevsky, Sutskever, and Hinton 2012 ; He et al. 2016 ; Devlin et al. 2019 ) , yet this progress is built upon an escalating and unsustainable energy budget. As models scale, their operational requirements are projected to consume between 3% and 4% of global electricity by 2030 ( World Economic Forum and Accenture 2025 ) , with the training of a single large model yielding a carbon footprint equivalent to that of several automobiles ( Abbas 2024 ) . The predominant training algorithm, backpropagation (BP), is the primary driver of this inefficiency ( Rumelhart, Hinton, and Williams 1986 ) . BP’s dependence on a full backward pass to compute gradients necessitates storing all intermediate activations, leading to high memory costs. Furthermore, its sequential nature imposes a “backward locking” bottleneck, which fundamentally constrains parallelization and prolongs training ( Jaderberg et al. 2017 ) . This sustainability crisis has spurred a search for alternative algorithms free of backpropagation (BP free), inspired by the local and efficient learning principles of biological neural systems. This paper investigates an evolutionary progression of these methods: • Forward-Forward (FF) ( Hinton 2022 ) , a foundational proof of concept demonstrating the viability of local, forward-only learning. • Cascaded Forward (CaFo) ( Zhao et al. 2023 ) , a structured evolution employing block-wise local classifiers to integrate supervisory signals more directly. • Mono-Forward (MF) ( Gong, Li, and Abdulla 2025 ) , a recent refinement using local projection matrices to achieve superior performance and efficiency. While the potential of these algorithms is well recognized, a comprehensive and fair evaluation based on direct hardware-level energy metrics remains insufficiently explored in the current literature. Existing work often overlooks architectural differences or inconsistent tuning practices, confounding the results ( Desislavov et al. 2024 ) . This paper addresses this gap by conducting a systematic, hardware-validated comparison of FF, CaFo, and MF against fairly tuned BP baselines on their respective native architectures. Contributions The main contributions of this work are threefold: • A Rigorous Comparative Framework: We establish a transparent framework for the fair comparison of BP-free algorithms, mandating identical native architectures, universal hyperparameter tuning, and direct hardware-level efficiency measurements. • Validation of a Superior BP-Free Algorithm: We provide the first hardware-validated evidence that the Mono-Forward (MF) algorithm not only matches but surpasses backpropagation in both classification accuracy and training efficiency on MLP architectures. • An Empirical Re-evaluation of Memory Efficiency: We conduct a critical, empirical analysis of memory usage in BP-free methods, revealing that theoretical benefits are often counteracted by practical implementation overheads, thus correcting a common misconception. 2 An Evolution of BP Free Algorithms The high memory cost, sequential backward locking, and lack of biological plausibility of backpropagation have motivated a search for alternative training paradigms ( Lillicrap et al. 2020 ; Jaderberg et al. 2017 ) . This search has led to an evolution of BP free, forward only learning algorithms. Forward-Forward (FF) Proposed by Geoffrey Hinton, the FF algorithm was a foundational conceptual advance ( Hinton 2022 ) . It replaces the forward and backward passes of BP with two forward passes, one using “positive” (real) data and another using “negative” (contrastive) data. Each layer learns locally to distinguish between these data types by optimizing a layer specific “goodness” metric. By eliminating the global backward pass, FF demonstrated that effective representation learning is possible with purely local objectives. However, its reliance on generating negative samples and its indirect goodness signal were identified as practical limitations, often leading to slow convergence and inefficient hardware utilization ( Hinton 2022 ; Zhao et al. 2023 ) . Cascaded Forward (CaFo) The CaFo algorithm builds on FF by introducing a more structured, block wise framework ( Zhao et al. 2023 ) . It comprises a cascade of neural blocks, each followed by a local predictor (an auxiliary classifier). This design provides a more direct supervisory signal at multiple network depths, eliminating the need for negative samples. The neural blocks can be randomly initialized and fixed (‘CaFo-Rand‘) or trained with a BP free method like Direct Feedback Alignment (‘CaFo-DFA‘). While CaFo addresses some of FF’s limitations, it introduces a critical trade-off, where ‘CaFo-Rand‘ suffers from poor feature quality on complex tasks, while ‘CaFo-DFA‘ improves accuracy but at a significant computational cost for block pretraining ( Zhao et al. 2023 ) . Mono-Forward (MF) MF represents a recent refinement that simplifies and enhances the local learning mechanism ( Gong, Li, and Abdulla 2025 ) . Instead of using contrastive data or auxiliary classifiers, MF equips each hidden layer with a dedicated, learnable projection matrix. This matrix directly maps the layer’s activations to class specific “goodness” scores, upon which a standard cross entropy loss is computed locally. Both the layer’s primary weights and its projection matrix are updated using only this local loss signal. This elegant mechanism reportedly achieves accuracy that matches or surpasses BP on MLP architectures with significant efficiency gains, which motivated the rigorous validation in this work. 3 Framework for Fair and Rigorous Comparison As a key contribution of this work, we propose a rigorous framework designed to isolate the impact of the training algorithm from confounding variables. Fairness Protocols A strict set of protocols was adhered to for all experiments: 1. Native Architecture Replication: Each alternative algorithm was implemented on its native architecture as described in its source publication. FF and MF were evaluated on their designated MLP structures, while CaFo was evaluated on its defined three block CNN. 2. Identical Architectures for BP Baselines: For each experiment, a corresponding BP baseline was constructed using an identical network architecture, omitting only algorithm specific components. 3. Systematic and Universal Hyperparameter Tuning: Systematic hyperparameter optimization using the Optuna framework ( Akiba et al. 2019 ) was applied to all algorithms, FF, CaFo, MF, and their respective BP baselines, for every configuration, maximizing validation set accuracy. 4. Consistent Early Stopping: Training for all models was governed by a consistent early stopping protocol based on validation performance. The model checkpoint yielding the best validation score was used for all final evaluations ( Prechelt 1998 ) . Experimental Setup Experiments were conducted on the MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets. All experiments were executed on NVIDIA

Read the complete text