Energy-Efficient Supervised Learning with a Binary Stochastic Forward-Forward Algorithm

Risi Jaiswal, Supriyo Datta, Joseph G. Makin · 2025

Read Energy-Efficient Supervised Learning with a Binary Stochastic Forward-Forward Algorithm by Risi Jaiswal, Supriyo Datta, Joseph G. Makin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06461.txt.

Introduction

Risi Jaiswal, Supriyo Datta, and Joseph Makin wrote this machine learning paper, published in 2025. It trains neural networks with a forward-forward method built for binary, coin-flipping units, aiming to slash the energy cost of learning.

The standard training method, backpropagation, is awkward for custom hardware because it stores every step and runs backward. Forward-forward training avoids that, but its heavy step is matrix multiplication. Making units binary turns those multiplications into simple table lookups. Randomness in the units, plus shared weights with varied biases, restores the information flow that binary units would choke. The authors note that probabilistic bits, tiny unstable magnets, can do the random sampling almost for free. On image benchmarks, accuracy neared the full-precision version while estimated energy fell about tenfold.

Catalog record

CREATORRisi Jaiswal, Supriyo Datta, Joseph G. Makin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06461

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Energy-Efficient Supervised Learning with a Binary Stochastic Forward-Forward Algorithm Risi Jaiswal, Supriyo Datta, Joseph G. Makin Abstract Reducing energy consumption has become a pressing need for modern machine learning, which has achieved many of its most impressive results by scaling to larger and more energy-consumptive neural networks. Unfortunately, the main algorithm for training such networks, backpropagation, poses significant challenges for custom hardware accelerators, due to both its serial dependencies and the memory footprint needed to store forward activations for the backward pass. Alternatives to backprop, although less effective, do exist; here the main computational bottleneck becomes matrix multiplication. In this study, we derive forward-forward algorithms for binary, stochastic units. Binarization of the activations transforms matrix multiplications into indexing operations, which can be executed efficiently in hardware. Stochasticity, combined with tied weights across units with different biases, bypasses the information bottleneck imposed by binary units. Furthermore, although slow and expensive in traditional hardware, binary sampling that is very fast can be implemented cheaply with p-bits (probabilistic bits), novel devices made up of unstable magnets. We evaluate our proposed algorithms on the MNIST, Fashion-MNIST, and CIFAR-10 datasets, showing that its performance is close to real-valued forward-forward, but with an estimated energy savings of about one order of magnitude. † † impactstatement: Over the last decade, neural networks have become ubiquitous in scientific, industrial, and commercial applications, but suffer from a major defect: they consume large amounts of energy, both during training and at run time. The resources needed to train AI systems have been doubling approximately every 3.4 months over this period, which is likely unsustainable, at least without major environmental costs. Here we propose an algorithm for deep learning that maintains high accuracy on moderately difficult benchmark image-processing tasks, but with a projected energy consumption approximately a tenth that of standard training algorithms. Our algorithm replaces multiplications with binary operations and minimizes data movement, allowing for operation on ultra-low-power hardware. We demonstrate performance on GPUs, and calculate from first principles the expected energy costs on dedicated hardware. We anticipate that these results will motivate the fabrication of a new type of AI chip, bringing on-device learning to everyday devices. I Introduction In recent years, deep neural networks (DNNs) have revolutionized natural language processing, computer vision, automatic speech recognition, and various scientific domains [ 1 , 2 , 3 , 4 ] . Tasks that seemed exceedingly complex even 15 years ago are now feasible, and have commercially deployed solutions. However, a major component of this progress has been the use of complex models that require substantial computational resources, typically in the form of high-powered GPU clusters. In fact, the computational resources needed to train AI systems have been doubling approximately every 3.4 months since 2012 [ 5 ] . It is not clear that the energy demands required to sustain this pace of development can be met, especially given concerns about anthropogenic climate change [ 6 , 7 ] . While there has been a considerable emphasis on accelerating inference methods [ 8 , 9 , 10 , 11 , 12 ] to run on low-energy devices, enhancements in training methodologies have not kept pace. This is because nearly all DNNs are trained with the backpropagation algorithm [ 13 ] , which presents two major obstacles to implementation on efficient custom hardware. First, backprop requires the unit activations computed in the forward pass to be stored (for use in computing the gradient in the backward pass). This imposes a significant memory cost. Furthermore, although the forward pass can be computed efficiently in analog hardware [ 14 , 15 , 16 , 17 ] , storage in memory requires a digital representation, and analog-to-digital conversion dominates the silicon area and power consumption in these methods [ 15 , 17 ] . Second, the forward passes cannot be run asynchronously or on pipelined data, but must be interrupted by weight updates that traverse the full depth of the network. Motivated in part by these considerations, there has been recent interest in developing alternatives to backprop [ 18 , 19 , 20 , 21 , 22 ] . For example, Hinton has recently proposed an algorithm for supervised learning, called “forward-forward,” that uses only forward passes through the DNN. Although less effective than backprop, forward-forward avoids both of the aforementioned obstacles to implementing backprop in hardware. Nevertheless, there is another obstacle to implementing neural networks in energy-efficient, analog hardware: noise accumulation. Analog/mixed-signal neurons are more susceptible to degradation in classification accuracy as bitwidth increases compared to their digital counterparts [ 23 ] . This requires careful tuning of design to make it work. A potential solution is to quantize the activations or the weights into fewer bits [ 11 , 24 ] . At the extreme, activations could even be binarized [ 25 , 26 ] . This is appealing because matrix multiplications against binary vectors can be carried out as indexing and adding operations, which can be executed very efficiently in hardware [ 27 , 28 , 29 ] . Moreover, some of the precision lost to binarization can be recouped by sampling from neurons. For example, a pair of binary, stochastic neurons reporting (let us suppose) the same feature will together provide (on average) more information than their deterministic counterparts. Setting aside the issue of matrix multiplications, this scheme might seem far less efficient than simply quantizing less severely. But in fact a recent hardware innovation called the p-bit has made it possible to draw Bernoulli-distributed voltages at very high speeds, low power consumption, and small footprint [ 30 ] . We discuss the p-bit below (Section II-C ). Taken together, these considerations point toward a new, more energy-efficient approach to supervised learning, based on (1) foward-pass-only algorithms rather than backprop; (2) binary, stochastic neurons; and (3) implementation in hardware with p-bits. To demonstrate the feasibility of this approach, in this work we extend to binary, stochastic activations both Hinton’s original forward-forward algorithm [ 19 ] and a more powerful recent extension [ 31 ] . We demonstrate that our binary stochastic forward-forward algorithms achieve accuracy comparable to their real-valued counterparts. Although we implement our algorithms in software, we estimate that the hardware implementation would save approximately one to two orders of magnitude in energy costs. II Background II-A Alternatives to backpropagation Our work combines two distinct research directions, one focused on non-backpropagation-based algorithms and the other on propagating gradients through binary stochastic neurons. The motivation behind non-backpropagation methods for training deep neural networks primarily arises from the biological implausibility of backpropagation. Many key challenges for biological implementations—weight transport, the need for symmetric weights, non-local computation, propagating global error signals from the output to every layer—likewise pose difficulties for efficient hardware implementations. However, certain other biologically implausible features, like weight sharing in convolutional layers, can enhance hardware performance by saving memory and improving energy efficiency in digital circuits. Feedback Alignment (FA) [ 32 ] addresses the weight transport problem by using random feedback weights, eliminating the need for symmetric weights. Similarly, Direct Feedback Alignment (DFA) [ 33 ] directly uses error signals from output layers with random feedback weights instead of sequentially propagating. Although FA and DFA circumvent the problems of weight

Read the complete text