Is Backpropagation Optimal? When Synthetic Gradients Improve Sample Efficiency

Yibo Jacky Zhang, Zeyu Tang, Sanmi Koyejo · 2026

Read Is Backpropagation Optimal? When Synthetic Gradients Improve Sample Efficiency by Yibo Jacky Zhang, Zeyu Tang, Sanmi Koyejo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27946.txt.

Introduction

Yibo Jacky Zhang, Zeyu Tang, and Sanmi Koyejo ask whether backpropagation is truly the most sample-efficient way to train neural networks. In a unified feedback framework, they show synthetic gradients can estimate gradients with lower error under certain conditions, sometimes by an arbitrarily large margin. Bandit and reinforcement experiments support the theory.

Catalog record

CREATORYibo Jacky Zhang, Zeyu Tang, Sanmi Koyejo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE108144
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27946

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Is Backpropagation Optimal? When Synthetic Gradients Improve Sample Efficiency Yibo Jacky Zhang, Zeyu Tang, Sanmi Koyejo Abstract Backpropagation is the default learning rule for artificial neural networks and is often treated as the settled approach whenever differentiability is available. In this work, we revisit this convention through a theoretical lens of sample efficiency. We introduce a unified vectorized feedback framework for loss-based and reward-based learning on computational graphs, in which synthetic gradients emerge as a natural alternative to backpropagation. We characterize the conditions under which synthetic gradients can achieve a lower gradient-estimation mean squared error than backpropagation. We construct examples illustrating that this sample efficiency advantage can be arbitrarily large. Experiments on contextual bandits and reinforcement learning tasks demonstrate the potential of our theoretical findings. 1 Introduction Backpropagation ( Linnainmaa1976taylor ; werbos1981applications ; rumelhart1986learning ) has become the default learning rule for artificial neural networks, repeatedly outperforming alternatives (e.g., ( lillicrap2016random ; lee2015difference ; jaderberg2017decoupled ; scellier2017equilibrium ; hinton2022forward ) ). As a result, backpropagation (BP) is often treated as the settled answer to neural networks learning whenever differentiability is available. In this paper, we revisit this convention from the perspective of gradient-estimation sample efficiency. To formalize the sample efficiency analysis, we consider the gradient estimation problem on a computational graph with vectorized terminal feedback. This vectorized feedback framework unifies loss-based learning, where the vectorized feedback is the gradient of a loss, and reward-based learning, where it may be given by policy gradients ( williams1992simple ; sutton1999policy ) . In this picture, as detailed in the next section, backpropagation is analogous to a Monte Carlo (MC) estimator, and temporal-difference (TD) learning ( sutton1988learning ) provides a classical bootstrapped alternative to MC estimation. In the vectorized feedback setting, this bootstrapped alternative turns out to be the synthetic gradients ( jaderberg2017decoupled ; czarnecki2017understanding ) . Synthetic gradients (SG) were originally introduced from the perspectives of decoupled computation and bio-plausibility, and their connection to TD learning was also noted early on ( jaderberg2017decoupled ) . However, prior work largely studied synthetic gradients in essentially chain-structured computational graphs, and established no clear empirical or theoretical advantage over unconstrained backpropagation. As a result, the topic has gradually fallen out of favor, with only limited recent interest ( pemberton2024bpmathbflambda ; janfaza2023ada ) . Our point of departure is that this negative picture reflects an incomplete analysis. As with TD versus MC methods, the advantage of synthetic gradient propagation is not universal, but regime-dependent. TD methods can offer practical sample-efficiency advantages over MC methods through bootstrapping, and their theoretical analysis is an active area of research ( kearns2000bias ; lazaric2010finite ; pmlr-v75-bhandari18a ; cheikhi2023statistical ; cheng2024surprising ) . In contrast, synthetic gradients remain theoretically underexplored, with limited existing work focused mainly on training convergence ( czarnecki2017understanding ; huo2018decoupled ) . However, the analogy between TD and synthetic gradients suggests a possible sample-efficiency advantage over backpropagation, which leads to our central question: When can synthetic gradients improve gradient-estimation sample efficiency over backpropagation? Formally, this paper studies gradient estimation on differentiable computational graphs. We compare the mean squared error of empirical gradient estimators formed from i.i.d. samples under backpropagation and synthetic gradient propagation. The main contribution of this paper is thus a formal theoretical investigation into this underexplored yet potentially important question. Our results show that backpropagation is not necessarily the minimum-MSE estimator of the true gradient when gradient feedback contains uncertainty. We identify conditions under which synthetic gradients reduce gradient-estimation MSE . Our results can be summarized as follows. We introduce a vectorized feedback framework in Section 2 that unifies loss-based and reward-based learning on computational graphs. Motivated by this framework, we formally define the gradient estimation problem in Section 3 . We then identify two conditions, defined formally in Section 4 , that govern the comparison. Condition (A) says there must be uncertainty in gradient estimations. Specifically, (A) requires at least one of the following to hold: (A.1) randomness in the terminal gradient feedback ; (A.2) internal stochasticity in the network computation ; (A.3) sparse connectivity in the network that leads to partial observability at individual nodes . Condition (B) is a local sufficiency requirement : each node’s local observation contains sufficient information for the conditional mean of its incoming gradient feedback. Under these conditions, the comparison between backpropagation and synthetic gradients becomes sharp. Our theory yields the following separation. With condition \neg(A)\wedge(B) , backpropagation is optimal in estimating mean squared error (MSE) among all conditionally unbiased gradient estimators. In contrast, with (A)\wedge(B) , an oracle synthetic gradient generator achieves a smaller estimation MSE than backpropagation. Thus, for synthetic gradient propagation to improve sample efficiency, some source of randomness or structural sparsity is necessary. This also helps explain why prior work often found synthetic gradients to be worse: synthetic gradients were typically evaluated in deterministic and fully connected networks on relatively clean datasets, essentially where condition (A) is absent. When condition (B) does not hold exactly, or when the synthetic gradient is not an idealized oracle, synthetic gradients may be biased. In this case, we show that a mixture of synthetic and backpropagated gradients can be preferable due to a better bias-variance tradeoff. Inspired by the conditions (A) and (B) , in Section 5 we show examples where synthetic gradient propagation can be arbitrarily more sample efficient than backpropagation with estimated non-oracle gradient predictors. This construction exploits a concrete realization of the local sufficiency condition (B) : a network of specialized expert nodes that decomposes a global task into locally sufficient components. Motivated by our theoretical findings, in Section 6 we apply synthetic gradients to a class of sparsely connected networks. We evaluate this approach on a contextual bandit problem with MNIST contexts ( lecun2002gradient ) and on POPGym, a recent reinforcement learning benchmark for partially observable environments ( moradpopgym ) . In both settings, the results reflect our theoretical findings: synthetic gradient propagation improves sample efficiency when non-trivial gradient uncertainty is present. We then discuss related work in Section 7 . Interestingly, the conditions that favor synthetic gradient propagation in our analysis also resemble several features of biological neural systems. We discuss this connection, along with limitations and future directions, in Section 8 . 2 Vectorized Feedback on Computational Graphs In this section, we formalize the analogy between TD versus MC estimation and synthetic gradients versus backpropagation on a layered DAG. We begin by fixing some general notation. Vectors and matrices use boldface characters, e.g., {\bm{g}} and {\bm{J}} . Unless otherwise specified, these are random variables. A sample is denoted with a subscript, e.g., {\bm{g}}_{i} , while an estimator formed from samples is denoted by a hat, e.g., \hat{{\bm{g}}} . We use a tilde to denote quantities associated with synthetic gradient propagation, e.g., \tilde{{\bm{g}}} . We write [n]:=\{1,2,\dots,n\} . Additional notation is introduced as needed. Consider a neural network modeled by a layered DAG G=(V,E) . Let {\bm{x}}_{v} denote the activation at a neuron v\in V . For each layer l=0,\dots,L , let V_{l} denote the set of nodes in that layer. The input-layer (layer l=0 ) receives external inputs, and then each non-input node v\in V_{l} computes its activation through a (possibly stochastic) differential function based on the collective activations {\bm{z}}_{v} of its parent neurons

Read the complete text