Accelerated training through iterative gradient propagation along the residual path

Erwan Fagnou, Paul Caillon, Blaise Delattre & Alexandre Allauzen · 2025

Read Accelerated training through iterative gradient propagation along the residual path by Erwan Fagnou, Paul Caillon, Blaise Delattre & Alexandre Allauzen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17086.txt.

Description

Erwan Fagnou and co-authors speed up deep network training in a 2025 paper. Backpropagation, the core training method, runs layer by layer in strict order, which caps how fast very deep models can learn. Residual links fixed vanishing gradients but left that serial cost in place.

Their method, Highway backpropagation, breaks the gradient into parts that flow along each path in the network. It gathers gradient guesses along the shortcut path, then pushes them through all layers in parallel, round after round. The loop fits ResNets, Transformers, and recurrent nets alike. Broad tests show large speed gains with only slight drops in scores.

Catalog record

CREATORErwan Fagnou, Paul Caillon, Blaise Delattre & Alexandre Allauzen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-17086

Browse the full library · About the Tower of Babel