Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks
Evan Gibson Smith, Bashima Islam · 2026
Read Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks by Evan Gibson Smith, Bashima Islam (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27759.txt.
Introduction
Binary neural networks store weights and activations as single bits. They are cheap to run but hard to train from scratch. The standard straight-through estimator creates a mismatch that deepens with depth. Evan Gibson Smith and Bashima Islam study a different axis in 2026: when and where binarisation is enforced. Their method, StoMPP, binarises layer by layer from input to output. Stochastic partial masks with soft refresh manage the transition.
StoMPP works as a fully estimator-free training rule on its own. On a binary ResNet-50, it beat the standard estimator clearly. Gains grew with depth across CIFAR and ImageNet tests. Adding the estimator only on frozen entries improved results further. One mechanism explains both regimes. Progression order is decisive. Forward progression prevents depth collapse, while reverse progression falls to near-chance accuracy. Committed binary activations block gradient flow upstream, the authors show. Ordering controls when those blockades form.
Catalog record
| CREATOR | Evan Gibson Smith, Bashima Islam |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27759
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks Evan Gibson Smith, Bashima Islam Abstract Training binary neural networks (BNNs) from scratch is dominated by the straight-through estimator (STE), whose forward/backward mismatch produces severe accuracy degradation as networks deepen. We study an orthogonal axis: when and where binarization is enforced during training. We introduce StoMPP (Stochastic Masked Partial Progressive Binarization), which gradually replaces clipped weights and activations with their hard binary counterparts layer by layer from input to output, using stochastic partial masks with soft refresh. StoMPP delivers two complementary benefits. As a standalone training rule, it provides a fully STE-free procedure that improves over vanilla STE with gains that grow with depth (ResNet-50 BNN: +18.0/+13.5/+3.8 on CIFAR-10/100/ImageNet), and the pattern holds across ResNet-18/34/50, MobileNetV2, and BERT fine-tuning. Composed with surrogate gradients by applying STE only to frozen entries, it reaches +27.1/+19.8/+17.7 over vanilla STE on the same setting. Underlying both regimes is a single mechanistic finding: progression order is decisive. Forward layerwise progression prevents depth collapse, reverse progression collapses to near-chance, and binary-weight networks (without binary activations) are insensitive to order. We trace this asymmetry to activation-induced gradient blockades: a committed binary activation severs gradient flow upstream, and ordering controls when these blockades form. To isolate the progression’s contribution from any benefit conferred by STE, we conduct all ablations in the STE-free regime; the resulting characterization (schedule, refresh, ordering, dynamics) thus reflects the progression itself rather than its interaction with surrogate gradients. 1 Introduction Deep neural networks have made dramatic progress on tasks ranging from vision to language, but their growing memory and compute footprints make deployment on resource-constrained hardware increasingly difficult. Binary neural networks address this by constraining weights, and sometimes activations, to \{-1,+1\} , which can reduce inference cost by an order of magnitude or more. These methods fall into two regimes: binary-weight networks (BWNs), where only weights are binarized, and full binary neural networks (BNNs), where activations are binarized as well. Training either regime from scratch is difficult: the forward pass uses a non-differentiable sign operator, and standard gradient-based optimization requires some way to propagate learning signal through it. The dominant solution is the straight-through estimator (STE) ( 8 ; 3 ) , which substitutes a surrogate gradient in the backward pass. STE has enabled most successful BNN training pipelines, but its forward/backward mismatch is associated with progressive accuracy degradation as networks deepen. This depth sensitivity is known ( 55 ) , and most prior work responds to it by improving STE itself. Approaches include refined approximations of the sign function, gradient shaping, learned scaling, and specialized optimizers ( 50 ; 46 ; 49 ; 40 ; 33 ; 32 ) . These efforts retain the basic STE structure of a hard forward operator paired with a surrogate backward pass, and seek to reduce the resulting mismatch from within that structure. We pursue an orthogonal direction: rather than improving the gradient that flows through a fixed binarization, we change when and where binarization is enforced during training. Progressive quantization methods offer one realization of this idea, gradually committing parameters from continuous to discrete values over the course of training, and have been used effectively for weight-only quantization 56 ; 34 ; 2 ; 51 ; 16 ; 24 . Extending this strategy to full BNNs is non-trivial. Naively applying a global progressive schedule, in which any parameter or activation in the network can be committed at any time, succeeds for BWNs but fails for BNNs: training collapses or stalls at depth even though the same schedule works in the weight-only setting. We trace this failure to a mechanism we call activation-induced gradient blockades . In a full BNN, freezing a binary activation replaces it with a hard sign operator whose derivative is zero almost everywhere; once a unit is committed in this way, gradients cannot propagate through it to earlier layers, preventing learning in deep networks (Fig. 1 ). Under a global progressive schedule, frozen activations can appear at arbitrary depths, and any path through a frozen activation loses gradient signal upstream. BWNs are resistant to this failure because their activations remain continuous, so freezing only the weights leaves gradient flow intact. This asymmetry is the central observation that motivates our method. We introduce StoMPP ( Sto chastic M asked P artial P rogressive Binarization), a layerwise progressive freezing procedure with stochastic masking. StoMPP progressively hardens both weights and activations from a differentiable clipped operator to a hard binary operator, scheduling commitment from input to output so that the layer currently transitioning always has an unfrozen suffix providing a gradient path to the loss. The frozen-entry gradient is a design choice: setting it to zero yields a fully STE-free training procedure, while applying STE only to frozen entries (StoMPP+STE) recovers a learning signal through committed units. Both regimes share the same progression structure, and we ablate the progression in the STE-free regime to isolate its contribution from any benefit conferred by surrogate gradients. Our contributions are as follows: 1. We introduce StoMPP, a layerwise progressive freezing procedure with stochastic masked partial freezing and soft refresh. The progression is independent of the choice of frozen-entry gradient, supporting both a fully STE-free training procedure and a composition with STE applied only to frozen entries. 2. We show that progression order is decisive for full BNNs: forward layerwise succeeds on deep networks, reverse layerwise collapses to near-chance, and global progression degrades at depth, while BWNs are largely insensitive to ordering. We discuss this asymmetry in terms of activation-induced gradient blockades. 3. We analyze the progression’s training dynamics, including non-monotonic sawtooth convergence as layers transition, and show that layerwise scheduling is substantially more robust to schedule shape and refresh rate than global scheduling. Our evaluation shows that both StoMPP variants improve over a BinaryConnect-style STE baseline under a matched minimal training recipe, with gains that grow with depth. On ResNet-50 BNNs, STE-free StoMPP improves over vanilla STE by +18.0/+13.5/+3.8 on CIFAR-10/100/ImageNet, while StoMPP+STE reaches +27.1/+19.8/+17.7 on the same setting. The same pattern holds across ResNet-18/34, MobileNetV2, and BERT fine-tuning on SST-2, indicating that the progression’s benefit is not specific to ResNet-style vision models. We further show that StoMPP transfers to binary-specific architectures by combining it with Bi-Real Net without architectural modification. 2 Related Work Binary Neural Networks and STE. Binary neural networks (BNNs) quantize weights and activations to \{-1,+1\} , while binary-weight networks (BWNs) quantize only weights. The dominant approach for training BNNs is the straight-through estimator (STE) ( 8 ; 3 ) , which applies the non-differentiable \mathrm{sign}(\cdot) in the forward pass and uses a surrogate gradient in the backward pass. STE enabled a wide range of successful BNNs ( 40 ; 33 ; 32 ; 49 ) , but it introduces an inherent forward/backward mismatch: the backward update does not correspond to the gradient of the discrete forward computation. This can result in optimization instability and degraded scaling in deeper networks under standard training recipes ( 55 ; 38 ) . Largely, literature improves STE training by proposing better surrogate gradients or gradient shaping to reduce the mismatch and stabilize training ( 50 ; 49 ; 46 ; 33 ; 39 ) . Beyond convolutional networks, binarization