Layerwise Progressive Freezing Enables STE-Free Training of Deep Binary Neural Networks
Evan Gibson Smith, Bashima Islam · 2026
Read Layerwise Progressive Freezing Enables STE-Free Training of Deep Binary Neural Networks by Evan Gibson Smith, Bashima Islam (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22660.txt.
Introduction
Evan Gibson Smith and Bashima Islam train deep binary networks by freezing layers in stages, with no gradient shortcut. Binary networks use one-bit weights and activations, which makes them tiny and fast. Their 2026 preprint tests progressive freezing in place of the straight-through estimator, the usual trick that fakes a gradient.
Plain global freezing works for binary weights, they find, but fails for full binary nets. Blocked gradients through activations are the cause. The find maps where the trick can and cannot go. Tiny nets matter for phones and sensors. Honest limits guide the design.
Catalog record
| CREATOR | Evan Gibson Smith, Bashima Islam |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22660
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Layerwise Progressive Freezing Enables STE-Free Training of Deep Binary Neural Networks Evan Gibson Smith, Bashima Islam Abstract We investigate progressive freezing as an alternative to straight-through estimators (STE) for training binary networks from scratch. Under controlled training conditions, we find that while global progressive freezing works for binary-weight networks, it fails for full binary neural networks due to activation-induced gradient blockades. We introduce StoMPP (Stochastic Masked Partial Progressive Binarization), which uses layerwise stochastic masking to progressively replace differentiable clipped weights/activations with hard binary step functions, while only backpropagating through the unfrozen (clipped) subset (i.e., no straight-through estimator). Under a matched minimal training recipe, StoMPP improves accuracy over a BinaryConnect-style STE baseline, with gains that increase with depth (e.g., for ResNet-50 BNN: +18.0 on CIFAR-10, +13.5 on CIFAR-100, and +3.8 on ImageNet; for ResNet-18: +3.1, +4.7, and +1.3). For binary-weight networks, StoMPP achieves 91.2% accuracy on CIFAR-10 and 69.5% on CIFAR-100 with ResNet-50. We analyze training dynamics under progressive freezing, revealing non-monotonic convergence and improved depth scaling under binarization constraints. 1 Introduction Quantized neural networks promise reduced memory footprint and faster, more energy-efficient inference, but training them remains challenging: optimization proceeds in a continuous parameter space, while deployment constrains parameters (and sometimes activations) to a discrete set. We focus on the extreme end of this spectrum, binary networks, where weights and/or activations take values in \{-1,+1\} . Binary-weight networks (BWNs) and full binary neural networks (BNNs) can substantially reduce bandwidth and arithmetic cost, but often suffer from accuracy degradation and optimization instability. Most successful approaches to training BWNs and BNNs rely on straight-through estimator (STE) style updates, popularized by BinaryConnect and subsequent BNN training methods ( 7 ) . These methods use a hard binarization operator (e.g., \mathrm{sign} ) in the forward pass and replace its derivative in the backward pass with a surrogate. While empirically effective, this introduces a mismatch between the executed forward operator and the gradient used for learning, and the mismatch can become more pronounced as networks deepen. Prior work improves BNN performance through refined approximations, optimizers, and architectural modifications ( 43 ; 41 ; 42 ; 36 ; 30 ; 29 ) , but most gradient-based training pipelines still depend on STE-like gradient replacements in some form. Motivated by the STE mismatch, we investigate whether progressive freezing can serve as an alternative training paradigm for binary networks trained from scratch . Progressive and soft-to-hard methods (e.g., incremental quantization, proximal objectives, and annealing schedules) offer principled ways to transition from continuous to discrete representations ( 46 ; 31 ; 1 ; 44 ; 13 ; 22 ) , but their extension to full BNNs at depth is non-trivial. In particular, we find that global progressive freezing, even with stochastic masking, can train BWNs, but fails to train full BNNs due to activation-induced gradient blockades : once a unit is committed to a hard step function, gradients cannot propagate through it to earlier layers, preventing learning in deep networks (Fig. 1 ). Early Training Mid Training Late Training Global Layer-wise Unfrozen node Frozen node Active gradient Frozen gradient Blocked gradient Figure 1 : Comparison of masking strategies in StoMPP. Top: Global stochastic masking randomly freezes activations (nodes) and weights (edges) throughout training. Bottom: Layer-wise stochastic masking freezes entire layers sequentially from input to output. Blue indicates active gradient paths, red indicates frozen elements (sign function), and orange indicates edges blocked by frozen target nodes. We introduce StoMPP ( Sto chastic M asked P artial P rogressive Binarization), a layerwise progressive freezing procedure with stochastic masking. StoMPP progressively hardens both weights and activations from a differentiable clipped operator to a hard binary operator, while backpropagating only through the unfrozen (clipped) subset. In contrast to STE training, StoMPP does not apply a hard binary operator in the forward pass and a different surrogate derivative in the backward pass; instead, gradients correspond to the forward operator used on trainable components, and committed components are treated as constants. This yields an estimator-free progressive training paradigm for both BWNs and full BNNs, especially beneficial at greater depth. Our key contributions are listed below. 1. We identify activation-induced gradient blockades as the key failure mode of global progressive freezing in full BNNs, while BWNs remain trainable under global schedules. 2. We propose layerwise progressive freezing with stochastic masking, mitigating gradient blockades by scheduling commitment from input to output. 3. Under a matched minimal training recipe that isolates algorithmic effects, StoMPP improves top-1 accuracy over a BinaryConnect-style STE baseline with gains that increase with depth (ResNet-18: +3.1, +4.7, +1.3 accuracy; ResNet-50: +18.0, +13.5, +3.8 on CIFAR-10/100/ImageNet). 4. We characterize StoMPP’s optimization behavior, including non-monotonic convergence and sensitivity to the freezing schedule. Under the same minimal recipe (full precision defaults without weight decay or learning rate schedulers), StoMPP achieves 80.9% top-1 on CIFAR-10 and 53.8% top-1 on CIFAR-100 with ResNet-18 BNNs, and the improvements over STE are consistent and larger for deeper architectures. We further show StoMPP pairs well with architectural improvements, applying it to Bi-Real Net and suggesting progressive freezing is largely orthogonal to model design. 2 Related Work Binary Neural Networks and STE. Binary neural networks (BNNs) quantize weights and activations to \{-1,+1\} , while binary-weight networks (BWNs) quantize only weights. The dominant approach for training BNNs is the straight-through estimator (STE) ( 2 ) , which applies the non-differentiable \mathrm{sign}(\cdot) in the forward pass and uses a surrogate gradient in the backward pass. STE enabled a wide range of successful BNNs ( 36 ; 30 ; 29 ; 42 ) , but it introduces an inherent forward/backward mismatch: the backward update does not correspond to the gradient of the discrete forward computation. Empirically, this mismatch is associated with optimization instability and degraded scaling in deeper networks under standard training recipes ( 34 ) . Largely, literature improves STE training by proposing better surrogate gradients or gradient shaping to reduce the mismatch and stabilize training ( 43 ; 42 ; 41 ; 30 ; 35 ) , and stochastic variants such as probabilistic binarization have also been explored ( 7 ) . These methods typically retain surrogate-gradient backpropagation through binarized operations. Differentiable Relaxations and Annealing. To avoid STE, several works optimize differentiable relaxations that are annealed toward a hard quantizer ( 22 ; 44 ; 13 ) . During the relaxed phase, forward and backward computations are aligned, but training can become sensitive as the relaxation hardens (e.g., brittle schedules or vanishing/exploding gradients near the discrete limit). In contrast, our goal is to train the discrete model without surrogate gradients by explicitly controlling when and where discrete variables are enforced. Progressive Quantization and Freezing. Progressive quantization methods convert a continuous model to a quantized one in stages by freezing subsets of parameters and retraining the remainder. INQ ( 46 ) progressively fixes groups of weights to quantized values in groups deterministically so remaining weights can adapt, and related stochastic methods probabilistically freeze weights during training ( 9 ) . These approaches are largely developed for weight-only quantization and do not address the additional difficulty of binary activations , which can block gradient flow in the absence of STE. Alpha-blending StoMPP extends progressive freezing to full BNNs (weights and activations) without STE by combining (i)