The Affine Divergence: Aligning Activation Updates Beyond Normalisation

George Bird · 2025

Read The Affine Divergence: Aligning Activation Updates Beyond Normalisation by George Bird (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22247.txt.

Introduction

George Bird questions how neural network training updates activations, the signals flowing through a network, as opposed to its parameters. Parameters follow steepest descent as intended. Bird argues activations matter more directly, sitting closer to the loss and carrying sample-specific information. Yet their propagated updates do not take the optimal steepest-descent step. They suffer non-ideal sample-wise scaling in affine, convolutional, and attention layers.

Fixing that scaling turns out to be simple, and the fix incidentally re-derives normalization from first principles. Bird also identifies a genuinely different solution with no scale invariance, an alternative to the affine map that beat conventional normalizers in several tests. A convolutional cousin called PatchNorm follows. The paper reframes normalizers as activation-like maps with parameterized scaling.

Catalog record

CREATORGeorge Bird
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE107 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-22247

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Affine Divergence: Aligning Activation Updates Beyond Normalisation George Bird Abstract A systematic mismatch exists between mathematically ideal and effective activation updates during gradient descent. As intended, parameters update in their direction of steepest descent. However, activations are argued to constitute a more directly impactful quantity to prioritise in optimisation, as they are closer to the loss in the computational graph and carry sample-dependent information through the network. Yet their propagated updates do not take the optimal steepest-descent step. These quantities exhibit non-ideal sample-wise scaling across affine, convolutional, and attention layers. Solutions to correct for this are trivial and, entirely incidentally, derive normalisation from first principles despite motivational independence. Consequently, such considerations offer a fresh and conceptual reframe of normalisation’s action, with auxiliary experiments bolstering this mechanistically. Moreover, this analysis makes clear a second possibility: a solution that is functionally distinct from modern normalisations, without scale-invariance, yet remains empirically successful, outperforming conventional normalisers across several tests. This is presented as an alternative to the affine map. This generalises to convolution via a new functional form, “ PatchNorm ”, a compositionally inseparable normaliser. Together, these provide an alternative mechanistic framework that adds to, and counters some of, the discussion of normalisation. Further, it is argued that normalisers are better decomposed into activation-function-like maps with parameterised scaling, thereby aiding the prioritisation of representations during optimisation. Overall, this constitutes a theoretical-principled approach that yields several new functions that are empirically validated and raises questions about the affine + nonlinear approach to model creation. 1 Introduction A deep learning model uses two primary forms of variables to produce the desired outputs. These are the parameters tuned by optimisation, such as weights and biases, and the transient activations/representations, which depend on the specific input, are present in intermediate calculations, and are forward-propagated to the output. The objective of deep learning is to make stepwise adjustments to such quantities to produce a meaningful output. This is undertaken by backpropagation and an optimiser such as the gradient descent algorithm. Backpropagation computes the gradients of all intermediate quantities with respect to the loss; these gradients represent the direction of steepest descent that changes the loss the fastest with respect to that quantity. This calculation includes both activations and parameters; yet only the gradients of the parameters are used to update the model. One cannot update the activations directly because they are functions of the input, and this is used to determine their numerical value. Therefore, we update parameters. These become tuned through time, which in turn influences those transient activations, which are then propagated to yield the desired result. Hence, in some respects, parameters are updated as a practical proxy for desired changes in representations. For gradient descent, parameters are updated by subtracting the gradient of the loss with respect to the parameters, then scaling it by a learning rate scalar. This produces a slight change in the parameters in their direction of steepest descent with respect to the loss. For the batch of interest, in the parameter-picture, this is the optimal correction to reduce the loss the fastest. However, from backpropagation, the optimal steepest direction in which activations should adjust to reduce loss is known. The question becomes: Does the steepest descent for parameters correspond to the steepest descent for representations when propagated? The answer is a surprising no — even in the simplest of cases. This constitutes a fundamental and unavoidable structural misalignment in the update step in the representation space for affine maps. A divergence arises between the theoretically ideal correction, analytically available from the representation’s gradient, and the effective updates obtained by changing parameters and propagating these corrections to the representations. Yet, representations more directly influence the loss, carry sample-dependent information and ultimately forward propagate to the overall network output. The question becomes, why then, is the steepest descent for parameters prioritised whilst representations are neglected to diverge? . Furthermore, can they be constructed to coincide? This paper explores the consequences of adapting computations until such an alignment occurs, forming the steepest direction in both parameters and propagated representation corrections. Surprisingly, the solutions to such constraints naturally yield normalisation-like adaptations. Hence, normalisation emerges as a solution, not a premise. This was unintended and therefore offers a priori derivation naturally yielding normalisers. This may provide a fresh perspective on the success of such implementations in addressing this theoretical ‘affine divergence’. Empirical validation supporting this alternative hypothesis is then provided. Moreover, auxiliary hypotheses are developed indicating that, if this affine divergence mechanism is valid, these new normalisers should, counterintuitively, negatively correlate performance with increases in batch size. This is also empirically validated, adding an independent hypothesis which supports this “affine divergence” as a mechanistic lens on the success of normalisation. Moreover, such affine divergence solutions are not limited to normalisation-like corrections. A secondary ‘structural’ solution is presented that is neither normalisation-like nor scale-invariant, yet it is empirically successful and outperforms several forms of normalisation across tests. Good performance from this peculiar functional form classically may seem surprising without the affine divergence mechanism. It also provides a theoretical and empirical counterargument that partially undermines scale-invariance as a primary mechanistic cause of empirical success. Overall, this is argued to at least warrant some further discussion. Following the divergence to the extreme, one could propagate such gradient corrections to the final layer, but this is computationally expensive. However, these considerations are conceptually related to the natural gradient descent approach [ 2 , 1 , 10 ] , which considers the Riemannian geometry of the loss. The natural gradient approach can be understood in terms of the contravariant gradient \nabla\tilde{\mathcal{L}}=G^{-1}\nabla\mathcal{L} with respect to the final loss, in effect prioritising the steepest direction in the model’s output function space instead of parameter space. This induces the Fisher information as the metric to adapt backpropagated gradients, and, in turn, the loss decreases more efficiently, in such a way that the steepest descent is invariant to reparameterisations. However, despite conceptual alignment on the suboptimality of classical gradient descent’s direction of steepest descent for the loss, this natural gradient approach differs substantially from the methods presented in this paper in several key ways. Firstly, the key difference is which quantity’s steepest descent is prioritised with respect to the loss. Classically, these are the parameters, with natural gradients considering the overall model’s output, whilst this work prioritises the intermediate activations for steepest descent. An advantage of this middleground is that the steepest descent for activations is available from backpropagation; moreover, it is trivial to derive analytically. Secondly, this paper discusses two approaches to solving the affine divergence: gradient-only and structural corrections. This paper investigates the latter for various reasons that fundamentally alter the mathematics of the forward pass to align with ideal-effective representation steps. Whereas, if the natural gradient methodology were to be classified in such a picture, it would constitute the former, gradient-based corrections, rather than structural amendments to the network. Thirdly, the Reimannian approach also constitutes a distinct consideration from the ideal-effective argument, which follows from propagated parameter corrections as ‘effective’ changes on subsequent activations. These are two entirely different routes to determine optimality, and they are not mutually exclusive. Hence, although both cases displace the primacy of parameter’s steepest-descent updates, natural gradients

Read the complete text