Naturally Computed Scale Invariance in the Residual Stream of ResNet18
André Longon · 2025
Read Naturally Computed Scale Invariance in the Residual Stream of ResNet18 by André Longon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16290.txt.
Introduction
André Longon inspects how the residual stream of ResNet18 builds invariance to image scale. Vision systems must know an object across changes in size, light, and turn that leave its identity whole. Past circuit-level work mapped such tricks in a different network that lacks a residual stream, leaving this design dark.
In his 2025 study, many mid-level channels prove scale invariant. The trick is an element-wise sum: a small-scale copy from the block input joins a large-scale copy from deeper in the block. Knockout tests then probe whether these traits drive scale-robust naming. The early verdict is tentative but pointed: the residual stream itself may compute the invariance.
Catalog record
| CREATOR | André Longon |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-16290
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Naturally Computed Scale Invariance in the Residual Stream of ResNet18 André Longon Abstract An important capacity in visual object recognition is invariance to image-altering variables which leave the identity of objects unchanged, such as lighting, rotation, and scale. How do neural networks achieve this? Prior mechanistic interpretability research has illuminated some invariance-building circuitry in InceptionV1, but the results are limited and networks with different architectures have remained largely unexplored. This work investigates ResNet18 with a particular focus on its residual stream, an architectural component which InceptionV1 lacks. We observe that many convolutional channels in intermediate blocks exhibit scale invariant properties, computed by the element-wise residual summation of scale equivariant representations: the block input’s smaller-scale copy with the block pre-sum output’s larger-scale copy. Through subsequent ablation experiments, we attempt to causally link these neural properties with scale-robust object recognition behavior. Our tentative findings suggest how the residual stream computes scale invariance and its possible role in behavior. Code is available at: https://github.com/cest-andre/residual-stream-interp 1 Introduction Deep neural networks (DNNs) and brains learn to perform sophisticated tasks but leave us ignorant of how they ultimately solve them. One such task is visual object recognition, which maps an image onto an object category. A critical property of object recognition is invariance: to recognize objects across arbitrary positions, angles, scales (from depth or size), lighting, and backgrounds. How do artificial and biological neural networks accomplish this? Invariant object recognition is a central focus of study in visual neuroscience [ 3 ] , but the complexities of the brain have rendered it difficult to elucidate the underlying neural algorithms, especially in deeper regions of the ventral stream. Alternatively, as DNNs are the leading models of neural response predictions in visual processing [ 23 , 10 , 20 , 19 ] , reverse-engineering these models may help uncover the mechanisms from which invariant object recognition emerges in the brain. In the digital realm, the field of mechanistic interpretability has progressed in elucidating circuitry of object recognition DNNs [ 16 ] . InceptionV1 [ 22 ] is the most well-studied DNN where many features have been categorized [ 15 ] , including detectors for curves [ 1 , 7 ] and high-low frequency boundaries [ 21 ] . Most relevant to this study, Olah et al . [ 17 ] discovered within InceptionV1 equivariances for hue, rotation, and scale, as well as some circuitry that build invariances. To expand the mechanistic interpretation of invariant object recognition in DNNs, we zoom into the residual stream of ResNet18 [ 8 ] . The residual stream, which InceptionV1 lacks, is an architectural mechanism that permits features to bypass layers of processing via their summation to the output of a downstream layer. We selected ResNet18 as it is the smallest member of the ResNet family, most manageable to perform a network-scale study of its stream. We present the following results: 1. With simple quantitative criteria, we find residual stream channels which exhibit properties of scale invariance. The input channel of a block possesses a smaller scale version of a feature, the same channel at the block’s pre-sum output possesses the larger scale, and these are summed together to produce the final scale invariant output. This pattern is especially pronounced in intermediate blocks. 2. When these channels are ablated, they cause more damage to scale-robust object recognition performance compared to other random channels. This is evidence of a causal link from scale invariant neurons to scale invariant behavior. Furthermore, this work suggests a function for so-called “bypass connections” in biological visual systems (macaque [ 13 ] and mouse [ 4 ] ), an anatomical analogue to the residual stream. The hypothesis that bypass connections may compute scale invariance is perhaps worth exploring for visual neuroscience and to further establish universality between artificial and biological neural networks. 2.1 : channel 15 21 48 68 88 108 3.1 : channel 22 29 80 113 178 215 Figure 1 : Grids of maximally exciting images for exemplary scale invariant channels in block 2.1 (top row) and 3.1 (bottom row). All channels are zero-indexed. Each 3x3 grid is for a single channel across the block’s target layers and is organized as follows. Columns from left to right: block channel at input, pre-sum, and post-sum layers. Rows from top to bottom: FZs optimized for the channel’s center neuron, FZs for the entire channel, and the top 9 center-neuron activating natural images from the ImageNet validation set [ 2 ] . 2 Methods We use the PyTorch [ 18 ] torchvision implementation of ResNet18 with the default ImageNet-trained [ 2 ] weights, IMAGENET1K_V1 . Neuron activations are obtained via the THINGSvision library [ 12 ] . 2.1 Residual stream architecture This work is only concerned with the activations from three layers for a given ResNet block. We name the layers \boldsymbol{In} , \boldsymbol{Pre} , and \boldsymbol{Post} for the block’s input, pre-sum output, and post-sum output respectively (the block index is not notated as we analyze them in isolation). \boldsymbol{Pre} is the block’s final batch normalization. \boldsymbol{In} is the layer whose output bypasses the main layers of the block and added to the output of \boldsymbol{Pre} (either previous block’s output or downsample module output). \boldsymbol{Post} is the sum of outputs from \boldsymbol{In} and \boldsymbol{Pre} followed by a ReLU. We denote the center-neuron activation of layer \boldsymbol{In} in channel c to image X as \boldsymbol{In}_{c}(X) (equivalent notation for \boldsymbol{Pre} and \boldsymbol{Post} ). 2.2 Feature visualization Our results rely on feature visualization, an optimization algorithm used to generate images which maximally activate a target channel (or an individual neuron) in a given layer. Such images are henceforth referred to as feature visualizations (FZs). We use the version of feature visualization described by [ 14 ] and implemented in the Lucent library 1 1 1 https://github.com/greentfrapp/lucent for PyTorch. We use near-identical regularizations as prescribed by [ 14 ] for improved interpretability, with altered jitter values described in the appendix (see Section 7.1 ). We use a channel’s center-neuron optimized FZs for all subsequent analyses. In a given block of the residual stream, for a given channel c , we obtain the center-neuron FZs for the three target layers, which we denote as \hat{X}_{In_{c}} , \hat{X}_{Pre_{c}} , and \hat{X}_{Post_{c}} . Thus, \boldsymbol{In}_{c}(\hat{X}_{In_{c}}) is the center-neuron activation elicited by its own FZ for channel c of a given block’s input layer. 3 Criteria reveal scale invariant channels Figure 2 : Scale-robust object recognition degradation from ablating scale invariant channels versus random non-invariant channels in block 2.1 (left) and 3.1 (right). The y-axis shows the mean ratios of top-1 ImageNet validation accuracy [ 2 ] between the two ablation conditions at a given scale transformation percentage (applied to all images), with black bars showing the standard error across the random trials. The blue line is the ratio when no scale transform is applied. The scale-transformed ratios are below the blue line, meaning scale invariant channel ablation did disproportionately more damage to accuracy during scale-robust object recognition. We assume that if a particular post-sum output channel \boldsymbol{Post}_{c} at a given block is to be scale invariant, then \boldsymbol{In}_{c} will represent the smaller-scale copy, and \boldsymbol{Pre}_{c} will represent the larger-scale copy. This is a reasonable assumption as the