Rethinking the Role of Spatial Mixing
George Cazenavette 1 Joel Julin 2 Simon Lucey 3 1 Massachusetts Institute of Technology 2 Carnegie Mellon University 3 University of Adelaide · 2025
Read Rethinking the Role of Spatial Mixing by George Cazenavette 1 Joel Julin 2 Simon Lucey 3 1 Massachusetts Institute of Technology 2 Carnegie Mellon University 3 University of Adelaide (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16760.txt.
Introduction
George Cazenavette, Joel Julin, and Simon Lucey rethought the role of spatial mixing in image models in 2025. A classic convolution mixes data across both space and channels at once. Newer models split those two jobs into apart steps, which lets researchers test each job on its own.
To their surprise, freezing the spatial mixers at their random start values cost almost no accuracy. That held on both a classic network and a newer mixer design. Models with fixed random spatial mixing also resisted attack images better. The effect reached beyond classification: such models could still decode images whose pixels had been shuffled. The results question how much learned spatial mixing really matters.
Catalog record
| CREATOR | George Cazenavette 1 Joel Julin 2 Simon Lucey 3 1 Massachusetts Institute of Technology 2 Carnegie Mellon University 3 University of Adelaide |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16760
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rethinking the Role of Spatial Mixing George Cazenavette 1 Joel Julin 2 Simon Lucey 3 1 Massachusetts Institute of Technology 2 Carnegie Mellon University 3 University of Adelaide Abstract Until quite recently, the backbone of nearly every state-of-the-art computer vision model has been the 2D convolution. At its core, a 2D convolution simultaneously mixes information across both the spatial and channel dimensions of a representation. Many recent computer vision architectures consist of sequences of isotropic blocks that disentangle the spatial and channel-mixing components. This separation of the operations allows us to more closely juxtapose the effects of spatial and channel mixing in deep learning. In this paper, we take an initial step towards garnering a deeper understanding of the roles of these mixing operations. Through our experiments and analysis, we discover that on both classical (ResNet) and cutting-edge (ConvMixer) models, we can reach nearly the same level of classification performance by only learning channel mixing and leaving the spatial mixers at their random initializations. Furthermore, we show that models with random, fixed spatial mixing are naturally more robust to adversarial perturbations. Lastly, we show that this phenomenon extends past the classification regime, as such models can also decode pixel-shuffled images. 1 Introduction For the better part of the last two decades, cascades of learned convolutions have formed the backbone of nearly every innovation in the field of computer vision and pattern recognition. From distinguishing ten digits with LeNet LeCun et al. 1998 to one thousand classes with AlexNet Krizhevsky et al. 2012 , from going very deep with VGG Simonyan and Zisserman 2014 to even deeper with InceptionNet Szegedy et al. 2015 , the learned 2D convolution has served as the workhorse that ushered in the new era of visual learning. Convolutional neural networks (CNNs) learned to exploit the correlations across channels and between spatially close pixels to solve a plethora of tasks. Recently, isotropic networks (those in which the size of the representation stays fixed throughout) such as Vision Transformer Dosovitskiy et al. 2020 , Image GPT Chen et al. 2020 , MLP-Mixer Tolstikhin et al. 2021 , and ConvMixer Trockman and Kolter 2022 have been grabbing the field’s attention. These isotropic models consist of repeated blocks wherein each block consists of a spatial mixing operation (self-attention Dosovitskiy et al. 2020 ; Chen et al. 2020 , spatial MLP Tolstikhin et al. 2021 , or depthwise convolution Trockman and Kolter 2022 ) followed by one or more strictly channel-mixing layers (1 \times 1 or “pointwise” convolutions). A strictly spatial mixing operation is defined here as any operation that is applied independently across each channel. Similarly, a strictly channel-mixing layer is applied independently across spatial pixel coordinates. The fact that all these recent state-of-the-art isotropic architectures spend a significant portion of their computation budget solely on channel-mixing parameters hints toward the possibility that channel mixing may have significantly more relative importance than previously appreciated. It has been understood for some time that impressive performance can be obtained from networks whose weights are not all completely learned Saxe et al. 2011 . Before the success of AlexNet Krizhevsky et al. 2012 in training deep end-to-end networks, the use of random weights in early convolutional layers played an important role in training deeper CNNs. The approach was attractive as it promised faster training times, better generalisation, and the ability to learn deeper networks. Saxe et al. Saxe et al. 2011 even offered strong theoretical motivations for why random filter weights in CNNs would offer good performance in terms of: (i) frequency selection, and (ii) translation invariance. One of the most notable works with respect to using random weights within networks can be found in the Extreme Learning Machines (ELMs) of Huang et al Huang et al. 2006 . In its simplest form an ELM takes a proposed network architecture and randomly initializes all hidden weights, leaving only the final layer to learn. ELMs are advantageous as they allow for extremely deep networks and rapid train times (as only the final layer needs to be learned). It is widely understood, however, that these training strategies have a significant performance gap in relation to their current end-to-end learned counterparts. In this work, we leverage the separable convolution to compare the relative effects of spatial and channel mixing on the performance of deep neural networks. By leaving either the spatial-mixing or channel-mixing parameters frozen at their random initialization and only training the others, we can isolate the contributions of both types of parameters. Doing so reveals that learning the channel-mixing parameters is far more critical to the effectiveness of a deep model, and such models that only learn channel-mixing parameters perform nearly just as well as their fully-learned counterparts. We also show that models that only learn channel-mixing parameters are naturally more resistant to adversarial attacks than fully-learned models. This robustness can be further enhanced by then directly smoothing the random, frozen spatial-mixing filters. Lastly, we show that this phenomenon extends beyond the regime of classification problems and find that models that only learn spatial mixing are even capable of learning to invert a permutation of pixels nearly just as well as fully-learned networks. We hope that these gained insights as to the role and effectiveness of spatial and channel mixing in deep models will be of great value to the vision community and that they will shed light on which parts of deep models are most critical to learn while helping to inspire the intelligent design of future architectures. 2 Related Work Isotropic Architectures First popularized by the transformer Vaswani et al. 2017 , isotropic architectures (those in which the size of the representation remains fixed throughout) have more recently made their way into the computer vision world. Image-GPT Chen et al. 2020 (based on the GPT language model Chen et al. 2020 ) was able to model sequences of pixels and generate new images in raster fashion. Designed for discriminative tasks, the Vision Transformer Dosovitskiy et al. 2020 used transformer blocks to achieve state-of-the-art image classification results, but only after extensive pre-training. The MLP-Mixer Tolstikhin et al. 2021 then built off Vision Transformer’s success, but used a simple spatial-mixing MLP instead of the expensive self-attention module. While the MLP-Mixer still required extensive pre-training, the newest addition to the line of isotropic architectures, the ConvMixer Trockman and Kolter 2022 , replaces the self-attention or spatial-mixing MLP with a simple depthwise filter bank and achieves comparable results without any pre-training. The main thing all these models have in common is their isotropic structure. Here, “isotropic” refers to repeated blocks of operations in which the latent representation does not change in shape. Furthermore, in all of these models, the second half of each isotropic block is made up of one or more strictly-channel-mixing layers. Such transformer-inspired isotropic models have recently come to be known as “meta-formers” Yu et al. 2022 . Separable Convolutions Instead of mixing across channel and spatial dimensions simultaneously, the separable convolution factorizes the operation into a depthwise and pointwise convolution. The depthwise convolution applies a disjoint set of depth-1 filters to each channel of the input independently while the pointwise convolution is simply a 1 \times 1 convolution (or linear projection) on