How Data Augmentation Shapes Neural Representations

Tianxiao He, Alex H. Williams, Sarah E. Harvey · 2026

Read How Data Augmentation Shapes Neural Representations by Tianxiao He, Alex H. Williams, Sarah E. Harvey (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15306.txt.

Introduction

Tianxiao He, Alex H. Williams, and Sarah E. Harvey map how data augmentation recasts neural representations. Using shape analysis, they embed hidden codes in a metric space fixed to scaling, translation, rotation, and reflection. Stronger boosting traces well-behaved trajectories in this space, and other boosting types steer codes in distinct ways. Warps along boosting paths predict which codes most improve groups. Shared shape patterns emerge across architectures and seeds.

Catalog record

CREATORTianxiao He, Alex H. Williams, Sarah E. Harvey
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42786
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15306

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Data Augmentation Shapes Neural Representations Tianxiao He, Alex H. Williams, Sarah E. Harvey Abstract Data augmentation is widely recognized for improving generalization in deep networks, yet its impact on the geometry of learned representations remains poorly understood. In this work, we characterize how different data augmentation strategies reshape internal representations in neural networks. Using tools from shape analysis, we embed network hidden representations into a metric space where distance is invariant to scaling, translation, rotation and reflection. We show that increasing augmentation strength leads to well-behaved trajectories in this space, and that different augmentation types steer representations in distinct directions. Moreover, we investigate how neural representation shapes are distorted along data augmentation trajectories, and show that insights from neural geometry can predict which representations provide the most improvement when ensembling models. Our results reveal shared geometric patterns across architectures and seeds, and suggest that analyzing shape-space trajectories offers a principled tool for understanding and comparing data augmentation methods. Open source code is available at https://anonymous.4open.science/r/netrep-analysis-00EC 1 Introduction Due to the overparametrized nature of deep neural networks, many hyperparameter settings often achieve similar task performance. Tuning hyperparameters such as learning rate, batch size, and momentum is known to be crucial for performance ( 24 ) , but there is little direct study on how these choices change the geometry of learned neural representations. In particular, when a single hyperparameter is isolated and gradually varied, how do features of hidden layer activity change? A particularly rich area to study this phenomenon is data augmentation. Data augmentation (DA) is a foundational technique for improving the performance of deep networks by diversifying existing datasets with simple transformations ( 23 ) . In the context of computer vision, classic DA techniques include rotation, cropping, color jitter and addition of Gaussian noise to some fraction of input images. DA is often interpreted as an implicit regularization technique ( 4 ) or the incorporation of prior knowledge about invariances in the task ( 18 ; 3 ) . Despite their centrality in modern training pipelines, data augmentations are often tuned using simple heuristics or searches that optimize task performance. However, many distinct combinations of augmentation hyperparameters, such as strength, probability, or spatial extent, can yield near-identical accuracy on held-out data. This degeneracy suggests that performance metrics alone may obscure meaningful differences in how models learn to represent the world under different training protocols. While the benefits of DA are intuitive, a deeper understanding remains elusive. For example, is there a sense in which DA methods move representations along well-defined trajectories as the hyperparameter is varied? Do distinct augmentation methods (e.g. image rotations and random cropping) sculpt hidden layer computations in fundamentally different ways, or are their effects redundant? How do two augmentation methods interact when they are simultaneously applied? More broadly, does diversity in representational geometry predict the extent of functional improvements when models are ensembled across different hyperparameter values? Can two augmentations synergistically interact or interfere with each other, and can this be observed from their hidden representational geometry? As a first step towards addressing these questions, we leverage previously developed metric spaces on the geometry of hidden layer representations ( 27 ; 17 ) . This framework identifies the geometry of hidden layer representations as points on a smooth manifold of possible geometries, enabling tools from differential geometry to study the effect of hyperparameter changes on learned representations. In particular, gradual changes in augmentation strength can be conceptualized as moving the representational geometry along a smooth path on this manifold. Different types of augmentation give rise to different paths over the manifold, and we can compute the lengths and curvatures of these paths, as well as the angles formed at intersections. Altogether, this enables a systematic comparison of how augmentation strength and types reshape the representation geometry across network layers, initializations, and architectures. Our main contributions are: • We propose a geometric framework for analyzing the effects of data augmentation hyperparameters on learned neural representations • We demonstrate that increasing data augmentation magnitude steers representations along structured trajectories in this shape space , but not in the naive Euclidean space • We study how the magnitude of representation shape change due to data augmentation compares with the magnitude of representation change due to a different random seed • We show that greater divergence between trajectories of different augmentations predicts larger gains from model ensembling • We find that data augmentation displaces shape landmarks non-uniformly, with effects that depend on network depth and data augmentation type. 2 Related Work Data augmentation. Data augmentation (DA) is a central technique in modern machine learning pipelines, used to increase data diversity, improve generalization, and enhance robustness ( 23 ; 26 ; 2 ; 4 ) . It is generally thought that augmentations shape neural computation by “collapsing” the manifold of representations along nuisance dimensions, so that augmented views of the same input are mapped to similar responses ( 18 ) . The broader principle of view-invariant representations can be traced back decades within the computer vision and visual neuroscience literature ( 21 ; 22 ; 7 ) . Importantly, the intuition that training with augmentations “collapses” augmented views onto a single point only pertains to the representational geometry of the augmented dataset—it does not tell us how the manifold of representations over clean images is altered by training with augmentations. This distinction is particularly important for augmentation techniques that produce out-of-distribution training samples (e.g. adding a large amount of Gaussian noise), with the goal of improving performance of unseen within-distribution samples. In this paper, we investigate how different augmentations alter representations of non-augmented “clean images” in the test set. Representational similarity. Comparing neural network representations is essential for understanding training dynamics, transferability, and model equivalence. Representational similarity metrics aim to characterize relationships between hidden layer activation patterns across layers and models. Metrics like centered kernel alignment (CKA) measure pairwise similarity between activation matrices using kernel-based alignment ( 16 ) , and are widely used to study how layer representations change across seeds or architectures ( 20 ) . More recent work has proposed using formal metric spaces to capture more nuanced geometric structure in representations ( 27 ) which allows for more advanced analysis methods on representations such as regression and clustering. 17 leveraged this idea to study the trajectories of representations through layers of deep neural networks, and also introduce concepts from Riemannian geometry to study these paths. While our work draws on similar methodological ideas, we apply them to study the trajectories of neural representations across models as hyperparameters are tuned. 3 Methods In this section, we define our notion of a metric space for neural representations. We then discuss geodesics of this metric and how to measure angles between trajectories of representations. Lastly, we discuss how to interpret the direction of trajectories in this space in terms of landmark displacements , which measure how much the hidden layer response changes to each image input after nuisance transformations are removed. Figure 1: Embedding into shape space enables more meaningful comparison of large ensembles of representations. (a) After two models are trained end-to-end using different data augmentation methods, M unaltered images from the test dataset are randomly

Read the complete text