Stretching Beyond the Obvious: A Gradient-Free Framework to Unveil the Hidden Landscape of Visual Invariance
Lorenzo Tausani, Paolo Muratore, Morgan B. Talbot, Giacomo Amerio, Gabriel Kreiman, Davide Zoccolan · 2025
Read Stretching Beyond the Obvious: A Gradient-Free Framework to Unveil the Hidden Landscape of Visual Invariance by Lorenzo Tausani, Paolo Muratore, Morgan B. Talbot, Giacomo Amerio, Gabriel Kreiman, Davide Zoccolan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17040.txt.
Introduction
Lorenzo Tausani and five co-authors present a probe for finding what changes an image without changing what a visual system sees in it, in a 2025 paper. Knowing a unit's favorite image is not enough; recognition depends on which changes leave the response alone. Their framework, Stretch-and-Squeeze, needs no gradients and works on both brain cells and artificial networks. It frames the search as a two-goal optimization problem.
One goal stretches the image as far as possible while the internal response holds steady, mapping the unit's invariance. The reverse search exposes weak spots open to adversarial tricks. A shared probe for biological and machine vision can show where the two truly agree and where artificial systems only mimic the surface of sight.
Catalog record
| CREATOR | Lorenzo Tausani, Paolo Muratore, Morgan B. Talbot, Giacomo Amerio, Gabriel Kreiman, Davide Zoccolan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-17040
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Stretching Beyond the Obvious: A Gradient-Free Framework to Unveil the Hidden Landscape of Visual Invariance Lorenzo Tausani, Paolo Muratore, Morgan B. Talbot, Giacomo Amerio, Gabriel Kreiman, Davide Zoccolan Abstract Uncovering which features’ combinations high-level visual units encode is critical to understand how images are transformed into representations that support recognition. While existing feature visualization approaches typically infer a unit’s most exciting images, this is insufficient to reveal the manifold of transformations under which responses remain invariant, which is key to generalization in vision. Here we introduce Stretch-and-Squeeze (SnS), an unbiased, model-agnostic, and gradient-free framework to systematically characterize a unit’s invariance landscape and its vulnerability to adversarial perturbations in both biological and artificial visual systems. SnS frames these transformations as bi-objective optimization problems. To probe invariance, SnS seeks image perturbations that maximally alter the representation of a reference stimulus in a given processing stage while preserving unit activation. To probe adversarial sensitivity, SnS seeks perturbations that minimally alter the stimulus while suppressing unit activation. Applied to convolutional neural networks (CNNs), SnS revealed image variations that were further from a reference image in pixel-space than those produced by affine transformations, while more strongly preserving the target unit’s response. The discovered invariant images differed dramatically depending on the choice of image representation used for optimization: pixel-level changes primarily affected luminance and contrast, while stretching mid- and late-layer CNN representations altered texture and pose respectively. Notably, the invariant images from robust networks were more recognizable by human subjects than those from standard networks, supporting the higher fidelity of robust CNNs as models of the visual system. Overall, SnS offers a powerful tool to uncover the invariant manifold of a unit, moving beyond tests with predefined transformations and advancing our understanding of generalization and robustness in visual systems. 1 Introduction Both visual neuroscience and deep learning seek to understand image processing systems composed of millions of interacting functional units (biological or artificial neurons), whose activity patterns are shaped by their experience with natural image statistics [ 1 , 2 ] . This common goal raises a fundamental question in both fields: which combination of image features do visual neurons become tuned for? Traditionally, this question has been addressed by developing feature visualization approaches that discover the ”preferred” stimuli that maximally activate a given unit within the network - often referred to as the unit’s most exciting images (MEIs) [ 3 , 4 , 5 , 6 , 7 ] . MEIs however only reveal a few instances within the vast set of images that strongly activate a given unit [ 8 , 9 ] , offering poor insight into the manifold of transformations under which the unit’s activity remains invariant. To overcome this limitation, we developed Stretch-and-Squeeze (SnS), an unbiased, model-agnostic method to probe visual invariance in both artificial and biological neurons. Based on previous work [ 10 ] , SnS optimization exploits evolutionary algorithms, but with a different objective: to find invariant or adversarial images, rather than MEIs. SnS integrates the search for invariant images and adversarial examples within a unified optimization objective, generating image perturbations starting from a reference image (e.g., a MEI of the unit). For invariance, SnS explicitly looks for images that are maximally distinct from this reference stimulus in the representation of a chosen visual processing stage, while preserving the response of a target unit upstream. This allows SnS to characterize the invariance manifolds of the selected unit when the representation of an effective stimulus (either a MEI or a natural image) is stretched at different processing stages. As a result, our approach reveals the actual image variation axes the unit is able to tolerate, providing a richer, more veridical description of its invariance landscape, as compared to traditional tests based on predefined (e.g., affine) transformations [ 11 , 12 , 13 ] . In our study, we carried out comprehensive tests of the effectiveness of SnS to achieve the goals listed above, using a popular CNN (ResNet50) as a benchmark. We found dramatic differences among the invariance landscapes of ResNet50 readout units when stretching the representations at different depths of the processing hierarchy. Stretching early, middle, and late representations yielded invariant images that differed from the reference (and among themselves) in terms of luminance/contrast, texture, and pose, respectively. Such qualitative differences were confirmed by measuring how the three classes of invariant images were accurately discriminated by Support Vector Classifiers (SVCs) in the pixel space. We also discovered important differences in these hierarchical invariances between the standard and an adversarially trained version of the network, with invariant images from the robust network being more recognizable by human subjects and other observer networks. Finally, we checked the potential applicability of SnS to visual neuroscience experiments, by showing that the method works even if the experimenter can record the activity of just a small fraction of the units in the processing stage where the stretching is applied. 2 Related works Probing CNN representations: feature visualization, invariance, and adversarial examples. The question of functional interpretability and feature visualization in deep learning has predominantly been approached by employing gradient-based optimization for image synthesis, taking advantage of the complete analytical description and differentiability of the network [ 3 , 4 , 5 ] . While most studies have focused on finding the MEIs of CNNs’ units, recent efforts have started to also explore their invariance landscape. A prominent approach relies on discovering model metamers - i.e., synthesized stimuli that match the internal representation of a reference (natural) image in a specific layer of a network [ 14 ] . Using metamers, Feather et al. [ 15 ] were able to demonstrate that standard CNNs display highly idiosyncratic invariances at the top layers of processing, with metamers being unintelligible to human observers or other neural networks. In contrast, CNNs trained to be robust to adversarial images (i.e., imperceptibly modified inputs capable of altering CNN object classification [ 16 ] ) yielded metamers that were substantially more interpretable by human observers. This shows how invariance and adversarial vulnerability are conceptually related [ 17 ] . For instance, adversarially trained ("robust") networks can craft subtle image perturbations that not only fool the network but can also impair [ 18 ] or enhance [ 19 ] human object recognition, thereby reinforcing the perceptual parallels between robust network representations and invariance in human vision. Applications to visual neuroscience. Advances in CNN interpretability have also impacted neuroscience, which has long suffered from a lack of effective methods to investigate the functional tuning of visual neurons in higher cortical areas. By leveraging the strong functional analogy between CNNs trained for image classification and the primate object recognition pathway (known as the ventral stream [ 20 ] ), it is possible to create digital twins of the ventral stream using CNNs [ 21 , 22 , 23 ] . This allows employing gradient-based optimization to synthesize images that modulate the activity of biological neurons [ 24 ] or investigate properties like adversarial robustness in visual neurons [ 25 ] . The digital twin paradigm has been also extended beyond primate vision, informing models of the auditory cortex [ 26 ] and visual processing in rodents [ 27 , 7 , 28 ] .