From Local Cues to Global Percepts: Emergent Gestalt Organization in Self-Supervised Vision Models
Tianqin Li, Ziqi Wen, Leiran Song, Jun Liu, Zhi Jing, Tai Sing Lee · 2025
Read From Local Cues to Global Percepts: Emergent Gestalt Organization in Self-Supervised Vision Models by Tianqin Li, Ziqi Wen, Leiran Song, Jun Liu, Zhi Jing, Tai Sing Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00718.txt.
Introduction
Tianqin Li and five co-authors asked whether vision models group parts into whole shapes the way human eyes do, in a 2025 paper. Human sight uses rules of grouping, such as closure and figure-ground split, to build forms from local cues. The team tested which training styles let such grouping appear in a model.
Models trained by hiding and rebuilding image patches showed grouping-like behavior, such as filling in missing edges. The team built a test set that scrambles global layout while keeping local texture, and the self-taught models beat standard trained ones there, at times beating people too. Extra training for naming images wore the skill down. A cap that keeps only the strongest signals switched on brought much of the skill back.
Catalog record
| CREATOR | Tianqin Li, Ziqi Wen, Leiran Song, Jun Liu, Zhi Jing, Tai Sing Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00718
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Local Cues to Global Percepts: Emergent Gestalt Organization in Self-Supervised Vision Models Tianqin Li, Ziqi Wen, Leiran Song, Jun Liu, Zhi Jing, Tai Sing Lee Abstract Human visual perception organizes local cues into coherent global forms using Gestalt principles such as closure, proximity, and figure–ground assignment—functions that rely on sensitivity to global spatial structure. In this work, we investigate whether modern vision models exhibit similar perceptual behaviors, and under what training conditions such behaviors emerge. We first show that Vision Transformers (ViTs) trained with Masked Autoencoding (MAE) display internal activation patterns consistent with Gestalt laws, including illusory contour completion, convexity preference, and dynamic figure–ground segregation. To better understand the computational basis of this behavior, we hypothesize that modeling global visual dependencies is a necessary condition for Gestalt-like organization to arise. To evaluate this capability directly, we introduce the Distorted Spatial Relationship Testbench (DiSRT)—a benchmark that tests a model’s sensitivity to global spatial relationship perturbations while preserving local textures. Using DiSRT, we find that self-supervised models (e.g., MAE, CLIP) outperform supervised baselines and, in some cases, even exceed human performance. Interestingly, ConvNeXt models trained with MAE also exhibit Gestalt-compatible representations, suggesting that global structure sensitivity can emerge independently of architectural inductive biases. Yet this capability proves fragile: standard classification finetuning substantially degrades a model’s DiSRT performance, indicating that supervised objectives may suppress global perceptual organization. Inspired by biological vision, we show that a simple Top-K activation sparsity mechanism can effectively restore global sensitivity in these models. Altogether, our results identify the training conditions that promote or suppress Gestalt-compatible perception in deep models, and establish DiSRT as a principled diagnostic for global perceptual sensitivity acoss architectures and training objectives. 1 Introduction Human visual perception is holistic wagemans2012centuryI ; wagemans2012centuryII ; wertheimer1923 : rather than interpreting scenes as collections of isolated elements, the brain spontaneously organizes them into coherent structures. This process is governed by a set of well-established heuristics known as the Gestalt principles—such as closure, proximity, continuity, and figure-ground organization—which describe how global perceptual structure arises from the spatial configuration of local cues. Importantly, these principles depend on the global relationships among visual elements. For example, perceiving a Kanizsa triangle requires integrating disconnected edge segments into an illusory shape ( kanizsa1976subjective ) ; identifying a figure in front of a background relies on the comparison of convexity and border ownership across an extended spatial field ( wagemans2012centuryI ) . In this sense, Gestalt perception is not merely a psychological curiosity—it is a behavioral indicator of how the visual system processes long-range dependencies across space. If a system exhibits Gestalt-like behaviors, it must encode information that spans beyond local receptive fields, capturing the joint configuration of distributed parts of the image. This insight raises a key question in modern computer vision: Do neural networks trained on natural images also develop Gestalt-like global sensitivities—and if so, under what conditions? While it has been suggested that Vision Transformers (ViTs), by virtue of self-attention, are well-suited to modeling long-range dependencies, the empirical evidence for their alignment with Gestalt perception remains limited. Prior work on “shape bias” and robustness has typically used style transfer or occlusion-based perturbations, which conflate texture insensitivity with structural understanding. In this paper, we provide a direct investigation of Gestalt perception in neural networks, using both classic visual illusions and quantitative benchmarks. We show that ViTs trained with Masked Autoencoding (MAE) exhibit internal representations aligned with Gestalt laws—e.g., forming coherent figure-ground separation, grouping elements by continuity and proximity, and preferring convex figures. We interpret this behavior as evidence that MAE encourages models to develop global perceptual integration capabilities, not merely through architectural inductive bias, but through the training objective itself. To evaluate this hypothesis more systematically, we introduce the Distorted Spatial Relationship Testbench (DiSRT), a new benchmark that assesses a model’s sensitivity to global structural perturbations while preserving local texture statistics. DiSRT isolates long-range visual dependencies by requiring models (or humans) to detect spatial relationship perturbation generated via texture synthesis. Through this benchmark, we demonstrate that: • Self-supervised models like MAE and CLIP dramatically outperform supervised baselines on DiSRT. • ConvNets, when trained with MAE, also develop Gestalt-like representations. • Standard classification finetuning erodes these global sensitivities, but can be partially recovered through a biologically inspired Top-K activation sparsity mechanism. Our results suggest that Gestalt perception can emerge naturally in modern vision systems, but is critically dependent on training objectives that promote global context modeling. These findings offer a bridge between classic theories of perception and current advances in self-supervised learning, providing new insights into the development of structured representations in neural networks. 2 Gestalt Perception in Mask Autoencoders Figure-Ground: The Kanisza Triangles Illusion. The Kanizsa Triangle illusion is a classic example of how our visual system perceives global structures through the Gestalt principle of closure. In this illusion, three ’pac-man’ shapes are arranged in such a way that the mind perceives an equilateral triangle, even though no triangle is physically drawn. This illusion highlights the brain’s ability to integrate local elements into a coherent whole, demonstrating the importance of global structure in visual perception. However, it’s not been clearly shown before that neural networks learned with natural image training also can see the Kanisza Illusion. Utilizing ViT models trained with Mask Autoencoder Objective he2022masked , we demonstrate that certain attention head in the ViT layers exhibits strong figure ground seperation capability – an ability that requires integrating the local visual cue into global perception as shown in Figure 1 (See caption for details). Figure 1 : Internal activation of ViT Mask Autoencoder on several illustory countour images. We project the internal activation of certain attention head to 3 dimension space using PCA and color the projection with RGB colors. The PCA basis is shared across different images. We can see that (1) the PCA clearly captures the direction that seperates the figure and the ground. (2) The Kanisza Trinagle is perceived when the illustory countour is aligned and not seen when the pac-man are not very well coordinated, demonstrating the ability of MAE trained ViT in Gestalt Perception. Please refer to the supplementary materials for method used here. Figure 2 : Other Gestalt Perception Rules learned in ViT-MAE: Law of Continuity and Proximity Law of Continuity and Proximity. In addition to demonstrating figure-ground separation, our analysis also reveals that ViT-MAE exhibit perceptual behaviors aligned with other Gestalt principles, notably Law of Continuity and Law of Proximity . The law of continuity posits that discrete elements aligned along a continuous line or curve are perceived as related. As shown in the Figure 2 Left, activations along the contour are much similar in activation regardless whether the pixel contains a dot or blank background. The entire curve is perceived as a whole rather than discretized composing dots. The model can also see law of proximity. On the right of Figure 2 , when groups of squares are spatially proximate, the ViT-MAE assigns similar activation values to these proximate objects (similar PCA projection color). We can see a clear grouping effect where two proximate groups have different activation values. These two additional laws further hint that ViT trained by MAE strategy could perceive global structure of the image exceptionally well, aligning with gestalt