Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering and Manipulating Human Perceptual Variability

Chen Wei, Chi Zhang, Jiachen Zou, Haotian Deng, Dietmar Heinke, Quanying Liu · 2025

Read Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering and Manipulating Human Perceptual Variability by Chen Wei, Chi Zhang, Jiachen Zou, Haotian Deng, Dietmar Heinke, Quanying Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03641.txt.

Introduction

Chen Wei and five co-authors synthesize images that sit exactly on a neural network's perceptual boundary, then use them to steer human judgment. Their 2025 paper starts from the spread in human decisions: two people can look at one image and honestly disagree. The team wanted stimuli that reliably split viewers.

Their framework, BAM, samples along a network's decision boundaries to mint such borderline images. Trials with 246 participants across more than 116,000 judgments validated the effect, yielding a dataset of 19,943 annotated images. Personalized model alignment plus adversarial generation then let them predict and even steer pairs of viewers' split decisions. The authors offer the toolkit for studying individual differences in perception. It turns disagreement from noise into the object of study.

Catalog record

CREATORChen Wei, Chi Zhang, Jiachen Zou, Haotian Deng, Dietmar Heinke, Quanying Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-03641

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering and Manipulating Human Perceptual Variability Chen Wei, Chi Zhang, Jiachen Zou, Haotian Deng, Dietmar Heinke, Quanying Liu Abstract Human decision-making in cognitive tasks and daily life exhibits considerable variability, shaped by factors such as task difficulty, individual preferences, and personal experiences. Understanding this variability across individuals is essential for uncovering the perceptual and decision-making mechanisms that humans rely on when faced with uncertainty and ambiguity. We propose a systematic Boundary Alignment Manipulation (BAM) framework for studying human perceptual variability through image generation. BAM combines perceptual boundary sampling in ANNs and human behavioral experiments to systematically investigate this phenomenon. Our perceptual boundary sampling algorithm generates stimuli along ANN decision boundaries that intrinsically induce significant perceptual variability. The efficacy of these stimuli is empirically validated through large-scale behavioral experiments involving 246 participants across 116,715 trials, culminating in the variMNIST dataset containing 19,943 systematically annotated images. Through personalized model alignment and adversarial generation, we establish a reliable method for simultaneously predicting and manipulating the divergent perceptual decisions of pairs of participants. This work bridges the gap between computational models and human individual difference research, providing new tools for personalized perception analysis. † † affiliationnotice: Equal contribution Figure 1: Human perceptual variability. For the same set of stimuli, individuals often exhibit varied responses, highlighting differences in their visual perception. For instance, the digit on the left may be interpreted as a ”3” by some and as a ”5” by others. Similarly, the image on the right might be initially perceived as a cat by some individuals, while others may perceive it as a bird. 1 Introduction A core goal of cognitive science is to establish models that reflect the relationship between external stimuli and human internal experiences. The development of ANNs has significantly contributed to this goal, particularly through the latent representations of ANNs that have shown a strong correlation with human psychological representations ( ( Wei et al., 2024a ; Wei et al., 2024b ; Muttenthaler et al., 2022a ; Mahner et al., 2024 ; Zheng et al., 2019 ; Hebart et al., 2020 ; Muttenthaler et al., 2022b ) ). Current studies focus on the phenomenon that even with identical physical stimuli, individuals may exhibit significant differences in their perceptual experiences. While such perceptual variability has been extensively documented in complex cognitive tasks (e.g., aesthetic or moral judgments), studies on simple visual decision tasks (e.g., handwritten digit classification) often neglect such inter-individual differences. As shown in Figure 1 , the stimulus on the left may be recognized as either a ”3” or a ”5”. This high perceptual variability in human perception has been inadequately explored in current research into human visual perception. Inspired by the similarity between ANNs and humans, we hypothesize that the perceptual boundaries of ANNs are also related to inter-individual variability in human perception and that images generated along these boundaries can evoke divergent perceptual experiences across individuals. Figure 2: Overview of BAM. We sample images from ANN decision boundaries using the perceptual boundary sampling algorithm for subsequent human evaluations. Our approach consists of three main components: 1. Labeling: The images generated by perceptual boundary sampling are labeled by human experiments, thus constructing the variMNIST dataset. In this process, a single image will be presented to multiple participants; 2. Aligning: Finetuning models with human behavioral data to align them with human perceptual variability at the group and individual levels, enhancing behavior prediction accuracy and aligning the models with humans; 3. Manipulating: Employing two individually aligned models, each corresponding to a specific individual, to generate images designed to elicit divergent responses between them, which are then validated through these two human participants. Recent advances in using ANNs as perceptual models have revealed novel relationships between stimuli and human experiences. ( Veerabadran et al., 2023 ; Zhou & Firestone, 2019 ; Elsayed et al., 2018 ) demonstrated that ANNs not only exhibit vulnerability to small changes but also that these changes can systematically influence human perceptual decisions in controlled environments. Similarly, ( Gaziv et al., 2024 ) leveraged robustified ANNs to identify low-norm perturbation patterns that significantly disrupt human classification behavior. Methodologically, ( Feather et al., 2023 ; Feather et al., 2019 ) proposed model metamers generation—constructing stimuli with identical ANN latent representations but distinct appearances—which revealed systematic discrepancies between model metamers and human perception from the perspective of representational invariance. Additionally, ( Golan et al., 2020 ; Golan et al., 2023 ) developed controversial stimuli that provoke clearly distinct responses among two or more models, further exposing their misalignment with human perception. Building on these frameworks, we extend the concept of controversial stimuli from model-to-model comparisons to human perception, generating stimuli that evoke perceptual divergences among human participants. We propose a systematic framework BAM ( B oundary A lignment & M anipulation framework) for studying human perceptual variability through image generation. As illustrated in Figure 2 , BAM builds upon the Perceptual Boundary Sampling (Sec. 3 ) algorithm and comprises three interconnected steps: 1. Labeling : We sample images from ANN perceptual boundaries and construct the variMNIST dataset through human behavioral experiments, systematically capturing inter-individual perceptual differences (Sec. 4 ). 2. Aligning : By fine-tuning ANN models with human behavioral data, we establish computational models of perceptual variability at both group and individual levels (Sec. 5 ). 3. Manipulating : Using individualized models as adversarial generators, we synthesize controversial stimuli that amplify perceptual differences, with experimental validation of their behavioral manipulation efficacy (Sec. 6 ). Our principal contributions are: (1) We develop an image sampling method along ANN perceptual boundaries, constructing the variMNIST dataset through human behavioral labeling. Experimental results demonstrate successful induction of high perceptual variability. (2) We achieve precise alignment between ANN and human perceptual variability using limited behavioral data, proving the computational feasibility of individual differences. (3) We design a dual-subject controversial generation framework that synthesizes stimuli inducing targeted divergent decisions. Human experiments confirm these stimuli significantly surpass random baselines in behavioral manipulation. 2 Related works Researchers have extensively used synthetic images generated by ANNs to study human perceptual space, uncovering differences between model and human perception while refining generation techniques to enhance their influence on human cognition. For instance, ( Golan et al., 2020 ; Golan et al., 2023 ) utilized controversial stimuli to highlight classification discrepancies between neural networks. Similarly, ( Veerabadran et al., 2023 ) demonstrated that adversarial perturbations could simultaneously influence ANN classifications and human perceptual choices, revealing shared sensitivities. However, ( Gaziv et al., 2024 ) found that while standard ANN perturbations fail to impact human perception, robustified ANN models can generate low-norm perturbations that significantly disrupt human percepts. Figure 3: Sampling on perceptual boundaries. (a) The sample space can be partitioned into four distinct regions based on two classification axes. Taking the digit pair (3,7) as an example, our objective is to generate samples that induce disagreement between models A and B, as illustrated in the figure. The upper region consists of stimuli classified as ”7” by model B and ”3” by model A, whereas the lower region contains stimuli classified as ”3” by model A and ”7” by model B. The left region includes stimuli that both models classify as ”3,”

Read the complete text