Generating metamers of human scene understanding
Ritik Raina, Abe Leite, Alexandros Graikos Seoyoung Ahn, Dimitris Samaras, Gregory J. Zelinsky, [0.8em] Stony Brook University · 2026
Read Generating metamers of human scene understanding by Ritik Raina, Abe Leite, Alexandros Graikos Seoyoung Ahn, Dimitris Samaras, Gregory J. Zelinsky, [0.8em] Stony Brook University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11675.txt.
Introduction
Ritik Raina and five coauthors generate metamers of human scene understanding. Their 2026 preprint presents a tool named MetamerGen. Human vision builds scenes from two kinds of data. The periphery gives low-resolution gist of the whole view.
Fixated spots add sparse but high-resolution detail. The brain merges both into a coherent scene sense. Metamers are scenes that feel alike despite physical differences. MetamerGen makes scenes aligned with latent human representations. Such scenes can probe what the visual system truly keeps. They can also fool it in controlled, useful ways. The work serves vision science and graphics alike.
Catalog record
| CREATOR | Ritik Raina, Abe Leite, Alexandros Graikos Seoyoung Ahn, Dimitris Samaras, Gregory J. Zelinsky, [0.8em] Stony Brook University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-11675
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generating metamers of human scene understanding Ritik Raina, Abe Leite, Alexandros Graikos Seoyoung Ahn, Dimitris Samaras, Gregory J. Zelinsky, [0.8em] Stony Brook University Abstract Human vision combines low-resolution “gist” information from the visual periphery with sparse but high-resolution information from fixated locations to construct a coherent understanding of a visual scene. In this paper, we introduce MetamerGen , a tool for generating scenes that are aligned with latent human scene representations. MetamerGen is a latent diffusion model that combines peripherally obtained scene gist information with information obtained from scene-viewing fixations to generate image metamers for what humans understand after viewing a scene. Generating images from both high and low resolution (i.e. “foveated”) inputs constitutes a novel image-to-image synthesis problem, which we tackle by introducing a dual-stream representation of the foveated scenes consisting of DINOv2 tokens that fuse detailed features from fixated areas with peripherally degraded features capturing scene context. To evaluate the perceptual alignment of MetamerGen generated images to latent human scene representations, we conducted a same-different behavioral experiment where participants were asked for a “same” or “different” response between the generated and the original image. With that, we identify scene generations that are indeed metamers for the latent scene representations formed by the viewers. MetamerGen is a powerful tool for understanding scene understanding. Our proof-of-concept analyses uncovered specific features at multiple levels of visual processing that contributed to human judgments. While it can generate metamers even conditioned on random fixations, we find that high-level semantic alignment most strongly predicts metamerism when the generated scenes are conditioned on viewers’ own fixated regions. 1 Introduction Understanding the latent representation of a scene formed by humans after viewing remains a fundamental unanswered challenge in cognitive science ( 13 ; 6 ; 33 ; 52 ) . What is clear is that humans represent coherent scenes by a mixture of “gist” information encoded from peripheral vision ( 39 ; 17 ) with high-resolution but sparse information that humans extract during their scene viewing fixations ( 29 ; 28 ; 11 ) . Related recent work on scene perception has focused on the concept of object and scene metamers—generated stimuli that, although physically different from originals, cannot be discriminated as different by humans when viewed under constrained experimental conditions. ( 15 ; 4 ; 44 ) . Understanding scene metamerism is important because metamers tell us the level of misalignment between an actual and generated image that is tolerated by humans and judged to be the same. Generated scenes that fail to become metamers also reveal the details that are important to a scene’s representation and that, if changed, result in the generation being detected. However, although several paradigms have been used to identify metamers (e.g., same-different tasks, A/B/X tasks, oddity judgment tasks) ( 45 ) , this work on scene perception used simple generative models to synthesize textures and shapes that were shown in behavioral experiments to be metameric with what humans perceive in their visual periphery when the eye position is fixed. These paradigms, however, were not designed to study how post-gist changes in fixation affect scene metamerism or what objects a person believes to exist in their blurred peripheral view of a scene, which are the problems we engage. Inspired by these previous studies showing that generated textures and shapes can become metamers for human scene perception , we introduce MetamerGen , a state-of-the-art generative model that extends the metamer generation approach to human scene understanding . Rather than seeking to generate simple patterns that share low-level statistics with peripheral vision, MetamerGen better captures a post-gist level of representation reflecting multiple free-viewing fixations. We see this topic as closer to scene understanding because we are seeking to generate a hypothesis for what a person believes to be in their peripheral vision, and henceforth we will use the term scene metamer to refer to two scenes that have an equivalent understanding. Our approach combines a gist-level scene representation extracted from peripherally blurred pixels with higher-resolution and fixation-specific “foveal” representations corresponding to scene-viewing fixations. Scene gist and the objects fixated during viewing are therefore used to generate in the non-fixated blurred pixels a scene context that is aligned with what a human understands to be in their peripheral vision. We not only show that many of the scenes generated by MetamerGen are metamers for human scene understanding, we also model the dynamic evolution of this understanding by leveraging the capability of a latent diffusion model ( 42 ) to generate photorealistic images from diverse conditioning signals ( 46 ; 62 ; 41 ) . Because MetamerGen is a latent diffusion model (Stable Diffusion; 42 ), we can use each viewing fixation as a conditioning signal to obtain an incremental fixation-by-fixation understanding of a scene (Figure 1 ). To adapt the Stable Diffusion model to our task of generating a scene in blurred peripheral pixels, we introduce a dual-stream representation of foveated scenes (i.e., ones with a high-resolution center and blurred periphery) using a self-supervised image encoder (DINOv2) ( 8 ; 38 ; 10 ) . We utilize an adapter-based framework ( 35 ) , where we condition a pre-trained text-to-image diffusion model on fixation-grounded features extracted by DINOv2 feature representations obtained at each of the fixation locations. We complement the fixation representations with peripheral information, adding a second source of conditioning that uses DINOv2 tokens extracted from a blurred-out version of the same image, capturing the context. Our conditioning mechanism allows us to generate plausible scene hypotheses from a variable information input, where more foveal glimpses of a scene will lead to a richer DINOv2 representation that enables MetamerGen to generate increasingly plausible and contextually appropriate content at the non-fixated scene locations, analogous to how human scene understanding becomes more elaborate with more viewing fixations. We see MetamerGen as a tool for generating fixation-specific scene understanding hypotheses that cognitive scientists can test in behavioral studies. We integrated MetamerGen into a same-different behavioral paradigm and conducted experiments to identify the generated scenes that are metamers for human scene understanding. In our paradigm, participants viewed a scene for a variable number of fixations (i.e., gaze contingent), followed by a 5-second delay (during which MetamerGen generated a scene from the viewing behavior) and then briefly viewed a second scene (200 msec). Their task was to judge whether this second scene was the same or different from the first. We define a scene metamer as a generation that a participant judges to be the same as the real scene that was first viewed. Our post-hoc analysis showed that while all features throughout the visual hierarchy contributed to the understanding of a scene, high-level semantic features emerged as the strongest predictors of scene understanding metamers. 2 Preliminaries 2.1 Image generation using latent generative models Diffusion models ( 46 ; 20 ) comprise two opposing processes—a diffusion process that gradually corrupts data and a denoising process that restores information. The diffusion process relies on Gaussian noise of increasing intensity at every step, while the denoising process uses a learned denoiser model to reverse the degradation. By iterating this process, starting from random Gaussian noise, diffusion models generate new samples. Latent diffusion models (LDMs) ( 42 ) reduce the overall