GenMatter: Perceiving Physical Objects with Generative Matter Models

Eric Li, Arijit Dasgupta, Yoni Friedman, Mathieu Huot, [0.3em] Vikash Mansinghka, Thomas O’Connell, William T. Freeman, Joshua B. Tenenbaum, [0.5em] MIT CSAIL, [0.3em] Corresponding author: [email protected] Project page: esli999.github.io/genmatter · 2026

Read GenMatter: Perceiving Physical Objects with Generative Matter Models by Eric Li, Arijit Dasgupta, Yoni Friedman, Mathieu Huot, [0.3em] Vikash Mansinghka, Thomas O’Connell, William T. Freeman, Joshua B. Tenenbaum, [0.5em] MIT CSAIL, [0.3em] Corresponding author: [email protected] Project page: esli999.github.io/genmatter (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22160.txt.

Introduction

Li and seven colleagues propose a generative model for perceiving physical objects, inspired by human vision. The model groups low-level motion cues and high-level looks into particles, small Gaussians standing for local matter, then groups particles into clusters holding together-moving things. A fast inference method based on parallel block Gibbs sampling recovers stable particle motion and groupings. The model works on random dots, stylized textures, and lifelike RGB video, matching human vision where current computer vision fails.

Catalog record

CREATOREric Li, Arijit Dasgupta, Yoni Friedman, Mathieu Huot, [0.3em] Vikash Mansinghka, Thomas O’Connell, William T. Freeman, Joshua B. Tenenbaum, [0.5em] MIT CSAIL, [0.3em] Corresponding author: [email protected] Project page: esli999.github.io/genmatter
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91029
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.22160

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GenMatter: Perceiving Physical Objects with Generative Matter Models Eric Li, Arijit Dasgupta, Yoni Friedman, Mathieu Huot, [0.3em] Vikash Mansinghka, Thomas O’Connell, William T. Freeman, Joshua B. Tenenbaum, [0.5em] MIT CSAIL, [0.3em] Corresponding author: [email protected] Project page: esli999.github.io/genmatter Abstract Human visual perception offers valuable insights for understanding computational principles of motion-based scene interpretation. Humans robustly detect and segment moving entities that constitute independently moveable chunks of matter, whether observing sparse moving dots, textured surfaces, or naturalistic scenes. In contrast, existing computer vision systems lack a unified approach that works across these diverse settings. Inspired by principles of human perception, we propose a generative model that hierarchically groups low-level motion cues and high-level appearance features into particles (small Gaussians representing local matter), and groups particles into clusters capturing coherently and independently moveable physical entities. We develop a hardware-accelerated inference algorithm based on parallelized block Gibbs sampling to recover stable particle motion and groupings. Our model operates on different kinds of inputs (random dots, stylized textures, or naturalistic RGB video), enabling it to work across settings where biological vision succeeds but existing computer vision approaches do not. We validate this unified framework across three domains: on 2D random dot kinematograms , our approach captures human object perception including graded uncertainty across ambiguous conditions; on a Gestalt-inspired dataset of camouflaged rotating objects, our approach recovers correct 3D structure from motion and thereby accurate 2D object segmentation; and on naturalistic RGB videos, our model tracks the moving 3D matter that makes up deforming objects, enabling robust object-level scene understanding. This work thus establishes a general framework for motion-based perception grounded in principles of human vision. Figure 1 : GenMatter is a generative model of moving matter. Conditioned on motion and appearance features extracted from RGB video, inference inverts this hierarchical generative model to group observations into particles (small Gaussians representing local regions of matter), themselves grouped into clusters (coherently and independently moveable physical entities). A hardware-accelerated inference algorithm based on parallelized block Gibbs sampling recovers stable particle motion and groupings. (a) RGB video input. (b) Inferred 3D matter particles shown as colored ellipses, each colored by the average color of its assigned data points. (c) The same particles colored by cluster assignment, revealing independently moving objects. 1 Introduction Human vision segments moving objects across diverse settings: random dot kinematograms with minimal form cues 48 , camouflaged textured objects perceivable only through motion 21 , and naturalistic scenes 42 . No existing computer vision system has this generality. This gap raises a scientific question about the computational principles that enable such broad perceptual capabilities. We introduce GenMatter , a generative model that segments moving matter across the diverse settings where biological vision succeeds. The model hierarchically groups low-level motion and appearance features into particles (local regions of matter represented as Gaussians), then groups particles into clusters representing coherently and independently moveable entities. The same Bayesian inference algorithm operates across diverse inputs by performing motion feature extraction, which is effective even in abstract textured scenes, and can be supplemented with shape and appearance features for naturalistic video. By jointly inferring particle trajectories and their organization into Spelke objects 51 , discovering cluster assignments and cluster-level rigid transformations, our approach identifies which particles belong to which moving entities while inducing soft rigidity priors within each entity. Unlike prior methods that impose hand-crafted regularization constraints and avoid explicit object-level grouping, GenMatter captures entities undergoing significant deformation. Figure 2 illustrates the pipeline from RGB frames to inferred particles and clusters. Figure 2 : GenMatter inference pipeline. RGB video is preprocessed (gray arrows) to extract dense depth and optical flow, lifting each pixel to a 3D point tagged with its velocity. The blue box depicts the GenMatter generative model (Sec. 3 ), which represents a scene as a hierarchy of clusters and particles that emit moving 3D points. Black arrows indicate the generative direction (clusters generate particles, which generate 3D points). Hollow arrows indicate the inference algorithm (Sec. 4 ), which conditions on observed 3D points to infer particle and cluster parameters. Red arrows depict motion at the point, particle, and cluster layers. We evaluate GenMatter across three diverse settings where biological vision robustly detects and groups independently moving matter, yet no single computer vision system succeeds across all three: • In random dot kinematogram (RDK) stimuli, where dots follow rigid motion or flicker randomly and correspondences are ambiguous, we test whether motion patterns alone suffice to infer object identity. GenMatter reproduces human perceptual groupings across all difficulty levels and captures graded uncertainty that aligns with intersubject variability. • We evaluate GenMatter on a new dataset of rotating objects with camouflaged textures inspired by Gestalt grouping principles. Human observers readily perceive 3D structure from motion in these stimuli. GenMatter recovers correct 3D structure, with groupings of independently moving matter emerging from probabilistic inference. • On naturalistic RGB videos, GenMatter maintains accurate tracking of the moving 3D matter that makes up deforming objects, matching the performance of supervised trackers without task-specific training. Our evaluation strategy has two complementary objectives. First, we compare GenMatter against standard computer vision approaches across all three settings, demonstrating that both biological vision and GenMatter possess generality that existing computer vision systems lack. Second, within each setting, we provide baselines and ablations to demonstrate specific technical advantages of our structured probabilistic approach: capturing graded perceptual uncertainty, integrating noisy motion signals over time, and generalizing without task-specific training (where supervised baselines may match our performance but lack our cross-domain generality). We emphasize that GenMatter provides a unifying framework that describes a class of models for motion-based perception. While the class of models defined by our approach requires pretrained feature extractors, the core inference engine requires no task or domain-specific training and fits within a few kilobytes of source code. By recasting perceptual grouping as online probabilistic inference, the structured prior defined by GenMatter’s generative model achieves performance comparable to data-driven learning across all studied perception tasks, spanning abstract dot stimuli and naturalistic videos. 2 Related Work Analysis-by-synthesis The analysis-by-synthesis approach posits perception as inference in a structured generative model. Purely model-based systems using probabilistic graphics programs have achieved success in CAPTCHA-breaking, pose estimation, and scene understanding 40 ; 35 ; 24 ; 25 , while hybrid approaches combining learned components with structured priors have improved accuracy and efficiency 20 ; 47 ; 66 ; 55 . Programming systems supporting inference automation have enabled competitive performance for object detection and tracking 36 ; 8 . Generative models of human perception Perception as Bayesian inference in structured generative models captures key aspects of human visual processing, including motion perception, shape recognition, and facial identification 64 ; 26 ; 10 ; 19 . This framework extends to physical scene understanding, modeling how humans infer object properties and interactions 61 ; 6 ; 65 ; 56 ; 59 ; 50 . However, computational complexity forces such models to impose restrictive assumptions on texture and geometry. Recent work shows that biologically-inspired motion energy features improve motion segmentation robustness and enable zero-shot generalization to random dot stimuli 53 ; 52 . GenMatter provides a tractable probabilistic model achieving human-like robustness to noise and ambiguity while operating on naturalistic visual input. Tracking any point The problem

Read the complete text