Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixels

Long Le Ryan Lucas Chen Wang Chuhao Chen, [2pt] Dinesh Jayaraman Eric Eaton Lingjie Liu, [2pt] University of Pennsylvania · 2025

Read Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixels by Long Le Ryan Lucas Chen Wang Chuhao Chen, [2pt] Dinesh Jayaraman Eric Eaton Lingjie Liu, [2pt] University of Pennsylvania (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17437.txt.

Introduction

Long Le and six co-authors teach a network to read the physical traits of a three-dimensional scene from images, in a paper from twenty twenty-five. Knowing stiffness or stretch from sight is key to building virtual worlds that react in a believable way. People do this at a glance, but past methods re-solved each scene slowly from scratch.

Their system learns once, from many scenes, and then infers material maps for new scenes in a single pass. Paired with a learned scene model, it can run live physics under outside forces. The team also built a large data set of assets with material labels. Tests show results several times better than slow per-scene methods, at a fraction of the time, and transfer to real scenes.

Catalog record

CREATORLong Le Ryan Lucas Chen Wang Chuhao Chen, [2pt] Dinesh Jayaraman Eric Eaton Lingjie Liu, [2pt] University of Pennsylvania
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17437

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixels Long Le Ryan Lucas Chen Wang Chuhao Chen, [2pt] Dinesh Jayaraman Eric Eaton Lingjie Liu, [2pt] University of Pennsylvania Abstract Inferring the physical properties of 3D scenes from visual information is a critical yet challenging task for creating interactive and realistic virtual worlds. While humans intuitively grasp material characteristics such as elasticity or stiffness, existing methods often rely on slow, per-scene optimization, limiting their generalizability and application. To address this problem, we introduce Pixie , a novel method that trains a generalizable neural network to predict physical properties across multiple scenes from 3D visual features purely using supervised losses. Once trained, our feed-forward network can perform fast inference of plausible material fields, which coupled with a learned static scene representation like Gaussian Splatting enables realistic physics simulation under external forces. To facilitate this research, we also collected PixieVerse , one of the largest known datasets of paired 3D assets and physic material annotations. Extensive evaluations demonstrate that Pixie is about 1.46-4.39x better and orders of magnitude faster than test-time optimization methods. By leveraging pretrained visual features like CLIP, our method can also zero-shot generalize to real-world scenes despite only ever been trained on synthetic data. https://pixie-3d.github.io/ Figure 1 : We introduce Pixie , a novel method for learning simulatable physics of 3D scenes from visual features. Trained on a curated dataset of paired 3D objects and physical material annotations, Pixie can predict both the discrete material types (e.g., rubber) and continuous values including Young’s modulus, Poisson’s ratio, and density for a variety of materials, including elastic, plastic, and granular. The predicted material parameters can then be coupled with a learned static 3D model such as Gaussian splats and a physics solver such as the Material Point Method (MPM) to produce realistic 3D simulation under physical forces such as gravity and wind. 1 Introduction Advances in learning-based scene reconstruction with Neural Radiance Fields [ 28 ] and Gaussian Splatting [ 18 ] have made it possible to recreate photorealistic 3D geometry and appearance from sparse camera views, with broad applications from immersive content creation to robotics and simulation. However, these approaches focus exclusively on visual appearance—capturing the geometry and colors of a scene while remaining blind to its underlying physical properties. Yet the world is not merely a static collection of shapes and textures. Objects bend, fold, bounce, and deform according to their material composition and the forces acting upon them. Consequently, there has been a growing body of work that aims to integrate physics into 3D scene modeling [ 31 , 27 , 23 , 11 , 10 , 41 , 32 , 12 , 26 , 42 , 6 ] . Current approaches for acquiring the material properties of the scene generally fall into two categories, each with significant limitations. Some works such as [ 41 , 12 ] require users to manually specify material parameters for the entire scene based on domain knowledge. This manual approach is limited in its application as it places a heavy burden on the user and lacks fine-grained detail. Another line of work aims to automate the material discovery process via test-time optimization. Works including [ 15 , 23 , 45 , 14 , 26 , 44 ] leverage differentiable physics solvers, iteratively optimizing material fields by comparing simulated outcomes against ground-truth observations or realism scores from video generative models. However, predicting physical parameters for hundreds of thousands of particles from sparse signals (i.e., a single rendering or distillation scalar loss) is an extremely slow and difficult optimization process, often taking hours on a single scene. Furthermore, this heavy per-scene memorization does not generalize: for each new scene, the incredibly slow optimization has to be run from scratch again. In this paper, we propose a new framework, Pixie , which unifies geometry, appearance, and physics learning via direct supervised learning. Our approach is inspired by how humans intuitively understand physics: when we see a tree swaying in the wind, we do not memorize the stiffness values for each specific coordinate (x,y,z) – instead, we learn that objects with tree-like visual features behave in certain ways when forces are applied. This physical understanding from visual cues allows us to anticipate the motion of a different tree or even other vegetation like grass, in an entirely new context. Thus, our insight is to leverage rich 3D visual features such as those distilled from CLIP [ 33 ] to predict physical materials in a direct supervised and feed-forward way. Once trained, our model can associate visual patterns (e.g., "if it looks like vegetation") with physical behaviors (e.g., "it should have material properties similar to a tree"), enabling fast inference and generalization across scenes. To facilitate this research, we have curated and labeled PixieVerse , a dataset of 1624 paired 3D objects and annotated materials spanning 10 semantic classes. We developed a sophisticated multi-step and semi-automatic data labeling process, distilling pretrained models including Gemini [ 36 ] , CLIP [ 33 ] , and human prior into the dataset. To our knowledge, this is the largest open-source dataset of paired 3D assets and physical material labels. Trained on PixieVerse , our feed-forward network can predict material fields that are 1.46-4.39x better and orders of magnitude faster than test-time optimization methods. By leveraging pretrained visual features, Pixie can also zero-shot generalize to real-world scenes despite only ever being trained on synthetic data. Our contributions include: 1. Novel Framework for 3D Physics Prediction : We introduce Pixie , a unified framework that predicts discrete material types and continuous physical parameters (Young’s modulus, Poisson’s ratio, density) directly from visual features using supervised learning. 2. PixieVerse Dataset : We curate and release PixieVerse , the largest open-source dataset of 3D objects with physical material annotations (1624 objects, 10 semantic classes). 3. Fast and Generalizable Inference : By leveraging pretrained visual features from CLIP and a feed-forward 3D U-Net, Pixie performs inference orders of magnitude faster than prior test-time optimization approaches, achieving a 1.46-4.39x improvement in realism scores as evaluated by a state-of-the-art vision-language model. 4. Zero-Shot Generalization to Real Scenes : Despite being trained solely on synthetic data, Pixie generalizes to real-world scenes, showing how visual feature distillation can effectively bridge the sim-to-real gap. 5. Seamless Integration with MPM Solvers : The predicted material fields can be directly coupled with Gaussian splatting models for realistic physics simulations under applied forces such as wind and gravity, enabling interactive and visually plausible 3D scene animations. 2 Related Work 2D World Models Some early works [ 3 , 2 ] learn to predict material labels on 2D images. Recently, learning forward dynamics from 2D video frames has also been explored extensively. For instance, Google’s Genie [ 30 ] trains a next-frame prediction model conditioned on latent actions derived from user inputs, capturing intuitive 2D physics in an unsupervised manner. While these methods achieve impressive 2D generation and control, they do not explicitly model 3D geometry or a physically grounded world. Other works such as [ 7 , 24 ] also explore generating or editing images based on learned real-world dynamics. While these methods achieve impressive results in 2D visual synthesis and can imply

Read the complete text