SOPHY : Learning to Generate S imulation-Ready O bjects with PHY sical Materials

Junyi Cao, Evangelos Kalogerakis · 2025

Read SOPHY : Learning to Generate S imulation-Ready O bjects with PHY sical Materials by Junyi Cao, Evangelos Kalogerakis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12684.txt.

Introduction

Junyi Cao and Evangelos Kalogerakis presented SOPHY in 2025, a content-making model that creates 3D objects ready to behave physically in a model. Earlier makers made static shapes with no physics, or animations that ignored real substance actions. An object that should bounce, bend, or topple needs not just a shape but the right substance traits attached to it.

SOPHY synthesizes shape, surface texture, and actual substance traits, so its outputs can go straight into dynamic, interactive settings. To train it, the team built a data set of objects labeled with detailed actual substance traits, plus a lean workflow for adding those labels. The system supports making interactive objects from a text account and rebuilding a plausible object from a single image, and joint modeling boosted both shape realism and actual plausibility.

Catalog record

CREATORJunyi Cao, Evangelos Kalogerakis
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-12684

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SOPHY : Learning to Generate S imulation-Ready O bjects with PHY sical Materials Junyi Cao, Evangelos Kalogerakis Abstract We present SOPHY, a generative model for 3D physics-aware shape synthesis. Unlike existing 3D generative models that focus solely on static geometry or 4D models that produce physics-agnostic animations, our method jointly synthesizes shape, texture, and material properties related to physics-grounded dynamics, making the generated objects ready for simulations and interactive, dynamic environments. To train our model, we introduce a dataset of 3D objects annotated with detailed physical material attributes, along with an efficient pipeline for material annotation. Our method enables applications such as text-driven generation of interactive, physics-aware 3D objects and single-image reconstruction of physically plausible shapes. Furthermore, our experiments show that jointly modeling shape and material properties enhances the realism and fidelity of the generated shapes, improving performance on both generative geometry and physical plausibility. Our project page is available at: https://xjay18.github.io/SOPHY_page . Figure 1: SOPHY is a physics-aware generative model designed for creating 3D objects that possess plausible geometries, textures, and physical properties, making them ready for simulation. ( Top ) Our method can generate these assets from a single image; ( middle ) or from a text prompt (conditions are shown as insets in the top right corner of each generated object). ( Bottom ) We further showcase SOPHY’s potential for constructing a physically accurate virtual world by combining various image and text conditions to generate diverse objects. 1 Introduction Generating high-quality 3D assets that can be incorporated in interactive virtual worlds or manufactured into functional 3D objects is a fundamental challenge in generative AI and digital content creation. Unfortunately, current generative models of 3D shapes [ 13 , 95 , 66 ] are limited to generating only static geometry and textures. Although there is significant effort in building 4D generative models for both geometry and motion synthesis [ 84 , 72 , 40 ] , current methods are limited to fixed animations and are incapable of producing motions resulting from complex, dynamic object interactions. In this work, we introduce a diffusion-based generative model for physics-aware 3D object synthesis, capable of generating 3D assets with detailed shape, texture, and, most importantly, physical material attributes governing kinematic object deformations ( e.g . , elastic deformations, plastic softening) and frictional interactions. Our approach produces physics-aware, interactive 3D objects, which are helpful for downstream applications such as physically based simulation, robotic interaction, and manufacturing. There are various challenges to developing such a generative approach. First, current 3D and 4D datasets do not contain physical material attributes for objects, making it hard to train, or even fine-tune such physics-aware generative models. Second, it is unclear how to jointly model the interplay of shape and material in 3D asset generation. Clearly, material attributes are not independent of geometric shapes, as repeatedly observed in prior works of physical material recognition from shapes [ 39 , 42 , 68 , 1 ] . For example, a chair’s backrest made of thin rods or strips is usually metal, while a thicker, block-like backrest is often made of fabric or soft padding. Our work aims to address the above challenges. First, we introduce a semi-automatic annotation pipeline, combining Vision Language Model (VLM)’s guidance with iterative expert (mechanical engineer) feedback for more efficient object annotation with material properties. Second, we introduce a generative model that builds upon recent advancements in latent diffusion models, yet incorporates novel insights on representing 3D objects as compact latent codes capturing shape geometry, albedo color, and material properties related to elasticity and plasticity, as well as capturing their interdependence through cross-attention blocks. Additionally, we integrate a texture enhancement module as a post-processing step that refines the generated albedo color, resulting in a high-quality texture map for improved appearance. Overall, we observe that modeling shape and material properties together enhances the realism and fidelity of the geometry in the generated shapes. Our contributions are: • We present a semi-automatic pipeline to obtain annotations describing object physical behavior, and a new dataset of 3K objects and 15K parts annotated with detailed physical material properties. • We introduce an autoencoder for representing physics-aware 3D objects into compact latents, along with a diffusion-based model that jointly synthesizes geometry, texture, and physical materials. • We showcase applications of our framework for synthesizing simulation-ready objects based on texts or images, as well as physics-aware virtual worlds filled with objects generated by our method. 2 Related Work 3D asset generation. Over the recent years, we have witnessed an explosion of 3D generative models capable of generating detailed geometry and texture. The advances have been driven by expressive neural representations, such as NeRFs [ 52 , 59 ] , Gaussian splats [ 29 ] , signed distance fields [ 14 , 57 ] , occupancy fields [ 51 ] , and numerous other representations for capturing shape or/and texture [ 54 , 64 , 23 , 41 , 88 , 10 , 11 , 27 , 91 , 58 , 20 , 22 , 87 , 2 ] to name a few. We also refer readers to the recent surveys of 3D generative model [ 36 , 81 ] . Unfortunately, the vast majority of existing 3D generative models neglect the physical material properties of the generated 3D objects, limiting their applicability to interactive and simulation-based environments. To address this gap, recent research has begun incorporating physical priors and constraints into the generative pipeline. For example, DiffuseBot [ 80 ] evaluates generated 3D robot models based on their simulation performance and refines the sampling distribution to favor more successful designs. Atlas3D [ 12 ] and Phys-Comp [ 21 ] enforce static equilibrium constraints during shape optimization, ensuring that generated objects remain stable under gravity. Despite these advances, all these prior works assume uniform or limited material properties, restricting their ability to generate diverse and physically intricate 3D assets. In contrast, our approach jointly optimizes 3D shape, albedo color, and material properties with a network that captures their natural dependencies. Material-annotated 3D datasets. The advances in 3D generative models have also been led by the development of influential 3D datasets such as ShapeNet [ 7 ] and Objaverse [ 16 ] . More recent datasets, including ABO [ 15 ] , Matsynth [ 77 ] , and BlenderVault [ 44 ] , incorporate surface texture information to train models for 3D generation and more plausible, photorealistic rendering. Despite this progress, a critical gap remains in the availability of detailed physical material information essential for accurately modeling material properties for physics-based simulations. While datasets such as ShapeNet-Mat [ 42 ] , 3DCoMPaT [ 39 ] and its extensions [ 68 , 1 ] attempt to address this limitation by providing part-level material labels, their annotations are coarse and unsuitable for direct use in physical simulators. To bridge this gap, we introduce a new dataset consisting of 3K objects spanning 12 shape categories, each annotated with precise part-level physical material properties. 4D content generation. 4D generation aims to create dynamic 3D content that aligns with input conditions such as images, text, or videos. Existing approaches primarily follow a data-driven pipeline [ 34 , 82 , 84 , 40 , 35 , 71 ] ,

Read the complete text