One-Shot Real-to-Sim via End-to-End Differentiable Simulation and Rendering

Yifan Zhu, Tianyi Xiang, Aaron M. Dollar, Zherong Pan · 2024

Read One-Shot Real-to-Sim via End-to-End Differentiable Simulation and Rendering by Yifan Zhu, Tianyi Xiang, Aaron M. Dollar, Zherong Pan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00259.txt.

Introduction

Yifan Zhu and co-authors present, in a 2024 paper, a way for a robot to build a full world model from a single run of action. Robots need models of new rooms and objects to plan tasks, yet past methods could not tune shape, look, and physics all at once. Their method learns all three from sparse sight and touch data.

A point-based form stands in for each rigid object, paired with a grid that holds how it looks. That pairing lets the system render views and detect collisions in a smooth, tunable way. A physics engine with the same smooth traits then tunes the whole model end to end. Tests in fake and real settings built ready-to-use models from one action sequence alone.

Catalog record

CREATORYifan Zhu, Tianyi Xiang, Aaron M. Dollar, Zherong Pan
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-00259

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

One-Shot Real-to-Sim via End-to-End Differentiable Simulation and Rendering Yifan Zhu, Tianyi Xiang, Aaron M. Dollar, Zherong Pan Abstract Identifying predictive world models for robots from sparse online observations is essential for robot task planning and execution in novel environments. However, existing methods that leverage differentiable programming to identify world models are incapable of jointly optimizing the geometry, appearance, and physical properties of the scene. In this work, we introduce a novel rigid object representation that allows the joint identification of these properties. Our method employs a novel differentiable point-based geometry representation coupled with a grid-based appearance field, which allows differentiable object collision detection and rendering. Combined with a differentiable physical simulator, we achieve end-to-end optimization of world models or rigid objects, given the sparse visual and tactile observations of a physical motion sequence. Through a series of world model identification tasks in simulated and real environments, we show that our method can learn both simulation- and rendering-ready rigid world models from only one robot action sequence. The code and additional videos are available at our project website: https://tianyi20.github.io/rigid-world-model.github.io/ . I Introduction An accurate internal model of a robot about how its actions can affect the surrounding environment is essential for robot planning and control. Such a model, which we refer to as a world model , needs to render realistic raw observations such as RGB images from arbitrary viewpoints and predict consistent and accurate physical interactions. However, constructing such a model from raw observations in novel real-world environments remains challenging as it requires the identification of the geometry parameters that describe the shape of all objects (e.g. vertices and faces of a mesh), appearance parameters that define how the objects look when rendered (e.g. color and reflectance), and physical parameters (e.g. mass) of the objects in the scene. These parameters are usually partially observable, and robots are typically limited in time and computational resources. Recently, there has been growing interest in learning world models from large offline datasets of action-labeled videos using generative modeling techniques [ 1 , 2 , 3 , 4 ] . However, these black-box models are susceptible to distribution shifts and cannot infer properties such as the coefficient of friction. In addition, they are not physically consistent and cannot provide physical information such as contact forces, which are essential for downstream tasks. Meanwhile, an alternative approach that identifies the geometry, appearance, and physical parameters (GAP) of the environment with strong priors coming from knowledge of physics can result in a generalizable and physically consistent world model. Many existing works employ differentiable simulators as strong physics priors that allow efficient identification of physical parameters such as inertia and coefficient of friction [ 5 , 6 , 7 ] . Differentiable simulators allow the gradient-based optimization of mass-inertial properties and frictional coefficients to match a physical motion sequence to sparse robot observations. However, these works assume known geometries and appearances of the objects in the scene and do not allow the algorithm to adapt the GAP simultaneously. On the other hand, we have witnessed recent advances in learnable geometry and appearance models, such as Neural Radiance Fields (NeRF) [ 8 ] and Gaussian Splatting (GS) [ 9 ] . These methods build the rendering equation into a learnable representation to enable the identification of geometries and appearances from raw observations. However, rigid body simulators [ 10 ] typically require the use of volumetric representations with a clear definition of object surfaces such as convex hulls to detect collisions and penetration depths. Unfortunately, NeRF and GS are incompatible with the requirements of rigid body simulators since NeRF represents objects with a continuous neural field and GS with individual 3D Gaussians. To the best of the authors’ knowledge, no existing method allows the simultaneous identification of the GAP properties of a world model of rigid objects from sparse robot observations. Fig. 1: From the visual and tactile observations of a single robot push (top), our method jointly optimizes the shape, appearance, and physical parameters of a world model consisting of rigid objects in the form of a rigid body simulator (bottom, the robot arm is not rendered in this picture and the end-effector is treated as a floating blue sphere robot). To address these challenges, this work presents a rigid object representation that is compatible with general-purpose rigid body simulators and allows the joint optimization of GAP. As shown in Fig. 1 , based on this representation, our work enables the identification of a rigid world model in the form of a full-fledged rigid body simulator from the observations of one robot push. Our proposed representation is the combination of a recently proposed point-based shape representation Shape-as-Points (SaP) [ 11 ] and a grid-based appearance field. SaP parameterizes an object’s geometry and topology using a set of surface points along with normal directions. It then uses differentiable Poisson reconstruction to recover a smooth indicator field of object occupancy, which can be converted to a mesh using a differentiable marching cubes algorithm [ 12 ] . The texture of the vertices of the mesh is then obtained by interpolating the appearance grid. Employing the mesh in a differentiable rigid body simulator [ 13 ] that provides gradients for the physical parameters and contact points of the objects, our method constructs a fully differentiable pipeline for jointly optimizing the GAP. Our contributions are: • A jointly differentiable representation of the shape, appearance, and physical properties of rigid objects. • An algorithm for identifying world models online from sparse robot observations, which we refer to as real-to-sim, with an end-to-end differentiable simulation and rendering pipeline. We evaluate our method on identification problems in both simulated and real-world environments. The results show that our method can infer accurate world models from a single episode of robot interactions with the environment. II Related Work Our work is closely related to differentiable rigid body simulators, learnable geometry and appearance models, and identifying world models, and we review these areas of study in this section. II-A Differentiable Rigid Body Simulator Rigid body simulators are essential tools in robotics and engineering for testing, verification, perception, control, and planning. Traditional rigid body simulators are not differentiable, but there have been many recently proposed differentiable rigid body simulators [ 13 , 14 , 15 , 16 , 17 , 18 ] for facilitating downstream system identification, robot planning, and policy optimization tasks. Different strategies are adopted to enable the calculation of gradients for the underlying non-differentiable contact dynamics, including employing a smooth contact model [ 18 , 15 , 16 ] , using sub-gradients of the linear complementarity problem [ 14 ] , and implicit gradients of nonlinear optimization [ 13 , 17 ] . However, most of these methods do not provide gradients with respect to the geometry, with the exception of [ 17 , 13 , 15 ] . In this work, we adopt the simulator proposed by Strecke et al. [ 13 ] for its physical realism, numerical stability, and fast computation from GPU acceleration. II-B Learnable Geometry and Appearance Models Learning 3D geometry and appearance models from 2D raw images is vital to robots’ understanding of the physical world. Earlier research has focused on learning only

Read the complete text