Object-centric Denoising Diffusion Models for Physical Reasoning
Moritz Lange, Raphael C. Engelhardt, Wolfgang Konen, Andrew Melnik, Laurenz Wiskott · 2025
Read Object-centric Denoising Diffusion Models for Physical Reasoning by Moritz Lange, Raphael C. Engelhardt, Wolfgang Konen, Andrew Melnik, Laurenz Wiskott (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04920.txt.
Introduction
Moritz Lange and co-authors build a diffusion model that reasons about moving objects in a 2025 paper. Predicting how several objects will roll, collide, and settle is core to physical reasoning. Standard models roll forward step by step, so they can only start from a beginning state, never aim at a required ending.
Their model treats the whole trajectory at once and works object by object. It stays consistent if the scene shifts in time or the objects swap order, and it can be pinned to conditions at any time step for any object. Tests vary the object count and path length mid-run. The design suits planning tasks where goals sit in the future.
Catalog record
| CREATOR | Moritz Lange, Raphael C. Engelhardt, Wolfgang Konen, Andrew Melnik, Laurenz Wiskott |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-04920
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Object-centric Denoising Diffusion Models for Physical Reasoning Moritz Lange, Raphael C. Engelhardt, Wolfgang Konen, Andrew Melnik, Laurenz Wiskott Abstract Reasoning about the trajectories of multiple, interacting objects is integral to physical reasoning tasks in machine learning. This involves conditions imposed on the objects at different time steps, for instance initial states or desired goal states. Existing approaches in physical reasoning generally rely on autoregressive modeling, which can only be conditioned on initial states, but not on later states. In fields such as planning for reinforcement learning, similar challenges are being addressed with denoising diffusion models. In this work, we propose an object-centric denoising diffusion model architecture for physical reasoning that is translation equivariant over time, permutation equivariant over objects, and can be conditioned on arbitrary time steps for arbitrary objects. We demonstrate how this model can solve tasks with multiple conditions and examine its performance when changing object numbers and trajectory lengths during inference. 1 Introduction Physical reasoning, in machine learning, often requires the prediction of object trajectories. In many scenarios, it is useful to imagine trajectories based on multiple conditions: In reinforcement learning, an agent might want to move objects to achieve a certain goal. In a counterfactual question scenario, a model might seek to explain what would have happened if certain conditions had been different. Such scenarios usually involve multiple objects that interact with each other, for instance through collisions or friction. To foster development on these physical reasoning challenges, a number of interactive or counterfactual benchmarks have been developed ( Melnik et al. 2023 ) . The conditions imposed on objects often affect various time steps in trajectories. Some objects might have a fixed initial state, while there are desired goal states for other objects. At the same time, the system retains certain degrees of freedom, which can be controlled by a model or agent. Figure 1: Left: Provided conditions fix initial and final states of green and blue ball, no conditions on the red ball. Right: The denoising process. The model is able to place the red ball so that conditions are fulfilled. Object positions (centers) are clipped to image boundaries for display purposes. Such an every-day physical setup, however, still presents unique challenges: There are many potential object interactions, and conditioning on various time steps prevents the use of regular autoregressive modeling approaches such as conventional differential equations or autoregressive generative models. Nevertheless, existing reasoning approaches ( Girdhar et al. 2020 ; Wu et al. 2023 ; Daniel and Tamar 2023 ) often rely on autoregressive forward models. These are conditioned on initial states only and require a trial and error approach: Values for the degrees of freedom are sampled and the corresponding trajectories modeled until desired goal states are reached. Building on denoising diffusion models for trajectory prediction of agents ( Janner et al. 2022 ; Ye et al. 2024 ) , we propose an object-centric denoising diffusion model for physical reasoning. It generates all time steps of the trajectories of multiple objects simultaneously, and can thus consider arbitrary conditions for different objects and time steps. Unlike existing denoising diffusion models – which lack an object-centric design – our model is translation equivariant over time and permutation equivariant over objects. This also grants it flexibility regarding trajectory lengths and object numbers during inference (within the limits of the training data distribution). We additionally employ two new methods to infuse and enforce conditions, so that our model can, in principle, solve physical reasoning scenarios with arbitrary conditions and degrees of freedom. Figure 1 illustrates an example scenario. Overall, our contributions are: • We propose a novel object-centric denoising diffusion model architecture for physical reasoning, which models temporal evolution and interactions on an object-level. • We propose and combine two novel ways of conditioning the denoising diffusion model: A soft conditioning approach that provides conditions to the model through masked interpolation, and a hard conditioning approach that shifts generated trajectories to enforce conditions. Our implementation is publicly available on GitHub 1 1 1 https://github.com/wiskott-lab/diffusion4phyre . 2 Background The following gives an overview and brief introduction into relevant concepts of physical reasoning and denoising diffusion. 2.1 Physical Reasoning Physical reasoning models make predictive, descriptive, counterfactual or explanatory statements, or perform planning, in a physical environment. State of the art in literature are modular setups that first process inputs (often visual), then perform the actual reasoning, and finally compute a solution to the present task ( Melnik et al. 2023 ) . The reasoning components are generally recurrent neural networks or transformers that model the temporal evolution of objects autoregressively. Recent models are object-centric, i. e. they decompose a scene into individual separate objects, and permutation equivariant with respect to the objects ( Daniel and Tamar 2023 ; Wu et al. 2023 ) . In other words, they treat each object individually – an important inductive bias for physical reasoning. If object interactions are modeled explicitly, this is commonly done with graph neural networks ( Li et al. 2022 ) or with transformers ( Qi et al. 2020 ) . PHYRE ( Bakhtin et al. 2019 ) is a popular physical reasoning benchmark. It presents two dimensional vertical scenes of multiple distinct, rigid objects. These have one of three shapes (ball, bar, cup) and different sizes, and are subject to gravitation and friction. Each scenario is specified through the initial states of all objects, except one (PHYRE-B) or two (PHYRE-2B) red balls. A simulator models the scene’s temporal evolution with objects falling and colliding. The task is to place the red balls, so that a green object will touch a blue or purple object for >3 seconds. Some objects (red, green, blue, grey) can freely move, while others (purple, black) are fixed. In PHYRE terminology, a pre-defined initial object configuration is called the task . A template contains multiple tasks with similar but slightly different configurations. Selecting location and size of the red ball (we only consider PHYRE-B here) is called an action . Solution approaches often rely on forward modeling of object trajectories ( Qi et al. 2020 ; Wu et al. 2023 ; Daniel and Tamar 2023 ) . Their autoregressive nature restricts them to a trial and error approach for finding appropriate actions. In this work, we present a general conditional denoising diffusion model for object dynamics. We use PHYRE benchmark scenarios to evaluate our model, and find it to be quite challenging – especially, as it turns out, due to their limited coverage of possible interactions. To maintain a focus on dynamics modeling, we disregard some aspects of PHYRE: We use the available object feature vectors rather than image data. Additionally, we do not attempt the actual task of placing the red ball: While our model can find a suitable initial position for the red ball (see Figure 1 ), deploying it automatically would require an additional module to generate absolute (rather than relative) values for goal states. 2.2 Denoising Diffusion Models Figure 2: Attention-convolution block. The variables are defined as follows: o is the index of an object, l is the timestep in the trajectory, d is the number of object features. Denoising diffusion models are generative models that approximate data through a