Counterfactual World Models via Digital Twin-conditioned Video Diffusion

Yiqing Shen, Aiza Maksutova, Chenjia Li, Mathias Unberath · 2025

Read Counterfactual World Models via Digital Twin-conditioned Video Diffusion by Yiqing Shen, Aiza Maksutova, Chenjia Li, Mathias Unberath (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17481.txt.

Description

Yiqing Shen and three co-authors teach world models to answer what-if questions. World models predict how a scene evolves under a control signal, letting agents rehearse the future, but standard models extrapolate only what they saw. Many uses, like stress-testing physical AI under varied conditions, need counterfactuals: what would happen if this object were removed? Pixel-space models cannot edit one object's properties cleanly, because objects and relations are entangled in the pixels.

Their framework first builds a digital twin of the scene, a structured-text description of its objects and relationships. A large language model then reasons over that description to predict how an intervention propagates through time, and a video diffusion model, conditioned on the edited description, renders the counterfactual sequence. On two benchmarks the approach reached state-of-the-art performance, suggesting structured scene descriptions are strong control signals for world models.

Catalog record

CREATORYiqing Shen, Aiza Maksutova, Chenjia Li, Mathias Unberath
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-17481

Browse the full library · About the Tower of Babel