Counterfactual World Models via Digital Twin-conditioned Video Diffusion
Yiqing Shen, Aiza Maksutova, Chenjia Li, Mathias Unberath · 2025
Read Counterfactual World Models via Digital Twin-conditioned Video Diffusion by Yiqing Shen, Aiza Maksutova, Chenjia Li, Mathias Unberath (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17481.txt.
Description
Yiqing Shen and three co-authors teach world models to answer what-if questions. World models predict how a scene evolves under a control signal, letting agents rehearse the future, but standard models extrapolate only what they saw. Many uses, like stress-testing physical AI under varied conditions, need counterfactuals: what would happen if this object were removed? Pixel-space models cannot edit one object's properties cleanly, because objects and relations are entangled in the pixels.
Their framework first builds a digital twin of the scene, a structured-text description of its objects and relationships. A large language model then reasons over that description to predict how an intervention propagates through time, and a video diffusion model, conditioned on the edited description, renders the counterfactual sequence. On two benchmarks the approach reached state-of-the-art performance, suggesting structured scene descriptions are strong control signals for world models.
Catalog record
| CREATOR | Yiqing Shen, Aiza Maksutova, Chenjia Li, Mathias Unberath |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-17481