SceneForge: Structured World Supervision from 3D Interventions
Jizhizi Li 1 Jiayang Ao 1 [email protected] [email protected] Danny Wicks 1 Petru-Daniel Tudosiu 1 [email protected] [email protected] 1 Canva Research · 2026
Read SceneForge: Structured World Supervision from 3D Interventions by Jizhizi Li 1 Jiayang Ao 1 [email protected] [email protected] Danny Wicks 1 Petru-Daniel Tudosiu 1 [email protected] [email protected] 1 Canva Research (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14399.txt.
Introduction
Li and colleagues introduce SceneForge, a framework that builds structured supervision for tasks spanning images and text from editable 3D world states. Each scene is kept as a lasting world with meaning, shape and physics links. Edits such as object removal or camera moves then propagate consistently through shadows, reflections and views. Built with Infinigen and Blender, the resource covers over two thousand indoor scenes with counterfactual pairs and aligned annotations. Under matched training budgets, SceneForge supervision improves object and scene removal across benchmarks in both numeric and judged tests.
Catalog record
| CREATOR | Jizhizi Li 1 Jiayang Ao 1 [email protected] [email protected] Danny Wicks 1 Petru-Daniel Tudosiu 1 [email protected] [email protected] 1 Canva Research |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42956 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14399
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SceneForge: Structured World Supervision from 3D Interventions Jizhizi Li 1 Jiayang Ao 1 [email protected] [email protected] Danny Wicks 1 Petru-Daniel Tudosiu 1 [email protected] [email protected] 1 Canva Research Abstract Many multimodal learning tasks require supervision that remains consistent across edits, viewpoints, and scene-level interventions. However, such supervision is difficult to obtain from observation-level datasets, which do not expose the underlying scene state or how changes propagate through it. We present SceneForge , an intervention-driven framework that generates structured supervision from editable 3D world states. SceneForge represents each scene as a persistent world with semantic, geometric, and physical dependencies. By applying explicit interventions (e.g., object removal or camera variation) and propagating their effects through scene dependencies, SceneForge renders supervision that remains consistent with object structure and scene-level effects. This produces aligned outputs including counterfactual observations, multi-view observations, and effect-aware signals such as shadows and reflections, all derived from a shared world state rather than post hoc image-space processing. We instantiate SceneForge using Infinigen and Blender to construct a licensing-clean indoor supervision resource with a large number of counterfactual pairs and aligned annotations from over 2K scenes, covering both diverse single-view and registered multi-view settings. Under matched training budgets, incorporating SceneForge supervision improves both object removal and scene removal performance across multiple benchmarks in both quantitative and qualitative evaluation. These results indicate that modeling supervision as structured state transitions in editable worlds provides a practical and scalable foundation for intervention-consistent multimodal learning. 1 Introduction Modern multimodal systems increasingly require supervision that is not only large-scale, but also structured, controllable, and aligned across edits, viewpoints, and modalities. In many practical settings, models must reason jointly over text, images, videos, geometric cues, and object-level structure while remaining consistent under scene edits, temporal evolution, and camera variation. Yet this kind of supervision is difficult to obtain from observation-level datasets, which expose rendered snapshots rather than the underlying scene state and therefore rarely provide matched interventions, physically consistent counterfactuals, layered decompositions, relation-aware annotations, or aligned multimodal signals for the same scene. Diagnostic synthetic datasets such as CLEVR ( 10 ) have demonstrated the value of controllable world generation for visual reasoning, but they remain focused on fixed reasoning tasks rather than intervention-driven supervision for modern generative and editing models. Existing efforts partially address this gap from two directions. Real or hybrid editing datasets provide task-relevant examples for image and video manipulation, including manually annotated and automatically generated instruction-based editing resources, but they typically construct supervision at the observation level, for example as before-and-after pairs, task-specific edits, or localized annotations ( 33 ; 12 ; 34 ; 15 ) . Synthetic 3D pipelines, in contrast, provide strong scene control and rich render-time annotations through controllable simulators and procedural generators ( 11 ; 7 ; 26 ) , but they often treat scenes primarily as configurations to be rendered rather than as persistent worlds whose state can be manipulated, queried, and updated in a structured manner. What remains missing is a framework that models supervision generation as structured state transition in an editable world, so that a single intervention can induce consistent updates across entities, views, modalities, and scene-level effects. In this work, we present SceneForge , a world-centric framework for structured supervision from 3D interventions. The key idea is to cast supervision generation as intervention on an editable world state rather than as post hoc construction from observations. SceneForge represents each scene as a structured world composed of stable structural elements and editable dynamic objects. It models semantic, geometric, and physical dependencies among these entities, and propagates the effects of interventions through the maintained world so that changes remain consistent with occlusion, support, visibility, lighting, and other scene-level constraints. Under this formulation, each world state and state transition induces aligned observation- and layer-level supervision, including RGB observations, multi-view renderings, counterfactual edits, decomposition outputs, geometry-aware signals, and effect-aware channels such as shadows and reflections. Because these outputs are derived from the same maintained scene graph, they are aligned by construction rather than matched after the fact in image space. This same formulation also provides a natural basis for extending SceneForge toward richer linguistic supervision and broader asset diversity. We evaluate SceneForge through dataset instantiation and statistics, compositional layer decomposition, intervention-driven counterfactual generation, and downstream applications on both object-level and scene-level removal tasks. Our contributions are fourfold. First, we introduce SceneForge, a world-centric framework in which structured multimodal supervision is generated through explicit interventions and dependency-aware state propagation. Second, we instantiate and release SceneForge with Infinigen and Blender as a licensing-clean framework and resource, including aligned object-removal datasets with complementary coverage-oriented single-view and multi-view splits together with the supporting code and documentation. Third, we characterize these datasets through statistics, compositional layer decomposition, and intervention-driven counterfactual analyses grounded in cross-view and cross-edit consistency. Fourth, we present downstream applications on both object-level and scene-level removal tasks under matched training settings. More broadly, the same formulation naturally extends to additional supervision types, such as dense language or point clouds, and is not restricted to Infinigen-native assets. 2 Related Work Editing datasets and hybrid supervision pipelines. Recent work has produced increasingly rich resources for instruction-guided and object-centric image editing, including instruction-following editors, manually curated datasets, automatically generated supervision, and hybrid pipelines that combine real imagery with controlled editing operations ( 2 ; 33 ; 20 ; 12 ; 34 ) . Related work on image matting and object-aware image manipulation further emphasizes fine-grained foreground understanding and decomposition-aware supervision ( 15 ) . These resources provide task-relevant supervision for image or video editing, but they typically construct labels at the observation level, such as paired edits, localized annotations, or instruction-conditioned targets. Concurrent work such as ROSE ( 22 ) focuses on side-effect-aware object removal in videos, but remains centered on paired counterfactual video supervision in more limited scene settings. In contrast, SceneForge treats supervision as the outcome of explicit interventions on an underlying world state, enabling broader editable functions, freer camera control, and aligned multimodal outputs in addition to edited observations, decompositions, relations, and cross-view or temporal consistency. Synthetic 3D data generation. A complementary line of work studies synthetic data generation through controllable simulators, synthetic benchmarks, procedural generators, and rendering pipelines ( 27 ; 28 ; 5 ; 25 ; 11 ; 4 ; 7 ; 26 ) . Related embodied and interactive environments further illustrate the value of procedural or world-based simulation for controllable learning settings ( 6 ; 29 ; 3 ) . These systems provide flexible scene control, scalable rendering, and rich annotations, but they often expose scenes primarily as rendering configurations rather than as persistent structured worlds whose interventions induce consistent updates across entities, modalities, and views. SceneForge builds on the strengths of synthetic generation while centering the editable world state and its transitions as the primary object of supervision. Controllable worlds, reasoning datasets, and structured scene representations. A broader literature has shown the value of structured scene control for visual reasoning and relational understanding. Diagnostic synthetic datasets such as CLEVR use controlled worlds to probe compositional reasoning ( 10 ) , while Visual Genome, GQA, visual relationship detection, scene graph modeling, and AGQA provide relation-rich supervision for scene understanding and grounded reasoning ( 13 ; 9 ; 21 ; 32 ;