SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation
Nadun Ranawaka 1,2∗ , Josiah Wong 1,3∗ , Wei-Lin Pai 3 , Wei-Teng Chu 3 , Tianyuan Dai 1,4 , Masoud Moghani 1,5 , Hang Yin 3 , Yunfan Jiang 1,3 , Wesley Durbano 1∗ , Brandon Huynh 1∗ , Yu Fang 1 , Linxi Fan 1 , Danfei Xu 1,2 , Ruohan Zhang 3 , Li Fei-Fei 3 , Bowen Wen 1 , Ajay Mandlekar 1† , Yuke Zhu 1,4† 1 NVIDIA 2 Georgia Institute of Technology 3 Stanford University 4 The University of Texas at Austin 5 University of Toronto ∗ Equal contribution † Equal advising · 2026
Read SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation by Nadun Ranawaka 1,2∗ , Josiah Wong 1,3∗ , Wei-Lin Pai 3 , Wei-Teng Chu 3 , Tianyuan Dai 1,4 , Masoud Moghani 1,5 , Hang Yin 3 , Yunfan Jiang 1,3 , Wesley Durbano 1∗ , Brandon Huynh 1∗ , Yu Fang 1 , Linxi Fan 1 , Danfei Xu 1,2 , Ruohan Zhang 3 , Li Fei-Fei 3 , Bowen Wen 1 , Ajay Mandlekar 1† , Yuke Zhu 1,4† 1 NVIDIA 2 Georgia Institute of Technology 3 Stanford University 4 The University of Texas at Austin 5 University of Toronto ∗ Equal contribution † Equal advising (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28276.txt.
Introduction
Nadun Ranawaka and seventeen colleagues at NVIDIA, Stanford, and partner universities built SimFoundry, a system that turns one real-world video into a ready-to-use simulation of the scene. Training and testing robot policies in reality is slow and expensive, so faithful digital twins offer a cheaper proving ground.
SimFoundry reconstructs an interactive twin automatically, then spawns digital cousins, variations that change objects, layout, or task while keeping what the objects afford. Policies trained with cousins transferred to real robots with no further tuning, and cousin training lifted success across object, scene, and task changes. Across seven manipulation tasks and five policy designs, simulation scores predicted real-world performance closely, making the system useful for judging policies before deployment.
Catalog record
| CREATOR | Nadun Ranawaka 1,2∗ , Josiah Wong 1,3∗ , Wei-Lin Pai 3 , Wei-Teng Chu 3 , Tianyuan Dai 1,4 , Masoud Moghani 1,5 , Hang Yin 3 , Yunfan Jiang 1,3 , Wesley Durbano 1∗ , Brandon Huynh 1∗ , Yu Fang 1 , Linxi Fan 1 , Danfei Xu 1,2 , Ruohan Zhang 3 , Li Fei-Fei 3 , Bowen Wen 1 , Ajay Mandlekar 1† , Yuke Zhu 1,4† 1 NVIDIA 2 Georgia Institute of Technology 3 Stanford University 4 The University of Texas at Austin 5 University of Toronto ∗ Equal contribution † Equal advising |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 110 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.28276
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation Nadun Ranawaka 1,2∗ , Josiah Wong 1,3∗ , Wei-Lin Pai 3 , Wei-Teng Chu 3 , Tianyuan Dai 1,4 , Masoud Moghani 1,5 , Hang Yin 3 , Yunfan Jiang 1,3 , Wesley Durbano 1∗ , Brandon Huynh 1∗ , Yu Fang 1 , Linxi Fan 1 , Danfei Xu 1,2 , Ruohan Zhang 3 , Li Fei-Fei 3 , Bowen Wen 1 , Ajay Mandlekar 1† , Yuke Zhu 1,4† 1 NVIDIA 2 Georgia Institute of Technology 3 Stanford University 4 The University of Texas at Austin 5 University of Toronto ∗ Equal contribution † Equal advising Abstract Training and evaluating robot policies in the real world is costly and difficult to scale. We introduce SimFoundry, a modular and automated system for zero-shot real-to-sim scene construction from a video. SimFoundry generates sim-ready digital twins and supports object, scene, and task editing, enabling the automated generation of diverse digital cousins: affordance-preserving variations of reconstructed real-world scenes. Policies trained on SimFoundry data transfer zero-shot to challenging real tasks involving multi-step manipulation, articulated object interaction, and bimanual interaction, and its digital cousins (variations of the original scene and objects) facilitate generalization to new real-world conditions. Across 7 manipulation tasks and 5 policy architectures, SimFoundry simulation evaluations strongly predict real-world performance, with mean Pearson correlation 0.911 and mean maximum ranking violation 0.018. When evaluating sim-trained policies zero-shot in the real world, policies trained with object, scene, and task cousins in simulation show average task success rate improvements of 17%, 21%, and 40%, respectively. Additional details at https://research.nvidia.com/labs/gear/simfoundry/ . keywords Real2Sim, Sim2Real, Scene Generation, Policy Learning, Policy Evaluation Figure 1 : Overview. SimFoundry takes a single real-world input video and automatically reconstructs an interactive, sim-ready digital twin of the scene. Based on the reconstructed digital twin, SimFoundry can further generate an unlimited number of digital cousins — affordance-preserving variants of the original scene, spanning three different axes of variation, which we term object , scene , and task cousins , respectively. These generated simulation environments support both real-to-sim policy evaluation and sim-to-real policy training, enabling policies to be benchmarked and improved at scale before deployment in the real world. \abscontent 1 Introduction Robotic foundation models [ 5 , 61 ] trained on large-scale robot manipulation datasets have enabled robots to perform a wide range of manipulation tasks autonomously. However, sourcing high-quality robot manipulation data in large volumes is labor-intensive, often involving large-scale robot teleoperation efforts spanning many months or years [ 20 , 6 , 39 , 5 ] . Moreover, evaluating trained foundation models in a systematic and scientific manner on real-world manipulation problems of interest can be costly and require thousands of trials across different tasks to make rigorous comparisons [ 4 ] . In response to these bottlenecks, recent work has explored simulation as a scalable alternative for training and evaluating robot manipulation models. Automated data generation tools can synthesize large volumes of diverse, high-quality demonstrations with minimal human effort [ 18 , 58 , 37 , 23 , 46 ] , and have been used to train and improve real-world agents [ 55 , 86 , 13 , 26 ] . Recent work has also shown that simulation-based evaluations can strongly correlate with real-world results, offering a time- and cost-efficient alternative to physical benchmarking [ 47 , 3 ] . However, manually constructing simulation environments remains challenging, especially when they must align with real-world scenes and tasks in visuals, geometry, and dynamics. To address the issues of manual simulation development, real-to-sim scene construction [ 79 , 19 ] has emerged as a paradigm for generating synthetic scenes grounded in the real world. By leveraging 3D reconstruction and generative models, users can create “sim-ready” environments that support physically grounded robotic interaction with minimal manual effort. While this reduces environment-authoring overhead and enables both sim-to-real transfer [ 17 , 79 , 105 , 19 , 27 , 34 ] and predictive real-world benchmarking [ 32 , 104 , 33 ] , few works provide a unified system that automates scene reconstruction while also performing real-to-sim policy evaluation and training policies that transfer across domains. Recent systems that primarily generate simulation-ready 3D scenes [ 96 , 63 , 85 ] often lack the physical interaction, task specification, or data-generation machinery needed to close the sim-to-real policy-learning loop. Conversely, systems designed for simulation-based policy evaluation often assume manually tuned scenes, focus on short-horizon atomic manipulation, or do not support automatically generating diverse objects, scenes, and tasks for policy training [ 32 , 27 , 34 , 101 , 40 ] . To this end, we introduce SimFoundry, a unified and modular system that turns a single real-world input video into interactive simulation environments for both policy evaluation and policy training. SimFoundry unifies three capabilities that are often treated separately in prior work: reconstructing a sim-ready digital twin, expanding that reconstruction into diverse training environments, and using the resulting simulations to both benchmark and train robot policies. Its modular design decomposes real-to-sim construction into interchangeable components for perception, asset generation, pose alignment, articulation, physics annotation, and data generation, allowing improved foundation models to be incorporated as they become available without redesigning the full system. To scale beyond a single reconstructed scene, SimFoundry automatically generates digital cousins : affordance-preserving variations that maintain the task-relevant semantics of the original scene while varying object instances, spatial layouts, and feasible manipulation tasks. These digital cousins, spanning object-, scene-, and task-level variations, enable large-scale synthetic data generation for training robot policies that are robust to changes in object morphology and scene layout and that generalize to related manipulation tasks. SimFoundry environments also produce simulation evaluations that strongly correlate with real-world policy performance. Finally, policies trained on SimFoundry simulation trajectories can successfully deploy in their real-world counterparts. Summary of Contributions: \bullet We introduce SimFoundry, a fully automated and modular real-to-sim system that generates interactive, sim-ready scenes from a single video. SimFoundry supports rigid and articulated objects, physics annotations, and automated object, scene, and task cousins, augmenting a single scene into diverse training environments. Across 12 reconstruction scenes, SimFoundry achieves zero-shot F1 scores of 0.81 – 0.92 , which can be further improved to 0.93 – 0.99 with only 3 minutes of per-object tuning. \bullet We demonstrate SimFoundry on manipulation tasks that exceed prior real-to-sim work in manipulation complexity, including multi-step tasks, articulated-object interaction, and bimanual coordination on both DROID and YAM robot embodiments. Across 7 tasks and 5 policy types, SimFoundry simulation evaluations strongly correlate with real-world performance with a mean Pearson correlation of 0.911 and an MMRV of 0.018 , outperforming the state-of-the-art baseline [ 32 ] by over 0.59 on the Pearson correlation. \bullet We demonstrate that SimFoundry-generated data can train policies that transfer to the real world and generalize beyond the reconstructed twin. Policies trained on SimFoundry simulation data achieve strong, and sometimes near-perfect, real-world success rates on both YAM and DROID. Multi-task SimFoundry data further improves generalist policies by up to 31\% in simulation and 18\% in the real world, while reaching 29\% success rate on held-out real tasks. 2 Related Work Table 1 : System comparison. SimFoundry provides a unified and modular pipeline for real-to-sim