LychSim: A Controllable and Interactive Simulation Framework for Vision Research
Wufei Ma, Chloe Wang, Siyi Chen, Jiawei Peng, Patrick Li, Alan Yuille · 2026
Read LychSim: A Controllable and Interactive Simulation Framework for Vision Research by Wufei Ma, Chloe Wang, Siyi Chen, Jiawei Peng, Patrick Li, Alan Yuille (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12449.txt.
Introduction
Ma and five colleagues lower the barrier to simulation for vision research with LychSim, built on Unreal Engine 5. Modern simulators demand deep graphics and game skills, yet simulation stays vital for closed-loop tuning and tough out-of-distribution tests. LychSim offers a simple Python interface, a procedural pipeline that builds varied, high-quality scenes with rich two- and three-dimensional labels, and native Model Context Protocol support. That last piece turns the simulator into a live playground for reasoning agents.
Catalog record
| CREATOR | Wufei Ma, Chloe Wang, Siyi Chen, Jiawei Peng, Patrick Li, Alan Yuille |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46324 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12449
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LychSim: A Controllable and Interactive Simulation Framework for Vision Research Wufei Ma, Chloe Wang, Siyi Chen, Jiawei Peng, Patrick Li, Alan Yuille Abstract While self-supervised pretraining has reduced vision systems’ reliance on synthetic data, simulation remains an indispensable tool for closed-loop optimization and rigorous out-of-distribution (OOD) evaluation. However, modern simulation platforms often present steep technical barriers, requiring extensive expertise in computer graphics and game development. In this work, we present LychSim, a highly controllable and interactive simulation framework built upon Unreal Engine 5 to bridge this gap. LychSim is built around three key designs: (1) a streamlined Python API that abstracts away underlying engine complexities; (2) a procedural data pipeline capable of generating diverse, high-fidelity environments with varying out-of-distribution (OOD) visual challenges, paired with rich 2D and 3D ground truths; and (3) a native integration of the Model Context Protocol (MCP) that transforms the simulator into a dynamic, closed-loop playground for reasoning agentic LLMs. We further annotate scene-level procedural rules and object-level pose alignments to enable semantically aligned 3D ground truths and automated scene modification. We demonstrate LychSim’s capability across multiple downstream applications, including serving as a synthetic data engine, powering reinforcement learning-based adversarial examiners, and facilitating interactive, language-driven scene layout generation. To benefit the broader vision community, LychSim will be made publicly available, including full source code and various data annotations. uto counter]lightcodefig[2][] enhanced jigsaw, listing only, listing engine=minted, minted language=#2, minted options=fontsize= , breaklines, linenos, numbersep=8pt, style=tango, baselinestretch=1.1 , arc=3pt, boxrule=0.5pt, colback=blue-bg, colframe=blue-frame, coltitle=blue-text, fonttitle= , attach boxed title to top left=yshift=-2mm, xshift=3mm, boxed title style=colback=blue-title, colframe=blue-frame, arc=2pt, boxrule=0.5pt, left=18pt, #1 uto counter]lightcode[2][] enhanced jigsaw, breakable, listing only, listing engine=minted, minted language=#2, minted options=fontsize= , breaklines, linenos, numbersep=8pt, style=tango, baselinestretch=1.1 , arc=3pt, boxrule=0.5pt, colback=blue-bg, colframe=blue-frame, coltitle=blue-text, fonttitle= , attach boxed title to top left=yshift=-2mm, xshift=3mm, boxed title style=colback=blue-title, colframe=blue-frame, arc=2pt, boxrule=0.5pt, left=18pt, #1 uto counter]orangecode[2][] enhanced, breakable, listing only, minted language=#2, segmentation style=draw=none, lower separated=false, minted options=fontsize= , breaklines, linenos, numbersep=8pt, style=tango, baselinestretch=1.1 , arc=3pt, boxrule=0.5pt, colback=orange-bg, colframe=orange-frame, coltitle=orange-text, fonttitle= , attach boxed title to top left=yshift=-2mm, xshift=3mm, boxed title style=colback=orange-title, colframe=orange-frame, arc=2pt, boxrule=0.5pt, left=18pt, #1 1 Introduction Recent advancements in self-supervised and weakly-supervised visual pretraining have revolutionized the field of computer vision. Pretraining models [ 39 , 16 , 66 , 3 , 35 , 47 ] have demonstrated impressive capabilities to learn rich and transferable visual representations from Internet-scale image and video data, using raw visual contents or naturally occurring text captions. These advances substantially reduce the amount of labeled data or task-specific fine-tuning required to achieve strong performance across a broad range of downstream tasks, spanning both 2D vision ( e.g. , classification and segmentation) and 3D vision ( e.g. , depth estimation, 3D object detection, and pose estimation). Consequently, the dependence on manually-curated synthetic datasets, which can help mitigate the scarcity of real-world annotations, has diminished as powerful visual representations can now be learned directly from large-scale, unannotated data. Despite the reduced reliance on synthetic data for direct supervised training, the role of simulation remains critically important for computer vision research, as driven by two key objectives. First, simulation environments provide an unparalleled platform for analyzing and understanding complex vision systems. They offer comprehensive and perfectly aligned 2D and 3D ground truths, enable the creation of diverse and controlled Out-of-Distribution (OOD) scenarios, and allow for rigorous analysis of a model’s robustness and generalization capabilities in ways that real-world data collection simply cannot replicate. Second, interactive, high-fidelity simulation is essential for closed-loop training and optimization, especially for embodied AI and robotics. In these applications, agents must learn complex control policies through interaction with their environment, making a realistic and safe virtual playground an indispensable tool for developing and testing advanced, interactive AI systems. In this work, we present LychSim , a controllable and interactive simulation framework featuring three key designs: (1) Ease of use. We provide a streamlined Python API that abstracts away various technical complexities in UE5 and C++ development, empowering researchers to script and manipulate high-fidelity 3D scenes without prior computer graphics expertise. (2) A built-in procedural data pipeline with rich 2D and 3D ground truths. LychSim seamlessly generates diverse environments with various out-of-distribution (OOD) visual challenges, paired with pixel-accurate annotations. Beyond standard labels, our engine models underlying 3D structures and provide ground truths for part segmentation, point maps, and occlusion ratios/relationships for objects extending beyond visible regions. This unlock new opportunities to explore richer 3D representations and modern 3D learning pipelines. (3) Interactive simulation. By natively integrating programmatic controls and Model Context Protocol (MCP), LychSim enables algorithms and agentic LLMs to easily navigate, query, and manipulate the 3D world in real-time. This dynamic, closed-loop playground enables many advanced applications, such as RL-based adversarial examiners that systematically identify vision models’ weaknesses and interactive, language-driven agentic scene planning. With the controllable and interactive simulation provided by LychSim, we hope to help advance computer vision research towards a better understanding and more accurate generation of the 3D world. We believe in the great potential of graphics-based simulation for computer vision research, as a rigorous evaluation framework with diverse 2D and 3D ground truths or a controllable and scalable data engine for model training. We will release our LychSim publicly, including: (1) the complete C++ and Python source code , and (2) associated data annotations , such as procedural rules for scene generation and pose alignments for object meshes. Figure 1: We introduce LychSim, a controllable and interactive simulation framework designed for computer vision research. Our simulation features three key designs: (1) ease of use with a streamlined Python interface, (2) a built-in procedural data pipeline with rich 2D and 3D ground truths, and (3) controllable and interactive simulation. The remainder of this paper is structured as follows. We introduce the system design and core functionalities of the LychSim simulation system in Section 2 . Then we describe the Python API and the Model Context Protocol (MCP) integration in Section 3 . Next in Section 4 we present three case compelling studies and demonstrate practical utilities of LychSim in advanced vision research. Lastly we discuss related works in Section 5 and summarize our contributions in Section 6 . 2 Simulation System Design 2.1 3D Assets and Data Annotations A key advantage of UE5-based simulation systems is direct access to a vast library of high-quality, artist-created 3D assets. By operating within this native ecosystem, we avoid the rendering artifacts and material inconsistencies that often arise when assets are ported across different simulation platforms. However, these raw assets are often unstructured and lack a unified representation, making automated manipulation challenging. To address this and better support advanced computer vision research, we introduce two key data extensions. First, we annotate the category, canonical scale, and pose alignment for the 3D object assets within these scenes. These annotations are critical for producing semantically aligned ground-truth 3D object poses and facilitating programmatic object placement and scene manipulation. Second, we define scene-level procedural rules, such as navigable floor spaces, road areas, pedestrian walks, and dynamic trajectories. These spatial priors guide the structural generation process, ensuring that newly synthesized layouts remain faithful to the original scene semantics. The list of 3D assets used in our LychSim and all corresponding data annotations will released