Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulation
Yajvan Ravan*, Adam Rashid*, Alan Yu, Kai McClennen, Gio Huh, Kevin Yang, [0.2em] Zhutian Yang, Qinxi Yu, Xiaolong Wang, Phillip Isola, Ge Yang*, [0.6em] MIT CSAIL, FortyFive Labs, Caltech, Harvard University, UC San Diego, [0.6em] Equal contribution · 2026
Read Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulation by Yajvan Ravan*, Adam Rashid*, Alan Yu, Kai McClennen, Gio Huh, Kevin Yang, [0.2em] Zhutian Yang, Qinxi Yu, Xiaolong Wang, Phillip Isola, Ge Yang*, [0.6em] MIT CSAIL, FortyFive Labs, Caltech, Harvard University, UC San Diego, [0.6em] Equal contribution (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00244.txt.
Introduction
Yajvan Ravan and 10 co-authors introduce Lucid-XR, a data engine that creates training material for robots inside extended reality. At its core sits vuer, a physics simulation that runs in a web browser on the headset itself. That setup lets people anywhere supply natural hand and body demonstrations without special laboratory gear.
Demonstrations are retargeted from human poses to robot poses, then amplified by a physics-guided video generator that a user can steer with plain language. Robot vision policies trained only on this synthetic data worked in real rooms they had never seen, including cluttered and badly lit ones. Example tasks involve soft materials, loose particles, and firm objects in contact.
Catalog record
| CREATOR | Yajvan Ravan*, Adam Rashid*, Alan Yu, Kai McClennen, Gio Huh, Kevin Yang, [0.2em] Zhutian Yang, Qinxi Yu, Xiaolong Wang, Phillip Isola, Ge Yang*, [0.6em] MIT CSAIL, FortyFive Labs, Caltech, Harvard University, UC San Diego, [0.6em] Equal contribution |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.00244
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulation Yajvan Ravan*, Adam Rashid*, Alan Yu, Kai McClennen, Gio Huh, Kevin Yang, [0.2em] Zhutian Yang, Qinxi Yu, Xiaolong Wang, Phillip Isola, Ge Yang*, [0.6em] MIT CSAIL, FortyFive Labs, Caltech, Harvard University, UC San Diego, [0.6em] Equal contribution Abstract We introduce Lucid-XR, a generative data engine for creating diverse and realistic-looking multi-modal data to train real-world robotic systems. At the core of Lucid-XR is vuer, a web-based physics simulation environment that runs directly on the XR headset, enabling internet-scale access to immersive, latency-free virtual interactions without requiring specialized equipment. The complete system integrates on-device physics simulation with human-to-robot pose retargeting. Data collected is further amplified by a physics-guided video generation pipeline steerable via natural language specifications. We demonstrate zero-shot transfer of robot visual policies to unseen, cluttered, and badly lit evaluation environments, after training entirely on Lucid-XR’s synthetic data. We include examples across dexterous manipulation tasks that involve soft materials, loosely bound particles, and rigid body contact. Project website: https://lucidxr.github.io Keywords: Extended-reality, world-model, synthetic data 1 Introduction: From Atoms to Bits When viewed from afar, training a robot controller is not too different from making a movie, in that both involve carefully curating content. Feature films over the years have transitioned steadily from practical special effects in the physical world to digitally created virtual special effects, driven primarily by creators’ demand for even greater freedom in storytelling [ 1 , 2 , 3 , 4 ] . Hardening robotic systems for real-world deployment requires a similar level of control over the training environment to cover rare but mission-critical events that, by definition, are scarce in real-world datasets. What makes robotics more difficult is that digitally creating millions of realistic-looking virtual worlds at the scale required for our robots to generalize to the real world is infeasibly expensive. In this work, we introduce Lucid-XR, an extended-reality (XR) data engine for robot manipulation. Lucid-XR uses a generative image pipeline to convert human demonstration data collected in low-fidelity virtual environments into diverse and visually realistic data to train the robot. Key to our vision is to enable internet scale deployment of real-time multi-physics simulations through the web browser to enable the crowd-sourcing of unlimited human demonstration data. Furthermore, virtual demonstrations in sparsely populated 3D environments alone are insufficient for training real-world computer vision systems. We leverage language and text-to-image generative models to construct a physics-guided video generation pipeline that amplifies a small number of simple designs into millions of diverse and realistic-looking multi-view images for the robot. One challenge that makes crowd-sourcing teleoperation difficult is that retargeting human poses to robot form factors that are kinematically different involves writing custom computer code for each robot and setting up a server. We solve this problem by leveraging an inverse kinematics solver, built-in to MuJoCo, and allowing users to define bindings between motion capture (MoCap) sites and robot parts in the markup schema. Our contributions are three-fold: first, moving physics simulation onto the XR device, to deliver a latency-free multi-physics simulation in an immersive environment via the open internet; second, a way to retarget human pose data to virtual robots, without requiring a custom computer program; and third, a demonstration of a policy trained on Lucid-XR’s synthetic data deployed on scanned digital twins of real environments. Figure 1: System Schematic of the Lucid-XR Pipeline. The results in this paper require hand-crafted, but basic 3D scenes. The data collection is done collectively by the authors. The resulting simulated datasets are augmented by a generative pipeline powered by language and text-to-image models. 2 A Touch of Physics in Extended Reality (XR) Modern internet browsers are de facto operating systems built according to a commonly agreed-upon programming interface. The key limitation is the constraint on compute and memory, as the device’s operating system needs to throttle the browser process to maintain a smooth user experience while handling a multitude of system and application tasks. In the past decade, however, the competition between vendors has significantly improved browser performance; on newer extended-reality devices, the browser receives first-class support in feature sets, system resource limits, and performance. Lucid-XR builds on three web standards. First, to ensure simulations are fast , we compile MuJoCo into WebAssembly, bypassing the single-threaded V8 engine and achieving near-native speed directly on XR hardware. Second, we built upon the web-XR standard, a user-interaction programming interface that is shared across device vendors. This enabled us to support hands , motion controllers , and XR devices from as low as $300 to $4000. The third web standard is webGL. We wrote a performant rendering, interaction, and programming interface from scratch using react-three/fiber, a modern lightweight and performant 3D framework with strong community backing. This has enabled real-time visualization of complex scenes and deformable objects. The current technological landscape is fast-evolving — in the near future, when webGPU becomes fully supported, we will be able to parallelize the physics simulation using compute shaders with hardware acceleration. This will enable more complex physics that involve a large quantity of interacting particles, deformable vertices, or liquid. Figure 2: Moving physics simulation on-device enables untethered access to immersive simulations. The key benefits are twofold: first, it enables the simulation of deformable objects that involve modifying a large amount of mesh data that are too slow to send over WiFi. Second, it eliminates delays due to network latency, allowing the simulation to run at the device’s native frame rate. 2.1 Multi-Physics Simulation in Vuer At the core of Lucid-XR is vuer , a simulator-agnostic XR framework running in the browser. We use MuJoCo [ 5 ] compiled to WebAssembly with a custom react-three/fiber front-end, achieving real-time simulation at 90 fps on Apple Vision Pro. Data is collected at 25 fps; each step takes under 12 ms to simulate, so latency is negligible unless overloaded. The MuJoCo decimation parameter (5–20 steps per frame) controls simulation fidelity. Rendering runs natively on the XR device at 90 fps for user comfort. Like Blender, vuer is simulator-agnostic and benefits as physics engines improve. Flexible materials. Prior VR setups run physics on an external computer [ 6 ] , which introduces latency when simulating flexible or particle-rich scenes. Running physics on-device eliminates this bottleneck and enables interactive deformable-object simulation (Fig. 3 ). Fluid forces. MuJoCo’s integrated fluid model [ 7 ] supports wind and fluid interactions with rigid and deformable objects. For example, Fig. 3 c shows poker cards falling against a capsule under air resistance. Collision without convex decomposition. We include the signed-distance function (SDF) collision solver in Mujoco by default, removing the need for convex decomposition during modeling. This simplifies setup, at the cost of higher runtime compute and memory. Figure 3: Multiple types of physics running natively inside the browser in real-time. From left to right: flexible material interacting with a solid; Signed-distance function (SDF) based collision solver for non-convex shapes; A deck of cards interacting with air/wind; Soft skin material interacting with a solid. Rendering and simulation are both native in a web browser. 2.2 Precise Interactions at A Distance: Hitchhiking Controllers The virtual world presents an opportunity to define new ways of interacting with robots beyond the constraint of real-world physicality. When the robot is situated