Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and Outputs CCS: Human-centered computing Interactive systems and tools CCS: Human-centered computing Virtual reality CCS: Computing methodologies Virtual reality CCS: Computing methodologies Computer vision

Cyrus Vachha, Yixiao Kang, Zach Dive, Ashwat Chidambaram, Anik Gupta, Eunice Jun, Björn Hartmann · 2025

Read Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and Outputs CCS: Human-centered computing Interactive systems and tools CCS: Human-centered computing Virtual reality CCS: Computing methodologies Virtual reality CCS: Computing methodologies Computer vision by Cyrus Vachha, Yixiao Kang, Zach Dive, Ashwat Chidambaram, Anik Gupta, Eunice Jun, Björn Hartmann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20129.txt.

Introduction

Cyrus Vachha and six co-authors built Dreamcrafter, a VR studio for editing captured 3D scenes. Making 3D content is central to spatial computing. Two philosophies compete. One is direct immersive handling of the content. The other is AI that edits radiance fields at a high level, at the cost of high latency. Dreamcrafter studies how to fuse the two in one real-time system.

Its design is modular, so generative tools can be swapped in. It offers many levels of control for making objects. They run from natural language to direct handling. Proxy forms keep the interaction smooth while slow tools run behind them. The work was presented at CHI 2025. It reports what controls users prefer. It argues that generative interfaces past plain text boost creativity in scene work.

Catalog record

CREATORCyrus Vachha, Yixiao Kang, Zach Dive, Ashwat Chidambaram, Anik Gupta, Eunice Jun, Björn Hartmann
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-20129

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and Outputs CCS: Human-centered computing Interactive systems and tools CCS: Human-centered computing Virtual reality CCS: Computing methodologies Virtual reality CCS: Computing methodologies Computer vision Cyrus Vachha, Yixiao Kang, Zach Dive, Ashwat Chidambaram, Anik Gupta, Eunice Jun, Björn Hartmann Abstract . Authoring 3D scenes is a central task for spatial computing applications. Competing visions for lowering existing barriers are (1) focus on immersive, direct manipulation of 3D content or (2) leverage AI techniques that capture real scenes (3D Radiance Fields such as, NeRFs, 3D Gaussian Splatting) and modify them at a higher level of abstraction, at the cost of high latency. We unify the complementary strengths of these approaches and investigate how to integrate generative AI advances into real-time, immersive 3D Radiance Field editing. We introduce Dreamcrafter, a VR-based 3D scene editing system that: (1) provides a modular architecture to integrate generative AI algorithms; (2) combines different levels of control for creating objects, including natural language and direct manipulation; and (3) introduces proxy representations that support interaction during high-latency operations. We contribute empirical findings on control preferences and discuss how generative AI interfaces beyond text input enhance creativity in scene editing and world building. For videos and additional materials visit our project page: https://dream-crafter.github.io/ Figure 1. System Overview: (Left) View of edits, proxy generations, and spatial annotations applied while scene editing. (Right) Pipeline overview of system including a Unity app interfacing with online and offline modules to edit scenes for editing radiance fields, creating new objects, and creating 2D stylized renders. (Left) - Visual 3D VR scene depicting the living room of a house. There are various objects such as a couch, chair, table, etc. spread out across the room, each accompanied by a spatial annotation representing the edit information inputted by the user. The couch has an annotation reading “make it grey”, followed by a generated 2D grey version of the couch displayed above the 3D asset. The chair has the edit instruction “make it retro” along with a retro-styled 2D image above. The table has the instruction “place a table”, and depicts a rendered 3D table on the floor. (Right) - Architecture diagram titled Independent Components for Interacting with and Editing Scenes, containing three components: Unity App, Edit Radiance Fields Objects, and Generate new objects in the scene. Edit Radiance Fields Objects contains the following workflow: Unity app feeds into JSON Edit Instruction, which feeds into Offline Radiance Field Editing Model, feeding into Save Updated Model, then Online Radiance Field Preview, and finally back to Unity App. JSON Edit Instruction also directly feeds into Online Radiance Field Preview. Generate new objects in the scene contains the following workflow: Unity app feeds into JSON Create Instruction, which feeds into Offline 3D Object Generation Model, feeding into Save New Generated Model, feeding into Online 3D Object Generation Preview, and finally back to Unity App. JSON Create Instruction also feeds into Online 3D Object Generation Preview. 1. Introduction Spatial computing applications such as Augmented and Virtual Reality rely on 3D content and scenes. Thus, creating appropriate tools for authoring and editing 3D content has been a long-standing key challenge for HCI researchers. Traditionally, mesh-and-texture-based approaches have been used to author 3D content. Various research efforts to introduce better editing techniques notwithstanding (e.g., ( Igarashi et al. 2006 ; Bae et al. 2008 ) ), the expertise hurdle to create and modify 3D content in this way has been high, generally leaving such authoring to a small number of expert users. One avenue to lower the authoring barrier has been to embrace authoring in VR (e.g. Google Tiltbrush ( Brush 2016 ) ), where direct 3D input is possible through VR controllers (or gestures) in an immersive environment. This approach decreases the gulf of execution ( Hutchins et al. 1985 ) inherent in prior approaches to modeling 3D content using 2D input devices. More recently, two additional developments hold the promise of reducing authoring burdens. First, novel approaches for representing 3D scenes based on radiance fields (e.g., NeRFs ( Mildenhall et al. 2020 ) and 3D Gaussian Splatting ( Kerbl et al. 2023 ) ) allow for straightforward capture of photorealistic environments from real scenes using common cameras, instead of having to model objects from scratch. Second, generative AI developments have introduced novel ways of editing 3D scenes like radiance fields at higher levels of abstraction, e.g. through text instructions (as in Instruct-NeRF2NeRF ( Haque et al. 2023 ) ). While offering the ability to edit at a semantic level rather than a lower geometry level, such techniques also tend to be compute-intensive and not yet amenable to run in realtime. The different approaches—rapid direct manipulation on the one hand and high-level instruction-based editing on the other hand—recall long-standing arguments in the HCI community on the benefits of direct control vs. delegation ( Shneiderman and Maes 1997 ) . In this paper, we investigate if it is possible to unify the complementary strengths of real-time, immersive editing on the one hand, and generative AI-based approaches to high-level scene editing (with high latency) on the other hand under a common interaction framework. We believe that generative AI can enable new forms of interactions through natural language and could lower the barrier to entry for users with limited 3D modeling experience and could make it easier for users to express creative ideas, while making it faster or easier to prototype for experienced users. We introduce Dreamcrafter, a Virtual Reality 3D content generation and editing system assisted by generative AI. 1 1 1 dream-crafter.github.io The core idea behind Dreamcrafter is to use direct manipulation for spatial positioning and layout; and leverage generative AI for editing style and appearance of objects. Because generative AI edits are unlikely to run in real-time, Dreamcrafter introduces rapid proxy representations, e.g. using a 2D diffusion model to create a stand-in image for a longer-running 3D generative task. Dreamcrafter enables both 2D (image) and 3D output. Dreamcrafter makes three technical contributions: (1) Scene editing system with photo-realistic scene representations. We use radiance fields (3D gaussian splatting) instead of traditional mesh-based representations. (2) Modular architecture. This enables the system to continuously integrate state of art generative AI models and leverage both 2D and 3D proxy representations. (3) Flexibility in scene editing. A combination of voice prompts with natural language instructions and sculpting using primitives gives both general and advanced users extensive flexibility. We chose to make Dreamcrafter a VR immersive editor since it affords interactions such as voice input and direct manipulation which are more natural in VR and allow for spatial 3D input. Radiance fields are intrinsically realistic and immersive, and are well suited to be viewed in VR, making their creation within VR representative of a user’s experience. Radiance fields are being applied to a variety of applications such as in geospatial systems ( Cesium 2024 ) , gaming ( AI 2023 ) , social experiences ( Worlds 2020 ) , e-commerce ( Amazon 2024 ) , and education ( Niedermayr et al. 2024 ) . There is rising industry interest in systems with immersive radiance fields such as Meta Hyperscape ( AI 2024b ) , Varjo Teleport ( Varjo

Read the complete text