Canvas3D: Empowering Precise Spatial Control for Image Generation with Constraints from a 3D Virtual Canvas DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Do No
Runlin Duan, Yuzhao Chen, Rahul Jain, Yichen Hu, Jingyu Shi, Karthik Ramani · 2025
Read Canvas3D: Empowering Precise Spatial Control for Image Generation with Constraints from a 3D Virtual Canvas DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Do No by Runlin Duan, Yuzhao Chen, Rahul Jain, Yichen Hu, Jingyu Shi, Karthik Ramani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07135.txt.
Introduction
Runlin Duan and five colleagues give image generation a drafting table: a three-dimensional canvas where users place objects before the picture is made. Text prompts alone leave arrangement to chance. Precise layout, who stands where and how big, is what designers actually need.
Canvas3D turns a prompt into movable objects inside a 3D engine view. Users arrange the scene directly, and those placements become hard spatial constraints for the generator. In a scored comparison, it beat a baseline on control, interactivity, and overall feel. An open-ended study then let people use it freely. The authors argue spatial intent deserves its own interface, not longer prompts.
Catalog record
| CREATOR | Runlin Duan, Yuzhao Chen, Rahul Jain, Yichen Hu, Jingyu Shi, Karthik Ramani |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-07135
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Canvas3D: Empowering Precise Spatial Control for Image Generation with Constraints from a 3D Virtual Canvas DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper Runlin Duan, Yuzhao Chen, Rahul Jain, Yichen Hu, Jingyu Shi, Karthik Ramani Abstract . Generative AI (GenAI) has significantly advanced the ease and flexibility of image creation. However, it remains a challenge to precisely control spatial compositions, including object arrangement and scene conditions. To bridge this gap, we propose Canvas3D, an interactive system leveraging a 3D engine to enable precise spatial manipulation for image generation. Upon user prompt, Canvas3D automatically converts textual descriptions into interactive objects within a 3D engine-driven virtual canvas, empowering direct and precise spatial configuration. These user-defined arrangements generate explicit spatial constraints that guide generative models in accurately reflecting user intentions in the resulting images. We conducted a closed-end comparative study between Canvas3D and a baseline system. And an open-ended study to evaluate our system "in the wild". The result indicates that Canvas3D outperforms the baseline on spatial control, interactivity, and overall user experience. Figure 1. The workflow of controlling spatial composition using Canvas3D. A user starts by prompting the desired contents to the system (a). The system parses this prompt to create an interactive 3D environment, referred to as "canvas" (b). The user then interacts with the canvas to specify the desired composition of objects and environmental factors (c). In one case, the user first rearranges the placement of the boat and the lighthouse, and then changes the viewpoint. In another case, the user repositions the human pose and changes the lighting direction from left to right. The system translates the user-specified arrangement into explicit spatial constraints, referred to as "spatial conditions" for generative models (d). Finally, these conditions guide the generative model to produce images that accurately reflect the user’s spatial intents (e). 1. Introduction Generative AI (GenAI) has emerged as a transformative technique for image generation ( Rombach et al. 2022 ; Ramesh et al. 2021 ; Ramesh et al. 2022 ) . This new technology enables users to easily create images on demand and rapidly explore different compositions of image features, such as color ( Hou et al. 2024 ) , style ( Oh et al. 2024 ) , and object shapes ( Lin et al. 2024 ; Sarukkai et al. 2024 ) . However, there remains a challenge to precisely control the spatial composition of images ( Zhang et al. 2023a ) . For example, text-to-image generation models do not properly capture spatial constraints, such as object placement and the relationship between different objects ( Huang et al. 2025 ; Sun et al. 2023 ) . Imagine a scenario in which a user wants to generate a room with a particular arrangement of furniture and decor, then visualize it from various perspectives. To control this image, the user must be able to precisely describe their spatial intents, such as furniture placement, to the generative model. Prior research has explored various methods to collect user spatial intents, such as text ( Brade et al. 2023 ; Choi et al. 2024 ) , sketches ( Sarukkai et al. 2024 ; Lin et al. 2024 ) , and dragging ( Shi et al. 2024b ; Avrahami et al. 2024 ) . However, these approaches depend on low-dimensional modalities to describe high-dimensional spatial compositions, which lack precision or demand significant user effort. Specifically, text-based methods, while intuitive, are typically imprecise and require extensive trial-and-error processes ( Brade et al. 2023 ) . In addition, sketch-based methods can provide precise spatial guidance but require substantial sketching skills, spatial awareness, and visualization abilities from users ( Sarukkai et al. 2024 ) . Meanwhile, dragging-based methods ( Pan et al. 2023 ; Shin et al. 2024 ; Shi et al. 2024b ) , simplify spatial edits via direct 2D manipulation but lack depth manipulation capabilities ( Avrahami et al. 2024 ) , which result in objects failing to blend into new positions naturally. On the other hand, precisely generating user-defined spatial composition requires the generative model to accurately leverage user spatial constraints. Recent research introduces conditional generative models complying with precise 3D spatial constraints, e.g., depth maps ( Zhang et al. 2023a ) and human skeletons ( Li et al. 2024a ) . However, initializing such spatial constraints for conditional generative models poses significant challenges for out-of-lab use. For instance, current approaches commonly rely on sliders to specify the position and rotation of each object to initialize the depth condition ( Bhat et al. 2024 ; Eldesokey and Wonka 2024 ) . The slider-based interaction causes strain due to the attention switch ( Visser et al. 1999 ; Gopher et al. 2000 ) -users have to switch their attention back and forth between the control widgets and the object states, which is tedious and impractical for compositing multiple objects. Given the challenge with existing approaches, we aim to improve the alignment between user intent and generated image by developing an intuitive interaction mechanism that (1) accurately captures user spatial intents, and (2) seamlessly translates user intents into inputs that generative models can understand and execute. To address these considerations, we turn our attention to 3D engines ( Unity Technologies [n. d.] ) , which have proven highly precise in other domains for native spatial manipulation ( Qian et al. 2022 ; Autodesk 2025 ; Open Robotics 2025 ) . The 3D engine simulates the object and renders the environment in a virtual 3D space. It offers natural and intuitive spatial interaction by empowering users to directly manipulate objects using common input modalities ( Jankowski and Hachet 2013 ; Jankowski and Hachet 2013 ) , such as a mouse and keyboard ( Schultheis et al. 2012 ) . By adopting 3D engine-driven interactions, we can develop a system that allows precise spatial control by supporting direct spatial manipulations in a 3D environment. We propose Canvas3D, an image generation system with precise spatial control. When using Canvas3D, the user first provides a textual prompt describing desired content (Figure 1-a); the system parses this description into an interactive 3D environment, referred to as "canvas" (Figure 1-b). Users then directly manipulate the objects in the canvas to specify the desired spatial composition (Figure 1-c). The completed object arrangements automatically translate into explicit spatial constraints, referred to as "spatial conditions", for generative models (Figure 1-d). These conditions guide generative models to produce images that accurately reflect user intents (Figure 1-e). We highlight our contributions as follows: • A systematic workflow empowering users to generate images with precise spatial control. • An integrated pipeline that automatically translates textual prompts into an interactive 3D canvas. • A unified user interface combining spatial manipulation with run-time image generation. • Two user studies evaluating system usability and demonstrating enhanced spatial controllability compared to a baseline system. 2. Related