Geometrically Consistent Multi-View Scene Generation from Freehand Sketches

Ahmed Bourouis, Savas Ozkan, Andrea Maracani, Yi-Zhe Song, Mete Ozay · 2026

Read Geometrically Consistent Multi-View Scene Generation from Freehand Sketches by Ahmed Bourouis, Savas Ozkan, Andrea Maracani, Yi-Zhe Song, Mete Ozay (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14302.txt.

Introduction

Bourouis and four colleagues turn a single freehand sketch into geometrically consistent multi-view scenes. Sketches convey intent through abstract strokes while warping space against any consistent 3D reading, and no method attempts the task. Their framework answers with three parts: a curated dataset of about 9,000 sketch-to-multiview samples, camera-aware adapters, and a sparse correspondence loss from Structure-from-Motion reconstructions. All views emerge in one denoising pass without reference images or per-scene optimization, beating baselines by over 60 percent on realism and 23 percent on geometric consistency.

Catalog record

CREATORAhmed Bourouis, Savas Ozkan, Andrea Maracani, Yi-Zhe Song, Mete Ozay
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48854
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14302

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Geometrically Consistent Multi-View Scene Generation from Freehand Sketches Ahmed Bourouis, Savas Ozkan, Andrea Maracani, Yi-Zhe Song, Mete Ozay Abstract We tackle a new problem: generating geometrically consistent multi-view scenes from a single freehand sketch. Freehand sketches are the most geometrically impoverished input one could offer a multi-view generator. They convey scene intent through abstract strokes while introducing spatial distortions that actively conflict with any consistent 3D interpretation. No prior method attempts this; existing multi-view approaches require photographs or text, while sketch-to-3D methods need multiple views or costly per-scene optimisation. We address three compounding challenges; absent training data, the need for geometric reasoning from distorted 2D input, and cross-view consistency, through three mutually reinforcing contributions: (i) a curated dataset of ∼ \sim 9k sketch-to-multiview samples, constructed via an automated generation and filtering pipeline; (ii) Parallel Camera-Aware Attention Adapters (CA3) that inject geometric inductive biases into the video transformer; and (iii) a Sparse Correspondence Supervision Loss (CSL) derived from Structure-from-Motion reconstructions. Our framework synthesizes all views in a single denoising process without requiring reference images, iterative refinement, or per-scene optimization. Our approach significantly outperforms state-of-the-art two-stage baselines, improving realism (FID) by over 60% and geometric consistency (Corr-Acc) by 23%, while providing up to a 3.7 × \times inference speedup. 1 Introduction Consider a rough, freehand sketch of a street scene: a few strokes for a car, some wobbly lines representing buildings, a stick figure on the pavement. Despite this inherent abstraction, a human observer effortlessly infers a plausible 3D world, understanding that the car has an occluded side, the buildings extend beyond the visible corners and the figure could be seen from different perspectives. This rises the fundamental question: Can a generative model, conditioned solely on a single freehand sketch, synthesize a set of geometrically consistent novel views representative of a complete 3D scene? Sketch Generated Multi-View Output FS-COCO “a girl sitting on a horse” FS-COCO “a dog with hat sitting on the grass” FS-COCO “a lady sitting on a bench with dog” TU-Berlin penguin (unseen domain) InstantStyle girl with a cake on the table (unseen domain) Figure 1 : Sketch-to-Multi-View generation. Given a single freehand sketch and a text caption, our method generates geometrically consistent multi-view images spanning a full 360^{\circ} azimuth orbit at four different elevations. Top three rows: in-domain examples from FS-COCO [ 9 ] . Bottom two rows: zero-shot generalisation to unseen sketch domains (TU-Berlin [ 11 ] and InstantStyle [ 43 ] ). We define this task as Sketch-to-Multi-View ( S2MV ) generation as illustrated in Fig. 1 . This task is a surprisingly hard problem for current generative models, and one that has not been studied before. Freehand scene sketches are arguably the most geometrically impoverished conditioning signal one could provide to multiview generators. Unlike photographs, which provide dense appearance and implicit 3D cues (i.e., shading and perspective), or text prompts, which offer semantic flexibility without spatial constrains, freehand sketches are simultaneously spatially informative yet geometrically unreliable. They convey the intent of a scene layout while introducing abstraction, non-linear distortions, and stroke-level noise that often conflict with a consistent 3D interpretation. Existing multi-view diffusion methods do not directly address these inconsistencies. Image-conditioned approaches [ 29 , 40 , 31 , 41 , 44 , 12 , 59 ] typically begin with high-fidelity photographs that already resolve geometric ambiguity. Text-conditioned methods [ 41 ] operate in a domain where spatial imprecision is expected and tolerated. Conversely, existing Sketch-to-3D methods, such as Sketch2NeRF [ 7 ] and SketchDream [ 28 ] , are largely restricted to single-object reconstruction or require multiple sketch views and costly per-scene optimisation. To our knowledge, no prior work generates geometrically consistent multi-view outputs from a single, freehand scene sketch. Bridging this gap requires solving three compounding challenges. (i) Data: no sketch-to-multi-view dataset exists, and freehand sketches cannot be consistently paired with multi-view images through standard rendering pipelines. (ii) Geometric reasoning: generative models must infer coherent 3D structures from a 2D input that lacks strict projective alignment, demanding camera-aware generation mechanisms. (iii) Cross-view consistency: generating different views simultaneously requires the model to enforce correspondences between views, rather than simply conditioning on viewpoint independently. We address these challenges with the following contributions: • A Curation Pipeline for S2MV Data ( Section 3.2 ). We introduce a novel pipeline to bridge the domain gap between freehand sketches and multi-view images. By leveraging FS-COCO [ 9 ] , we synthesize photorealistic frontal views using a pretrained generative model [ 22 ] , verify sketch-image alignment through semantic segmentation matching [ 4 , 6 ] , and generate multiple views per scene [ 10 ] . The resulting dataset includes 9 222 curated samples with 33 views each, providing a rigorous foundation for S2MV training. • Camera-Aware Attention Adapters (CA3) ( Section 3.3 ). We propose a lightweight camera attention adapters that inject Projective Rotary Position Encoding (PRoPE) [ 26 ] into a pretrained video diffusion transformer [ 45 ] . This mechanism adds only 2.7% additional parameters while enabling attention layers to explicitly account for relative projective geometry between views. • Correspondance Supervision Loss (CSL) ( Section 3.4 ). We introduce a sparse InfoNCE-based correspondence loss applied to the adapter’s latent projections to maintain geometric consistency between cross-views. Using positive pairs extracted through Structure-from-Motion [ 39 ] ,this regularization enforces cross-view identity and merges distinct viewpoints into a unified latent representation. The components of our framework are mutually reinforcing, creating a cohesive system for S2MV synthesis. While our curated dataset provides the foundational sketch–multiview pairs previously missing from the literature, our CA3 provides the necessary geometric inductive bias and the capacity to reason about viewpoints. Last but not least, our CSL provides the explicit supervisory signal that teaches the model which spatial locations across views should attend to each other, turning geometric capacity into geometric consistency. Removing any one of these components significantly degrades the final performance. Crucially, our method generates all multi-views jointly in a single end-to-end denoising process directly from a sketch, bypassing any intermediate photo-generation stage, requiring no reference photograph, no per-scene optimisation, no iterative refinement. To the best of our knowledge, this is the first method to produce geometrically coherent, photorealistic multi-view scenes from freehand sketch input. 2 Related Work Multi-View Generation and Camera Conditioning. Diffusion models have been widely applied to multi-view image generation. Early methods fine-tune image diffusion models to synthesise novel views from a single photograph: Zero-1-to-3 [ 29 ] conditions on relative camera pose, and follow-up works [ 40 , 31 , 41 , 25 , 32 ] improve consistency by generating multiple views jointly through synchronised diffusion or multi-view attention. A parallel line of work repurposes video diffusion models, treating the temporal axis as a viewpoint axis, for object-centric orbiting [ 44 , 8 ] or scene-level [ 12 ] generation. More recent DiT-based architectures push the scale further: Hunyuan3D 2.0 [ 59 ] trains a 2B-parameter shape-generation DiT, TRELLIS [ 51 ] employs rectified flow transformers over structured 3D latents, and See3D [ 34 ] learns 3D priors from 16M video clips without pose annotations. How camera geometry is communicated to the generative model is a critical design axis. Absolute conditioning methods [ 29 , 13 ,

Read the complete text