Map2World: Segment Map Conditioned Text to 3D World Generation

Jaeyoung Chung * Suyoung Lee * † Jianfeng Xiang, Jiaolong Yang Kyoung Mu Lee · 2026

Read Map2World: Segment Map Conditioned Text to 3D World Generation by Jaeyoung Chung * Suyoung Lee * † Jianfeng Xiang, Jiaolong Yang Kyoung Mu Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00781.txt.

Introduction

Jaeyoung Chung and four co-authors introduce Map2World, a framework that builds 3D worlds from user-drawn segment maps paired with text. Earlier generators often forced grid layouts and let object scale drift across a scene. Map2World accepts maps of arbitrary shapes and scales, which aims to keep global scale consistent while giving users more control over expansive environments.

A detail enhancer then adds fine surface detail without breaking the overall scene structure, by using global information during generation. The pipeline leans on strong priors from asset generators, which helps it generalize even with limited scene training data. In experiments, the method beats prior approaches on user control, scale consistency, and content coherence.

Catalog record

CREATORJaeyoung Chung * Suyoung Lee * † Jianfeng Xiang, Jiaolong Yang Kyoung Mu Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.00781

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Map2World: Segment Map Conditioned Text to 3D World Generation Jaeyoung Chung * Suyoung Lee * † Jianfeng Xiang, Jiaolong Yang Kyoung Mu Lee Abstract 3D world generation is essential for applications such as immersive content creation or autonomous driving simulation. Recent advances in 3D world generation have shown promising results; however, these methods are constrained by grid layouts and suffer from inconsistencies in object scale throughout the entire world. In this work, we introduce a novel framework, Map2World, that first enables 3D world generation conditioned on user-defined segment maps of arbitrary shapes and scales, ensuring global-scale consistency and flexibility across expansive environments. To further enhance the quality, we propose a detail enhancer network that generates fine details of the world. The detail enhancer enables the addition of fine-grained details without compromising overall scene coherence by incorporating global structure information. We design the entire pipeline to leverage strong priors from asset generators, achieving robust generalization across diverse domains, even under limited training data for scene generation. Extensive experiments demonstrate that our method significantly outperforms existing approaches in user-controllability, scale consistency, and content coherence, enabling users to generate 3D worlds under more complex conditions. * * footnotetext: indicates equal contribution. † † footnotetext: works done during internship at Microsoft Research. Figure 1 : (Left) A generated sample of the center of the city, where the four edges consist of green forest. Our model takes as input a segment map with text prompts for each segment and produces a world that corresponds to the input segment map (violet box) with high quality (blue and orange boxes). (Right) Two examples of generated worlds in large-scale. 1 Introduction Three-dimensional (3D) world generation plays a pivotal role in a wide range of industrial applications, including immersive content creation, gaming, autonomous driving simulation, and virtual realities. While recent years have witnessed remarkable progress in 3D asset or object generation, extending these capabilities to the world scale remains challenging. The primary bottleneck is the lack of high-quality world-level datasets, which are far more difficult to construct than object-centric collections. Several works, such as BlockFusion [ 38 ] , NuiScene [ 18 ] , LT3SD [ 27 ] , WorldGrow [ 22 ] or SCube [ 30 ] , trained their own generator with existing datasets. However, these methods can only generate scenes within a limited domain, such as indoor or driving scenes, severely limiting their applicability. To circumvent the absence of world-scale data, researchers often resort to leveraging the power of pre-trained diffusion models. One approach combines image diffusion models with a depth estimator to lift generated 2D images into 3D; however, these pipelines suffer from view-dependent inconsistencies, yielding incomplete reconstructions. Another direction employs video diffusion models to generate scene-like sequences. Still, these methods struggle with 3D-consistency and are fundamentally constrained by the limited memory span of diffusion models. Recent 3D generation models, such as TRELLIS [ 39 ] and CLAY [ 45 ] , demonstrate the feasibility of high-quality, domain-agnostic 3D generation at an asset scale. Building on these foundations, several attempts have been made to scale asset generators to the world level. For instance, SynCity [ 9 ] divides the ground into grids and generates proper 3D assets for each tile using TRELLIS. Then, the asset boundaries are blended using an image inpainting module. Although these approaches effectively leverage the expressive power of 3D asset generators, the failure to model relationships among generated assets poses challenges for large-world generation. For example, contextual disconnectedness between adjacent assets and inconsistencies in object scale can make the overall scene appear less coherent and harmonious. In addition, the assets should be arranged in a grid-like manner, which is impractical in real scenarios, as district boundaries are irregularly shaped. In this paper, we present a novel text-conditioned 3D world generation framework called Map2World that builds on TRELLIS and explicitly addresses the limitations. To ensure global context and scale consistency, we introduce a multi-diffusion strategy applied within the structured latent space. By coordinating overlapping diffusion windows, our method not only preserves the latent prior of TRELLIS but also enables seamless connections that extend beyond the boundaries of individual cubes. This design naturally supports arbitrary resolution, as the scene can be generated progressively while maintaining coherence across local neighborhoods. Furthermore, our approach can flexibly incorporate semantic maps as conditions without additional training, enabling controllable, semantically aligned scene synthesis. The ability to balance local detail, global context, and semantic structure allows our method to generate globally coherent, high-resolution 3D scenes that surpass the limitations of existing modular approaches. Through extensive experiments, we demonstrate that our framework significantly improves both structural fidelity and perceptual realism, paving the way toward scalable and versatile 3D scene generation. Our contributions can be summarized as follows: • Flexible segment map conditioning: generates 3D worlds from any type of user-defined segment maps, not limited to grid-based layouts. • Consistent detail enhancement: adds fine details to the assets while preserving overall structure. • Domain-generalized world generation: leverages powerful asset generator priors to achieve robust generation across domains with limited data. 2 Related work 3D world generation. Compared to 3D asset generation, creating a world is considered a more complex task, as a world consists of multiple objects that are harmonized with each other. We classify the approaches to generate a 3D world into two categories: 3D reconstruction from generated rendered views, and direct explicit 3D generation. The former approach uses diffusion models to generate the images or videos, and reconstructs a 3D scene from the generated views. 2D image diffusion model-based approaches [ 13 , 5 , 43 , 42 , 21 , 34 ] lift the pixels of the initial image into 3D with a monocular depth estimator. The image is then outpainted, and the generated region is lifted and stitched to the original scene to expand the world. The video diffusion model-based approaches [ 36 , 47 , 25 , 40 ] generate a video that navigates a virtual environment and reconstructs the scene from the video frames. Yet, these methods cannot guarantee the 3D consistency since the diffusion models are not trained to be aware of such consistency. The other approach directly creates the scene with explicit 3D representations, achieving perfect 3D consistency. Some methods, such as BlockFusion [ 38 ] , NuiScene [ 18 ] , and LT3SD [ 27 ] , train the generator from scratch to learn the distribution of cubes that correspond to a scene part. However, they share two major drawbacks resulting from the limited availability of scene datasets. First, these models can only generate geometry, and the generated scene does not have textures. Second, the domain of the generation is limited to the dataset they used for training. SCube [ 30 ] and InfiniCube [ 26 ] add color estimation and use sparse-voxel-hierarchy [ 29 ] to improve global consistency, but they are also constrained to representing only driving scenes, which poses a significant limitation. Several works [ 9 , 48 ] leverage off-the-shelf 3D asset generators [ 39 , 29 , 23 , 4 , 45 ] and expand the scope of generation to 3D world. Although the quality of each asset can be prominent, the overall quality of the scene

Read the complete text