NuiWorld: Exploring a Scalable Framework for End-to-End Controllable World Generation Conference: ; ;

Han-Hung Lee, Cheng-Yu Yang, Yu-Lun Liu, Angel X. Chang · 2026

Read NuiWorld: Exploring a Scalable Framework for End-to-End Controllable World Generation Conference: ; ; by Han-Hung Lee, Cheng-Yu Yang, Yu-Lun Liu, Angel X. Chang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19048.txt.

Introduction

Han-Hung Lee and three co-authors build a system that creates whole virtual worlds users can steer. World generation feeds games, simulation, and robotics, but three walls block it: control, scale, and speed. Past end to end models had too little data, while object by object methods fixed the resolution and lost detail on big scenes.

Their framework, NuiWorld, aims to clear all three walls at once. Users should be able to shape the world, grow it large, and get it fast. Worlds that obey their makers would serve game design, training sims, and robot practice grounds alike.

Catalog record

CREATORHan-Hung Lee, Cheng-Yu Yang, Yu-Lun Liu, Angel X. Chang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-19048

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NuiWorld: Exploring a Scalable Framework for End-to-End Controllable World Generation Conference: ; ; Han-Hung Lee, Cheng-Yu Yang, Yu-Lun Liu, Angel X. Chang Abstract . World generation is a fundamental capability for applications like video games, simulation, and robotics. However, existing approaches face three main obstacles: controllability, scalability, and efficiency. End-to-end scene generation models have been limited by data scarcity. While object-centric generation approaches rely on fixed resolution representations, degrading fidelity for larger scenes. Training-free approaches, while flexible, are often slow and computationally expensive at inference time. We present NuiWorld, a framework that attempts to address these challenges. To overcome data scarcity, we propose a generative bootstrapping strategy that starts from a few input images. Leveraging recent 3D reconstruction and expandable scene generation techniques, we synthesize scenes of varying sizes and layouts, producing enough data to train an end-to-end model. Furthermore, our framework enables controllability through pseudo sketch labels, and demonstrates a degree of generalization to previously unseen sketches. Our approach represents scenes as a collection of variable scene chunks, which are compressed into a flattened vector-set representation. This significantly reduces the token length for large scenes, enabling consistent geometric fidelity across scenes sizes while improving training and inference efficiency. Figure 1 . NuiWorld proposes a data pipeline for constructing open-domain , variable-sized scenes with pseudo sketch pairs to train controllable world generation models. Furthermore, our representation scales with scene size while maintaining fidelity across scenes of varying sizes. 1. Introduction The ability to generate virtual worlds at a click of a button has wide-ranging applications. For artists, it could enable rapid visualization of environments for concept design. In robotics, generating diverse environments for training can improve generalization. Similar benefits can extend to VR, film and other industries. As a result, fast and open-domain world generation is increasingly important, and we believe will be best achieved through end-to-end models. In this paper, we define world generation as the ability to produce large-scale, open-domain scenes. We argue that addressing the two core challenges of scalability and data scarcity is essential to achieving this and enabling world generation for broader applicability. Figure 2 . Object generators like Trellis 2 do not adapt to scene scale or aspect ratio, causing such scenes to be represented with less voxels and degraded fidelity while our method scales with scene sizes and maintains fidelity. With the rise of LLMs and agent-based systems used to tackle tasks with limited training data, these models have also been explored for large-scale scene generation. Several works ( 32 ; 25 ; 13 ; 33 ) employ LLMs to aid the scene generation process. However, this reliance introduces additional latency time and potential costs. Additionally, it remains unclear whether LLMs can reliably perform spatial reasoning for scene layout generation or generalize to open-domain scene layouts. Our approach instead focuses on synthesizing a sufficiently large training dataset of scenes for world generation as seen in Figure 3 . We propose a generative bootstrapping pipeline that begins with a small set of scenario images produced using text-to-image models. These images are reconstructed into 3D scenes with Trellis 2 ( 36 ) , which are used to train NuiScene ( 17 ) to generate varying scene layouts and scales. These scenes are converted into pseudo sketches and used to train a controllable world generation model. This approach eliminates the need for agents or complex inference pipelines, and allows for the generation of new scenarios using a few input images. Recent 3D object generation methods achieve impressive high quality results by utilizing the sparse voxel representation ( 37 ; 34 ; 11 ; 20 ; 15 ; 36 ) . However, these methods are constrained by the overall voxel grid resolution. In large scenes, the geometry becomes increasingly blurry and loses fine details due to the upper bound on the total number of voxels. Which is further exacerbated by uniformly distributed voxel grids, which allocate the same spatial resolution to empty and occupied regions alike limiting their use to large-scale scene generation as seen in Figure 2 . In contrast, we represent the scene as a variable-length sequence of scene chunks. Each scene chunk is encoded as a flattened vector set ( 39 ) , as illustrated in Figure 1 , resulting in a compact token sequence whose length corresponds to the number of scene chunks. Importantly, this flattened representation shifts part of the computation from the token length, which has quadratic memory growth, to the model width, which scales linearly. Because all the vector sets have a homogeneous size, they can be directly reshaped into token channels, enabling efficient scaling across scenes of different sizes while maintaining consistent fidelity. To circumvent the resolution limit of object generators, another line of work ( 7 ; 41 ; 3 ; 38 ) decomposes large scenes into chunks and leverages Trellis ( 37 ) to generate each chunk in a training-free manner. These approaches require multiple runs of Trellis sequentially or in parallel, leading to slow inference speeds and/or heavy memory usage. Moreover, some methods ( 7 ) rely on additional resource-intensive modules, further increasing computational overhead. By comparison, our method is trained end-to-end and diffuses the entire scene at once during inference. This results in a significantly simpler inference pipeline and more efficient use of computational resources. To summarize, our contributions are as follows: Generative Boostrapping Strategy. Our data generation pipeline combines a 3D reconstruction model with an expandable scene generation model, enabling the creation of scenes with various sizes and layouts from only a few input images of the target scenario. Paired with pseudo-sketches of these scenes, this pipeline supports end-to-end training of a controllable world model alleviating data scarcity. Scalable Scene Representation. We represent scenes as a variable-length sequence of scene chunk tokens, where each chunk is encoded as a flattened vector set. This yields a compact and efficient representation that scales with scene size while preserving consistent geometric fidelity. Efficient End-to-End System. Our model is trained end-to-end to generate complete scenes at once, eliminating the need for complex training-free pipelines or agent-based systems that may incur additional computational overhead. Figure 3 . Our framework begins with generative bootstrapping, shown on the left. Using Nano Banana to generate images and Trellis 2 to reconstruct 3D scenes. NuiScene is then trained on these scenes to produce new scenes with varying size and layouts. Finally, using these scenes and their pseudo sketches we train our variable-length sketch-to-world model on the right. 2. Related Work 2.1. Unbounded 3D Scene Generation We review methods that perform expandable generation of arbitrarily large 3D scenes directly in the 3D domain. PDD ( 22 ) uses a multi-scale pyramid diffusion framework to generate urban scenes. While several works ( 18 ; 35 ; 27 ) follow the latent diffusion paradigm, learning to generate chunks through diffusion in the latent space. Expandable generation is enabled with RePaint ( 26 ) by conditioning on overlapping regions, requiring additional diffusion steps. SemCity ( 18 ) and BlockFusion ( 35 ) use triplane representations for urban and indoor scenes, respectively, while LT3SD ( 27 ) uses hierarchical feature grids for modeling finer indoor details. NuiScene ( 17 ) introduces a vector set representation for outdoor scenes

Read the complete text