Aligning Text, Images, and 3D Structure Token-by-Token

Aadarsh Sahoo, Vansh Tibrewal, Georgia Gkioxari · 2025

Read Aligning Text, Images, and 3D Structure Token-by-Token by Aadarsh Sahoo, Vansh Tibrewal, Georgia Gkioxari (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08002.txt.

Introduction

Aadarsh Sahoo and co-authors teach one model to speak of the world in words, pictures, and 3D structure using a single token language. Their 2025 paper builds a unified autoregressive framework where complex 3D objects are tokenized so a language model can reason over scenes the way it reasons over sentences. Designers and robots both need machines that understand space structurally, not just visually.

The team publishes a detailed cookbook of design choices: how to represent the data, which training objectives each modality needs, and where naive unification fails. The model is tested on rendering, recognition, instruction following, and question answering across synthetic and real datasets. It can rebuild complete scenes of complex objects from a single image. The broader claim is that 3D belongs inside the same modeling paradigm as text and images, not in a specialist silo. Token by token, the modalities begin to share one grammar.

Catalog record

CREATORAadarsh Sahoo, Vansh Tibrewal, Georgia Gkioxari
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-08002

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Aligning Text, Images, and 3D Structure Token-by-Token Aadarsh Sahoo, Vansh Tibrewal, Georgia Gkioxari Abstract Creating machines capable of understanding the world in 3D is essential in assisting designers that build and edit 3D environments and robots navigating and interacting within a three-dimensional space. Inspired by advances in language and image modeling, we investigate the potential of autoregressive models for a new modality: structured 3D scenes. To this end, we propose a unified LLM framework that aligns language, images, and 3D scenes and provide a detailed “cookbook” outlining critical design choices for achieving optimal training and performance addressing key questions related to data representation, modality-specific objectives, and more. We evaluate performance across four core 3D tasks – rendering, recognition, instruction-following, and question-answering – and four 3D datasets, synthetic and real-world. We extend our approach to reconstruct complex 3D object shapes by enriching our 3D modality with quantized shape encodings, and show our model’s effectiveness on real-world 3D object recognition tasks. Project webpage: https://glab-caltech.github.io/kyvo/ . Figure 1 : Kyvo: a decoder-only transformer aligns a structured 3D modality with language and vision. This 3D modality represents scenes as lists of objects, each defined by its 3D shape, type, 3D position, pose and size parameters. Kyvo unifies the token space of images, text, and 3D to enable a variety of complex visual 3D tasks. 1 Introduction Large language models (LLMs) that fuse text and images have unlocked unprecedented visual capabilities, such as visual captioning and text-guided image generation. Inspired by this success, we explore a third modality – structured 3D scenes – and show how aligning it with text and images allows LLMs to tackle a new suite of visual tasks in 3D, such as 3D reconstruction from a single image and 3D-conditioned image generation. Our structured 3D modality encodes a scene as a list of its objects, where every object is specified by its 3D shape ( e.g. , 3D mesh, 3DGS), type, 3D position, 3D size and 3D pose, shown in Fig. 1 . This modality captures aspects of the physical world that are not directly conveyed through language or images alone. Additionally, it slots naturally into the unified token space shared by language and vision through an object-by-object tokenization scheme that integrates seamlessly with image and text tokens, so any modality can serve as input or output. As a result, it supports a broad range of tasks in 3D, such as image generation conditioned on the 3D scene structure (3D → image), predicting 3D objects, their shapes and locations from a single image (image → 3D), and language-guided object-centric 3D editing (3D + image + text → 3D + image) – all within the same model design – shown in Fig. 2 . These capabilities can reshape workflows. Robots can parse an image into a 3D scene composed of 3D objects, while designers can create complex scenes of objects through language in one forward pass of an LLM instead of wrestling with hard-to-use software like Blender. But, how does one design and train an LLM that aligns this structured 3D modality with image and text? While there is extensive work on language-only achiam2023gpt ; grattafiori2024llama3herdmodels ; team2023gemini or vision-and-language models tong2024cambrian ; laurenccon2024matters ; beyer2024paligemma , there is limited work on training with an additional structured 3D modality. We explore the vast design space and evaluate the impact of design choices in architecture, training strategies, data representation, modality-specific objectives, and more. Our testbed consists of four core 3D tasks – image generation, recognition, instruction-following, and question-answering – and four challenging datasets, both synthetic and real-world. Through an extensive “cookbook” of rigorously validated empirical findings we hope to provide guidelines to design 3D-aligned multi-modal LLMs. We then extend our findings to demonstrate the effectiveness of our unified LLM framework, Kyvo (Greek for “3D cube"), to single-view 3D reconstruction of complex object geometries via a tokenization technique for 3D shape encodings. Finally, we apply our model to real-world domains of indoor and outdoor scenes for the task of image-based 3D object recognition, where we show that our model can outperform prior vision specialists designed for the task. 2 Related Work LLMs. The success of LLMs is attributed to the scalability of transformers vaswani2017attention taking the form of encoder-decoder tay2022ul2 ; chung2024scaling and most popular decoder-only achiam2023gpt ; grattafiori2024llama3herdmodels ; team2023gemini models. LLMs showcase the effectiveness of autoregressive formulations in achieving task generalization, a finding we extend to 3D structured representations. Vision-Language Models (VLMs). Early VLMs radford2021learning ; jia2021scaling ; alayrac2022flamingo pioneered learning from image-text pairs, focusing on zero-shot classification, cross-modal retrieval, and question-answering. With the advent of LLMs, modern VLMs liu2024visual ; chen2024internvl ; lu2024deepseek ; ma2024janusflow ; wang2024qwen2 ; wu2024deepseek expand these capabilities, leveraging internet-scale training data and improved alignment strategies. Several works laurenccon2024matters ; beyer2024paligemma ; tong2024cambrian conduct vision-centric analysis and investigate key design choices for training VLMs. However, they limit images only as input. Others team2024chameleon ; zhou2024transfusion ; tong2024metamorph explore early fusion and diffusion for images along with text to support image generation. LLMs & 3D. Prior work integrates LLMs with various 3D formats for tasks like VQA and semantic grounding ma2024llms . Some ma2024spatialpin ; cheng2024spatialrgpt ; liao2024reasoning extend VLMs to 3D reasoning via depth-aware image queries. Others hong20233d ; tang2024minigpt ; xu2025pointllm ; ma2024find ; marsili2025visual ; kerr2023lerf ; shen2023distilled embed 3D data ( e.g. , point clouds, NeRFs) with CLIP-style features to support free-form QA. Specifically, 3D-LLM hong20233d inputs holistic 3D scene point clouds (obtained from scanning techniques or structure-from-motion) and trains a language model for the task of language captioning and QA. In contrast, our model, via its structured 3D modality, decomposes scenes into 3D objects, enabling direct alignment with language and vision. This allows broader tasks like 3D shape and pose prediction from a single image and image generation from object-centric 3D scene input. SceneScript avetisyan2025scenescript uses an autoregressive decoder over videos and holistic scene point clouds to predict 3D bounding boxes. In contrast, our method focuses on aligning images with structured 3D scene and object representations. Figure 2 : 3D task examples with Kyvo’s unified autoregressive framework using a structured 3D modality. (1) 3D shape and scene reconstruction: From a single input image, Kyvo reconstructs individual objects with accurate geometry and spatial relationships. (2) 3D object recognition: Given an input image, Kyvo identifies objects and predicts their 3D positions in real-world scenes. (3) Shape and scene rendering: Kyvo generates semantically consistent images from structured 3D scene inputs. (4) Instruction-Following: Given an image, 3D scene and text instruction, Kyvo produces coherent modifications to both image and the 3D representation. 3 Building our model token-by-token We present our approach, Kyvo, that aligns the 3D modality with text and images. The result is a unified multimodal LLM framework than can perform a range of visual 3D tasks, as shown in Fig. 2 . Sec. 3.1 outlines our experimental setup, including tasks and datasets. Sec. 3.2 presents a bottom-up analysis of key design choices – covering data representation, sequence design, and scaling – and addresses challenges in modality-specific tokenization and optimization. We summarize empirical insights and adopt optimal configurations for each subsequent experiment. Finally, we explore generalization to complex shapes (Sec. 3.3 ), 3D shape representations (Sec.

Read the complete text