3D Primitives are a Spatial Language for VLMs

Junze Liu, Kun Qian, Florian Dubost, Kai Zhong · 2026

Read 3D Primitives are a Spatial Language for VLMs by Junze Liu, Kun Qian, Florian Dubost, Kai Zhong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12586.txt.

Introduction

Liu and ten colleagues show that 3D geometric primitives can serve as a spatial language for vision-language models. These models can generate code that rebuilds a 3D scene from cubes, spheres and cylinders with correct counts, classes and rough positions. Yet they fail simpler spatial questions on the same image. Their SpatialBabel benchmark tests fourteen models on primitive-based reconstruction across six scene-code languages, finding one model's detection F1 varies by up to 5.7 times across languages. A training-free Code-CoT strategy routes spatial reasoning through primitive code.

Catalog record

CREATORJunze Liu, Kun Qian, Florian Dubost, Kai Zhong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75061
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12586

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

3D Primitives are a Spatial Language for VLMs Junze Liu, Kun Qian, Florian Dubost, Kai Zhong Abstract Vision-language models (VLMs) exhibit a striking paradox: they can generate executable code that reconstructs a 3D scene from geometric primitives with correct object counts, classes, and approximate positions, yet the same models fail at simpler spatial tasks when directly answering natural-language questions on the same image. We show that 3D geometric primitives (cubes, spheres, cylinders, expressed in executable code) serve as a powerful intermediate representation for spatial understanding, and we exploit this insight through three contributions. First, we introduce SpatialBabel , a benchmark that evaluates fourteen VLMs on primitive-based 3D scene reconstruction across six scene-code languages (programming languages and declarative formats for expressing 3D primitive scenes), revealing that the same model’s object-detection F1 can vary by up to 5.7 × 5.7\times across scene-code languages. Second, we propose Code-CoT (Code Chain-of-Thought), a training-free inference strategy that routes spatial reasoning through primitive-based code generation. Code-CoT improves the SpatialBabel-QA-Score by up to + 6.4 +6.4 % on primitive scenes and lifts real-photo CV-Bench-3D accuracy by + 5.0 +5.0 % for VLMs with strong coding capabilities. Third, we propose S 3 -FT (Self-Supervised Spatial Fine-Tuning), which self-supervisedly distills primitive spatial knowledge into general visual reasoning by parsing the model’s own primitive-reconstruction code (generated in Three.js) into structured annotations and fine-tuning on the result, requiring no human labels and no teacher model . Training on primitive images alone, S 3 -FT improves Qwen3-VL-8B by + 4.6 +4.6 to + 8.6 +8.6 % on SpatialBabel-Primitive-QA, + 9.7 +9.7 % on CV-Bench-2D, and + 17 +17 % on HallusionBench; the recipe transfers across model families. These results establish geometric primitives in executable code as both a diagnostic tool for VLM spatial understanding and a transferable spatial vocabulary. We will release the benchmark, training data, checkpoints, and evaluation toolkit upon publication. Figure 1 : Geometric primitives as a spatial language for VLMs: the paradox and two approaches to exploit it. (A) The paradox: A VLM produces a faithful 3D primitive-reconstruction of the input image in code, yet fails simple spatial questions on the same scene (B) Code-CoT (Section 4 ) closes the gap at inference time : the model first reconstructs the scene as primitive-based executable code, then reasons over its own structured output. (C) S 3 -FT (Section 5 ) closes the gap at training time : the model’s own primitive-reconstruction outputs are deterministically parsed into structured spatial annotations, which serve as self-supervised labels for LoRA fine-tuning, requiring no human labels. Training on primitives alone yields gains on real-photo benchmarks. 1 Introduction Vision-language models (VLMs) possess more spatial knowledge than their standard outputs reveal. Show a VLM a scene of geometric primitives (a cube and a cylinder) and ask “how many objects?”; it may hallucinate, answering “three” or “four.” Yet ask the same model to write scene-reconstruction code (e.g., Three.js) from the same image, and it produces a program that places the correct number of primitives at roughly the correct 3D coordinates (Figure 1 A). The same model, given the same pixels, contradicts itself. Across both proprietary APIs and open-weights models, VLMs generate compilable, spatially faithful 3D primitive-reconstruction programs while simultaneously failing to count, localize, or compose spatial relations when directly answering natural-language questions. Geometric primitives are uniquely suited for diagnosing this gap: procedurally generated scenes have zero label noise, a no-occlusion constraint removes visibility ambiguity, and simple shapes eliminate object-recognition difficulty, so any failure on these scenes is purely a failure of spatial reasoning. Primitive-reconstruction code is also deterministically parseable into structured scene graphs, enabling automatic evaluation without human annotation. This paradox points to a task-routing failure : scene-code generation is a strict superset of direct QA, as every visible object must be classified, sized, positioned, and named in a single coherent program. So, a model that emits accurate scene code demonstrably possesses the spatial understanding needed to answer the simpler question, yet fails to transfer it to natural-language QA. We propose a methodology to test this hypothesis, quantify the phenomenon, and close the gap. Our investigation yields four progressive findings: Finding 1: A multi-scene-code-language benchmark surfaces the routing failure (Section 3 ). We introduce SpatialBabel , a benchmark that evaluates VLM spatial understanding through 3D scene reconstruction across six scene-code languages (programming languages and declarative formats for expressing 3D primitive scenes) and a complementary spatial-QA task. Reconstruction quality varies sharply with the scene-code language, and each model exhibits a distinct pattern. Finding 2: Primitive-based Code-CoT is a useful inference scaffold for capable VLMs (Section 4 ). We introduce Code-CoT (Code Chain-of-Thought): the model first reconstructs the scene as primitive-based executable code, then answers spatial questions conditioned on its own reconstruction. Code-CoT lifts accuracy of the spatial-QA task derived from primitive scenes for mid-tier frontier models and lifts real-photo CV-Bench-3D 24 accuracy for VLMs with strong coding capability. Finding 3: 3D geometric primitives serve as a compositional spatial vocabulary for VLMs (Section 4.3 ). The Code-CoT lifts on primitive scenes and CV-Bench-3D suggest that VLM code training has implicitly tokenized 3D space through geometric primitive shapes. Unlike coordinate-based scaffolds, 3D primitives encode shape, scale, and orientation in a single structured representation that aligns with how VLMs are trained to write 3D scene code. Finding 4: Self-supervised fine-tuning on primitive scenes distills spatial knowledge to general vision (Section 5 ). S 3 -FT (Self-Supervised Spatial Fine-Tuning) parses each VLM’s own Three.js primitive-reconstruction code into structured spatial annotations and fine-tunes on the resulting multi-task mixture, requiring no human labels and no teacher model . Training on primitive images alone yields gains on real-photo benchmarks; the recipe transfers across five base VLMs. We will release the benchmark, evaluation code, S 3 -FT training datasets, and checkpoint adapters upon publication. 2 Related Work Spatial understanding in VLMs. Prior work investigates VLM 3D reasoning through targeted training (SpatialVLM 5 , SpatialRGPT 8 , SpatialLadder 17 ) or single-image probing (Theory of Space 28 ); mechanistic analyses identify structural causes of VLM spatial limitations including vision-token-norm suppression of positional encoding 21 , 1D flattening of 2D features 1 , attention imbalance 7 , and rigid linear spatial bindings 14 . SpatialBabel complements these by evaluating through generative primitive-reconstruction code , a stricter task than qualitative QA that externalizes the model’s spatial representation into a parseable form. 3D scene generation and spatial-reasoning benchmarks. Scene-code generation methods (Scene Language 29 , SceneScript 3 , SceneCraft 12 , BlenderGym 11 , Design2Code 23 ) and spatial-reasoning benchmarks (SpatialBench 25 , VSI-Bench 26 , 3DSRBench 20 , the spatial slices of MMMU 27 / MathVista 18 ) each fix a single scene-code language or output channel, conflating spatial competence with format familiarity; code-generation benchmarks (HumanEval 6 , MBPP 2 , SWE-Bench 13 ) evaluate programming in isolation from spatial reasoning. SpatialBabel disentangles the two by holding spatial content constant while varying the scene-code language across six formats. Primitives for spatial reasoning. Concurrent work by 19 also uses “primitives” for VLM spatial reasoning, but with a different definition: DeepSeek interleaves 2D points and bounding boxes (in pixel coordinates) into the chain of thought as referencing markers, while SpatialBabel uses 3D geometric primitives as compositional scene elements with 3D position, rotation,

Read the complete text