SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoning
Byungwoo Jeon, Dongyoung Kim ⋆ \star, Huiwon Jang, Insoo Kim, Jinwoo Shin · 2026
Read SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoning by Byungwoo Jeon, Dongyoung Kim ⋆ \star, Huiwon Jang, Insoo Kim, Jinwoo Shin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22057.txt.
Introduction
Jeon and co-authors present SpatialBoost, a framework that teaches vision encoders about 3D space. They often miss how objects sit in real 3D scenes. SpatialBoost turns dense 3D spatial facts from an image into plain language. A large language model then feeds that knowledge back into the encoder. Training uses multi-turn chain-of-thought reasoning to build spatial understanding step by step. Applied to DINOv3, the authors report a gain on ADE20K from 55.9 to 59.7 mIoU. That score sets a new state of the art on the benchmark.
Catalog record
| CREATOR | Byungwoo Jeon, Dongyoung Kim ⋆ \star, Huiwon Jang, Insoo Kim, Jinwoo Shin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72020 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.22057
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoning Byungwoo Jeon, Dongyoung Kim ⋆ \star, Huiwon Jang, Insoo Kim, Jinwoo Shin Abstract Despite the remarkable success of large-scale pre-trained image representation models (i.e., vision encoders) across various vision tasks, they are predominantly trained on 2D image data and therefore often fail to capture 3D spatial relationships between objects and backgrounds in the real world, constraining their effectiveness in many downstream applications. To address this, we propose SpatialBoost, a scalable framework that enhances the spatial awareness of existing pre-trained vision encoders by injecting 3D spatial knowledge expressed in linguistic descriptions. The core idea involves converting dense 3D spatial information from 2D images into linguistic expressions, which is then used to inject such spatial knowledge into vision encoders through a Large Language Model (LLM). To this end, we adopt a multi-turn Chain-of-Thought (CoT) reasoning process that progressively incorporates dense spatial knowledge and builds hierarchical spatial understanding. To validate effectiveness, we adapt SpatialBoost to state-of-the-art vision encoders such as DINOv3, and evaluate its performance gains on a wide range of benchmarks requiring both 3D perception and general vision abilities. For instance, SpatialBoost improves DINOv3 performance from 55.9 to 59.7 mIoU on ADE20K, achieving state-of-the-art performance with 3.8% gain over the pre-trained DINOv3. Project page . 1 Introduction Pre-trained image representation models [ 38 , 26 , 15 , 27 , 52 , 5 ] have shown remarkable success in various downstream tasks, such as image classification [ 50 , 22 ] , semantic segmentation [ 53 , 106 ] , monocular depth prediction [ 76 , 32 ] , and vision-language understanding [ 3 , 43 ] . The core idea behind these successes is extracting transferrable representation from large-scale image datasets such as ImageNet [ 25 ] , enabling the model to understand semantic information within images that is significantly useful for various downstream tasks. Despite their success, these models are predominantly trained on 2D images and hence face a fundamental challenge in acquiring 3D spatial awareness capabilities. Consequently, large vision language models struggle to discern 3D spatial relationships between objects in images [ 54 , 30 , 86 , 19 ] , and demonstrate sub-optimal performance in vision-based robotic control tasks compared to approaches that directly utilize 3D information [ 99 , 48 , 104 ] . To address these limitations, naïve approaches are to train vision models on multi-view images that inherently encode spatial information [ 101 , 87 , 12 ] . While these approaches have shown promise in robot control tasks [ 73 , 74 ] , their broader applicability remains constrained by the need to use carefully curated data [ 97 ] or obtain multi-view datasets from simulation environments [ 72 ] , creating significant limitations for scaling up these approaches. These challenges highlight the need for a novel framework that enables effective learning of 3D information with substantially less data. Figure 1 : Overview of SpatialBoost. We enhance spatial and geometric understanding of pre-trained vision encoders by leveraging language-guided spatial reasoning. SpatialBoost consists of (a) spatial knowledge extraction through depth estimation, 3D reconstruction, segmentation, and region captioning, (b) converting spatial knowledge into multi-turn spatial reasoning from pixel to scene levels, and (c) building a spatial-aware vision encoder with LLM using generated data in (b). To address this problem, we note that vision models specialized for individual tasks are able to infer object positions and point depths from standard 2D images. These extracted cues make it possible to extend spatial information by modeling geometric relationships between objects in a scene. We hypothesize that such spatial information can be systematically converted into explicit representations by leveraging language. Moreover, since language naturally composes information in a sequential and structured form, this property allows the construction of labels that capture dense spatial relationships within a scene. Importantly, several recent works have shown that language can serve as a scalable supervision signal for learning visual representations [ 29 , 46 , 11 ] . While these approaches primarily focus on semantic understanding or object localization, they highlight the potential of linguistic descriptions as scalable supervision for training vision encoders. Based on these insights, we introduce SpatialBoost, a training framework that enhances the spatial understanding of pre-trained vision encoders by leveraging language-guided reasoning (see Figure 1 ). We inject linguistically described spatial knowledge through decoder-based fine-tuning with Large Language Models (LLMs), enabling the framework to process large-scale textual descriptions in a single pass while handling both single-view and multi-view images. In particular, to leverage this knowledge without forgetting the existing knowledge, we incorporate additional learnable parameters ( i.e. , dual-channel attention module) into the vision encoder and train only them while freezing the existing parameters. Furthermore, to incorporate dense spatial information in a structured manner, we present a multi-turn visual spatial reasoning approach that builds hierarchical spatial understanding through pixel-level, object-level, and scene-level sub-questions and answers. To validate the effectiveness of our method, we apply SpatialBoost to state-of-the-art image encoders, including DINOv3 [ 77 ] and SigLIPv2 [ 82 ] , and evaluate them across a diverse set of vision tasks: monocular depth estimation, semantic segmentation, 3D scene understanding, vision-based robotic control, image classification, and image retrieval. Our experiment first shows that SpatialBoost consistently improves performance on tasks requiring 3D spatial knowledge. For example, on the 3D scene understanding task, SpatialBoost improves DINOv3 by 3.5%p (51.4% \rightarrow 54.9%) on the SQA3D task from Lexicon3D Benchmark [ 60 ] . In addition, on depth estimation tasks, SpatialBoost improves SigLIPv2 from an RMSE score of 0.51 to 0.39 on NYUd linear probing. Moreover, we show that SpatialBoost even improves the performance of the vision encoders across all benchmarks, notably in image classification: SpatialBoost improves ImageNet linear probing performance of DINOv3 from 88.4% to 90.2%. In addition, DINOv3 with SpatialBoost achieves state-of-the-art performance across all evaluated tasks. 2 Related Work 2.1 Self-supervised Learning for Image Representation In earlier years, most approaches relied on supervised learning with large-scale labeled datasets to train models [ 25 , 78 , 80 , 39 ] . However, the dependence on annotated data introduced scalability challenges due to label expense. To address this, self-supervised learning (SSL) has emerged as a dominant paradigm, leveraging unlabeled data to learn image representations. Contrastive learning methods, including SimCLRv2 [ 16 ] , MoCov3 [ 17 ] , DINOv2 [ 64 ] , and iBOT [ 107 ] , are trained to distinguish between representations of augmented views of the same image and those of different images. Concurrently, mask prediction approaches such as BEiT [ 8 ] and MAE [ 37 ] , learn representations by reconstructing masked portions of input images. While these methods excel at capturing rich semantic features within 2D images, they lack mechanisms to effectively encode 3D spatial knowledge. On the other hand, we overcome this limitation by enhancing image representations through a novel method that injects 3D spatial knowledge by utilizing language decoding. 2.2 Multi-modal Learning for Image Representation The increasing prominence of multi-modal tasks has catalyzed the development of vision-language models that jointly represent visual and textual information. These models typically employ weakly supervised learning by leveraging text caption. Contrastive learning schemes, e.g . , CLIP [ 66 ] , SigLIP