TinyGiantVLM: A Lightweight Vision-Language Architecture for Spatial Reasoning under Resource Constraints

Vinh-Thuan Ly, Hoang M. Truong, Xuan-Huong Nguyen · 2025

Read TinyGiantVLM: A Lightweight Vision-Language Architecture for Spatial Reasoning under Resource Constraints by Vinh-Thuan Ly, Hoang M. Truong, Xuan-Huong Nguyen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17595.txt.

Introduction

Vinh-Thuan Ly, Hoang Truong, and Xuan-Huong Nguyen present a 2025 compact vision-language model built for spatial reasoning on a tight budget. Warehouse scenes demand a grasp of layout, object placement, and depth that big models often miss. Their two-stage design reads both whole-scene and region views from color and depth images through ready-made vision backbones.

A mixture-of-experts mixer blends those views on the fly, suiting each question type. Training runs in two phases: free-form answers first to grow reasoning, then tidy answers for scoring. The base model, at sixty-four million parameters, placed fifth in its challenge track. A larger eighty-million variant with a wider mixer did better still. The authors aim the design at real sites where compute is scarce.

Catalog record

CREATORVinh-Thuan Ly, Hoang M. Truong, Xuan-Huong Nguyen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17595

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TinyGiantVLM: A Lightweight Vision-Language Architecture for Spatial Reasoning under Resource Constraints Vinh-Thuan Ly, Hoang M. Truong, Xuan-Huong Nguyen Abstract Reasoning about fine-grained spatial relationships in warehouse-scale environments poses a significant challenge for existing vision-language models (VLMs), which often struggle to comprehend 3D layouts, object arrangements, and multimodal cues in real-world industrial settings. In this paper, we present TinyGiantVLM, a lightweight and modular two-stage framework designed for physical spatial reasoning, distinguishing itself from traditional geographic reasoning in complex logistics scenes. Our approach encodes both global and region-level features from RGB and depth modalities using pretrained visual backbones. To effectively handle the complexity of high-modality inputs and diverse question types, we incorporate a Mixture-of-Experts (MoE) fusion module, which dynamically combines spatial representations to support downstream reasoning tasks and improve convergence. Training is conducted in a two-phase strategy: the first phase focuses on generating free-form answers to enhance spatial reasoning ability, while the second phase uses normalized answers for evaluation. Evaluated on Track 3 of the AI City Challenge 2025, our 64M-parameter base model achieved 5th place on the leaderboard with a score of 66.8861, demonstrating strong performance in bridging visual perception and spatial understanding in industrial environments. We further present an 80M-parameter variant with expanded MoE capacity, which demonstrates improved performance on spatial reasoning tasks. 1 Introduction Understanding fine-grained spatial relationships in industrial environments presents a significant challenge for artificial intelligence. While AI systems have achieved impressive performance in image-based recognition and short-form spatial tasks, they often struggle to reason about 3D object layouts, dimensions, and spatial relations in real-world, logistics-scale environments. Prior research in visual question answering and 3D scene understanding has largely focused on synthetic datasets or domestic scenes [ 8 , 22 , 21 ] , leaving a substantial gap in industrial settings such as warehouses and logistics hubs. Figure 1 : An example of multiple-choice question (MCQ) in the dataset The AI City Challenge [ 17 ] Track 3: Warehouse Spatial Intelligence addresses this gap using the PhysicalAI-Spatial-Intelligence-Warehouse dataset for warehouse-scale 3D scene understanding through natural language questions. The challenge encompasses four distinct types of spatial reasoning tasks: distance estimation, object counting, multiple-choice questions (MCQ) for spatial grounding, and left/right spatial relation queries. These tasks demand AI systems capable of integrating visual perception, geometric reasoning, and language comprehension while operating under resource constraints typical of industrial deployment scenarios. Figure 1 illustrates an example of a multiple-choice question (MCQ) from the dataset. To address these challenges, we propose TinyGiantVLM , a lightweight multimodal encoder-decoder architecture designed for precise spatial reasoning under resource constraints. Our approach processes both RGB and depth information at global and region levels through a dual-branch design, integrating cross-attention fusion mechanisms with a Mixture-of-Experts (MoE) framework based on FuseMoE [ 4 ] . The architecture employs specialized expert networks for each of the four question types, enabling task-specific reasoning while maintaining computational efficiency through sparse expert routing. Unlike traditional geographic reasoning models that rely on symbolic or coordinate-based knowledge (e.g., ”Is Beijing north of Shanghai”), our focus is on physical spatial reasoning through visual perception, learning relative object positioning and geometric relationships (e.g., ”Which pallet is furthest right”) from RGB-D inputs. This distinction emphasizes the need for models that ground spatial language in perceptual understanding rather than abstract knowledge bases. Our key contributions are as follows: • We introduce TinyGiantVLM, a resource-efficient vision-language architecture that achieves strong spatial reasoning performance while maintaining low computational overhead suitable for industrial deployment. • We propose a dual-branch multimodal feature extraction approach that effectively combines RGB and depth information at both global and region levels. • Through extensive experiments, we validate our approach on the AI City Challenge [ 17 ] Track 3, achieving competitive results with significantly reduced computational requirements. In Section 2 , we provide a brief review of existing methods related to our problem and solution approach. We then present the details of our proposed framework in Section 3 . The experimental setup and evaluation results are discussed in Section 4 , followed by concluding remarks, future directions and limitations in Section 5 . 2 Related Work Spatial Vision Language Models. Recent works have focused on addressing spatial problems through the reasoning capabilities of LLMs, leading to the development of various 3D vision-language models with distinct input modalities and alignment strategies. Image-based methods derive 3D understanding from 2D images [ 27 , 24 , 10 , 1 ] , with approaches like SpatialVLM [ 1 ] enhancing spatial reasoning through Internet-scale 3D spatial reasoning data with quantitative metric capabilities. Point cloud-based approaches employ different alignment strategies [ 19 , 6 , 11 , 16 , 26 , 18 , 28 , 23 ] , ranging from direct alignment methods like PointLLM [ 19 ] to task-specific methods like ScanReason [ 26 ] for reasoning grounding. Hybrid approaches combine multiple modalities [ 3 , 25 , 5 ] , such as Point-bind [ 3 ] which integrates point clouds and images for cross-modal understanding. More recently, SpatialRGPT [ 2 ] introduces regional representation learning from 3D scene graphs with flexible depth integration modules, demonstrating strong generalization in spatial reasoning tasks and robotic applications. However, most of these methods heavily rely on large-scale language models (LLMs), making them computationally expensive and less suitable for low-resource environments. In contrast, our work focuses on enabling spatial vision-language reasoning with reduced computational requirements. Language and Visual Backbones. Pretrained language and vision backbones have proven essential for multimodal reasoning tasks. The Text-to-Text Transfer Transformer (T5) [ 13 ] is a unified encoder-decoder model that reformulates diverse NLP tasks into a text-to-text format, enabling broad generalization. For vision, CLIP-ViT [ 12 ] leverages contrastive learning on image-text pairs to produce global semantic embeddings, while DPT [ 14 ] introduces dense spatial features for monocular depth estimation. These models have been widely adopted as backbones in multimodal systems, where global and region-level visual features are combined with pretrained language representations to support complex spatial understanding. Mixture-of-Experts. The Mixture-of-Experts (MoE) framework was first introduced by Jordan et al . [ 7 ] , establishing a theoretical foundation for modular learning with conditional computation. Shazeer et al . [ 15 ] scaled MoE to deep neural networks using softmax-based sparse gating, enabling parameter efficiency and specialization. More recently, MoE has been adapted to vision-language tasks and multimodal reasoning, where semantic alignment is critical. SparseMoE [ 9 ] proposed cosine similarity as a gating function to encourage expert selection based on domain similarity. FuseMoE [ 4 ] introduced a novel Laplace-based gating mechanism, theoretically shown to improve convergence and empirically demonstrated to enhance expert utilization and predictive performance in heterogeneous, multimodal settings. 3 Methodology Figure 2 : Proposed architecture of TinyGiantVLM. The model processes both global and region-level RGB and depth images through feature extractors, followed by projection and fusion of RGB-depth features. The resulting global and region features are integrated using cross-attention with question embeddings. Region features are injected into masked positions in the question and passed through a self-attention layer and content extractor. Task-specific reasoning is designed to be handled by a Mixture-of-Experts (MoE) fusion layer guided by a sparse gate, with separate experts for different question types. While this MoE module is part of our

Read the complete text