SAT : Dynamic Spatial Aptitude Training for Multimodal Language Models

Arijit Ray, Jiafei Duan, Ellis Brown, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A. Plummer, Ranjay Krishna, Kuo-Hao Zeng, Kate Saenko · 2024

Read SAT : Dynamic Spatial Aptitude Training for Multimodal Language Models by Arijit Ray, Jiafei Duan, Ellis Brown, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A. Plummer, Ranjay Krishna, Kuo-Hao Zeng, Kate Saenko (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07755.txt.

Introduction

Arijit Ray and co-authors present a computing paper from 2024 on space sense in smart models. Models that mix text and images often fail at tasks about space and motion. Past tests looked only at still relations, like left of or above. Real tasks also need a sense of how motion changes those relations.

The team built a training set with a three-dimensional simulator. It holds 175 thousand question-and-answer pairs over twenty thousand scenes. They added a small, hard test set drawn from real photos. Training on the fake data lifted scores on many space tests, even on real photos. One model gained about eleven points on average. Dynamic questions, about motion, still left much room to improve.

Catalog record

CREATORArijit Ray, Jiafei Duan, Ellis Brown, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A. Plummer, Ranjay Krishna, Kuo-Hao Zeng, Kate Saenko
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-07755

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SAT : Dynamic Spatial Aptitude Training for Multimodal Language Models Arijit Ray, Jiafei Duan, Ellis Brown, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A. Plummer, Ranjay Krishna, Kuo-Hao Zeng, Kate Saenko Abstract † † ∗ equal advising, † joint second author Reasoning about motion and space is a fundamental cognitive capability that is required by multiple real-world applications. While many studies highlight that large multimodal language models (MLMs) struggle to reason about space, they only focus on static spatial relationships, and not dynamic awareness of motion and space— i.e., reasoning about the effect of egocentric and object motions on spatial relationships. Manually annotating such object and camera movements is expensive. Hence, we introduce SAT , a simulated spatial aptitude training dataset utilizing 3D simulators, comprising both static and dynamic spatial reasoning across 175,000 question-answer (QA) pairs and 20,000 scenes. Complementing this, we also construct a small (150 image-QAs) yet challenging dynamic spatial test set using real-world images. Leveraging our SAT datasets and 6 existing static spatial benchmarks, we systematically investigate what improves both static and dynamic spatial awareness. Our results reveal that simulations are surprisingly effective at imparting spatial aptitude to MLMs that translate to real images. We show that perfect annotations in simulation are more effective than existing approaches of pseudo-annotating real images. For instance, SAT training improves a LLaVA-13B model by an average 11 % 11\% and a LLaVA-Video-7B model by an average 8 % 8\% on multiple spatial benchmarks, including our real-image dynamic test set and spatial reasoning on long videos—even outperforming some large proprietary models. While reasoning over static relationships improves with synthetic training data, there is still considerable room for improvement for dynamic reasoning questions. Figure 1 : We propose Spatial Aptitude Training (SAT), which generates simulated spatial training data that allows us to go beyond simple static relationships in existing datasets. Inspired by cognitive science, we add more challenging dynamic questions that require reasoning about ego and object motion. SAT improves performance on both existing spatial benchmarks and dynamic reasoning on our new real-image benchmark. 1 Introduction Reasoning about space and motion is a fundamental cognitive capability that allows humans and animals to survive and operate in the real world ( Gardner, 2011 ; Vasilyeva and Lourenco, 2012 ; Blades and Spencer, 1994 ) . While simple visual relationships (such as left-right) may be trivial to recognize for humans, more dynamic reasoning involving motion such as taking perspectives and reasoning about the effect of ego-motion or object movement requires more cognitive support ( Gardner, 2011 ; Tversky, 2019 ) . Hence, it is not surprising that such “dynamic” spatial reasoning capabilities in children correlate with their aptitude in geometry, physics, and even linguistic reasoning ( Taylor and Tversky, 1992 ; Mallot, 2024 ) . Despite the promise of multimodal language models (MLMs) as general-purpose intelligent agents ( Achiam et al., 2023 ; Zhu et al., 2023 ; Liu et al., 2024 ) , recent studies reveal that these models still struggle with spatial reasoning ( Chen et al., 2024 ; Cheng et al., 2024 ; Tong et al., 2024 ; Fu et al., 2024b ) . Specifically, MLMs often fail to predict the relative positions of objects in static images—a limitation attributed to the scarcity of spatial relationship annotations in their training data. To address this, recent methods ( Cheng et al., 2024 ; Chen et al., 2024 ) introduce pseudo-annotations to encode static object relationships in real images ( Cheng et al., 2024 ) , which require extensive engineering and refining to scale. Furthermore, many real-world applications require reasoning that extends beyond static object positions. For example, smart glasses and embodied AI applications need to reason about dynamic scenes, where reasoning about the effect of object and camera movements are essential ( Duan et al., 2024 ; Yuan et al., 2024a ; Liu et al., 2025 ) . Generating pseudo-annotations using existing methods for such dynamic settings is non-trivial due to the complexity of predicting action causality on real images. We explore the possibility of using simulation data as a simple solution to impart both static and dynamic spatial reasoning into MLMs. We propose Spatial Aptitude Training ( SAT ), an approach to generate spatial question-answer (QA) data without any human supervision. Manually annotating 3D movements is expensive; hence, SAT leverages 22K ProcTHOR ( Deitke et al., 2022 ) scenes composed of 1K assets to generate 175K QA pairs. With perfect 3D information and control of the assets, SAT goes beyond static object relationships to questions that require reasoning about egocentric and object movements. Since our data is generated procedurally by composing assets, it can be scaled up arbitrarily. Hence, SAT is programmatically controllable, unlike existing 3D ( Brazil et al., 2023 ; Azuma et al., 2022 ) or spatial datasets ( Cheng et al., 2024 ) , which are also not compositional. With SAT , we analyze what types of training data improve spatial reasoning. We focus on two types of spatial reasoning data. First, spatial-QA about object relations in static scenes, shown in Fig. 1 (under “Existing Spatial Bench”), to impart reasoning about the relative locations of objects in the scene ( e.g., where is object X with respect to object Y?). Next, we evaluate the effect of dynamic spatial tasks, Fig. 1 (shown under “Our SAT Dynamic Bench”). This includes questions about egocentric movement, object movement, allocentric perspective, goal aiming, and action consequences. These dynamic spatial reasoning skills are well-studied in human cognitive development: children understand the consequences of self-motion ( e.g., the moving room test ( Anderson et al., 2013 ) ), track how entities move in a scene ( Anderson et al., 2013 ) , and can adopt others’ viewpoints when planning actions ( Brucato et al., 2023 ; Vasilyeva and Lourenco, 2012 ; Blades and Spencer, 1994 ) . We use two widely adopted open-source MLMs, an image-based LLaVA-1.5-13B and a video-based LLaVA-Video-7B, as our base models for evaluations. To test static spatial reasoning, we use four contemporary real image benchmarks: CV-Bench ( Tong et al., 2024 ) , BLINK ( Fu et al., 2024b ) , Visual Spatial Relations (VSR) dataset ( Liu et al., 2023 ) , and GQA-Spatial ( Hudson and Manning, 2019a ) . In addition, we test the effect of SAT on spatial reasoning on videos using the recent VSI-Bench ( Yang et al., 2024a ) benchmark and MME-RealWorld ( Zhang et al., 2024c ) , which include out-of-domain outdoor videos in realistic self-driving scenarios. Since no dataset exists for dynamic spatial reasoning on images (predicting action causality on images), we construct a difficult SAT test set on real images. Challenging conventional wisdom about sim-to-real transfer in model training, we find simulations to be surprisingly effective at imparting spatial intelligence to MLMs. Our simulated SAT training improves the baseline LLaVA-13B model by 11\% and LLaVA-video-7B by 8\% (absolute) on average on a wide range of spatial benchmarks, including our challenging dynamic test set—even outperforming some larger closed-source and spatially-tuned models ( Achiam et al., 2023 ; Team et al., 2023 ) . Interestingly, SAT also improves spatial reasoning on long videos ( Yang et al., 2024a ) —especially

Read the complete text