Predicting When to Trust Vision-Language Models for Spatial Reasoning

Muhammad Imran, Yugyung Lee · 2026

Read Predicting When to Trust Vision-Language Models for Spatial Reasoning by Muhammad Imran, Yugyung Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11644.txt.

Introduction

Muhammad Imran and Yugyung Lee predict when to trust VLM spatial calls. Their 2026 preprint deals with vision-language models and space. VLMs show impressive skill across many multimodal tasks. Their spatial reasoning, though, fails in systematic ways.

Basic directional relationships expose the weakness clearly. CLIP scores only 49 percent on them in the record. BLIP-2 reaches 54 percent, still near a coin flip. Safe deployment in robotics needs a trust signal instead. The team predicts when a spatial prediction is reliable. Rejecting shaky calls beats accepting all outputs blindly. Autonomous systems gain a guard for a known weak spot.

Catalog record

CREATORMuhammad Imran, Yugyung Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-11644

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Predicting When to Trust Vision-Language Models for Spatial Reasoning Muhammad Imran, Yugyung Lee Abstract Vision-Language Models (VLMs) demonstrate impressive capabilities across multimodal tasks, yet exhibit systematic spatial reasoning failures—achieving only 49% (CLIP) to 54% (BLIP-2) accuracy on basic directional relationships. For safe deployment in robotics and autonomous systems, we need to predict when to trust VLM spatial predictions rather than accepting all outputs. We propose a vision-based confidence estimation framework that validates VLM predictions through independent geometric verification using object detection. Unlike text-based approaches relying on self-assessment, our method fuses four signals via gradient boosting: geometric alignment between VLM claims and coordinates, spatial ambiguity from overlap, detection quality, and VLM internal uncertainty. We achieve 0.674 AUROC on BLIP-2 (34.0% improvement over text-based baselines) and 0.583 AUROC on CLIP (16.1% improvement), generalizing across generative and classification architectures. Our framework enables selective prediction : at 60% target accuracy, we achieve 61.9% coverage versus 27.6% baseline (2.2× improvement) on BLIP-2. Feature analysis reveals vision-based signals contribute 87.4% of model importance versus 12.7% from VLM confidence—validating that external geometric verification outperforms self-assessment. We demonstrate reliable scene graph construction where confidence-based pruning improves precision from 52.1% to 78.3% while retaining 68.2% edges. 1 Introduction Vision-Language Models (VLMs) excel at high-level semantic tasks but exhibit critical weaknesses in spatial reasoning Liu and others 2023 ; Wang et al. 2025 . State-of-the-art VLMs achieve only 49–54% accuracy on spatial relations—barely better than chance—frequently confusing basic directions like “left” versus “right” Liu and others 2023 . This deficit prevents deployment in applications requiring precise spatial understanding: robotic navigation, autonomous driving, and image editing. While recent work attempts to enhance spatial reasoning through specialized architectures Kamath et al. 2023 ; Wang et al. 2025 or structured frameworks Huang et al. 2025 , these methods focus on improving predictions rather than quantifying uncertainty. For production systems, the ability to predict when to trust VLM spatial predictions is equally critical—enabling selective prediction where uncertain cases defer to human review. Existing confidence approaches rely on text-based signals: verbalized confidence Khan and others 2023 , multi-prompt consistency Manakul et al. 2023 , or token probabilities Kadavath and others 2022 . Khan et al. Khan and others 2023 achieve only 0.503 AUROC on BLIP-2—barely above random. These fail because VLMs exhibit systematic overconfidence: producing consistent but incorrect predictions with high token probabilities even when wrong. We propose vision-based confidence estimation that validates VLM predictions through independent geometric verification. Our key insight: spatial relationships are objectively verifiable—if a VLM claims “the dog is left of the person,” we detect both objects using GroundingDINO Liu et al. 2023 and verify positions through coordinates. This provides external ground truth independent of VLM internal state. Our method operates in four stages: (1) VLM produces a spatial prediction, (2) we detect objects and compute geometric relationships from bounding boxes, (3) we extract four confidence features: geometric alignment , separation confidence , detection quality , and VLM token confidence , and (4) we fuse signals via XGBoost to predict whether to trust the VLM. We train on 705 samples and evaluate on 312 held-out images from the VSR benchmark Liu and others 2023 . We evaluate on BLIP-2 Li and others 2023 (generative) and CLIP Radford and others 2021 (contrastive), comparing against geometric-only confidence and Khan et al.’s text-based method Khan and others 2023 . Results show vision-based confidence with learned fusion substantially outperforms both baselines. Contributions • Framework for predicting VLM trust : We achieve 0.674 AUROC on BLIP-2 (34.0% improvement) and 0.583 AUROC on CLIP (16.1% improvement), generalizing across architectures. • Selective prediction for deployment : At 60% target accuracy, we achieve 61.9% coverage versus 27.6% baseline on BLIP-2 (2.2× improvement) and 54.5% versus 12.2% on CLIP (4.5× improvement). At 50% accuracy, BLIP-2 reaches 91.7% coverage (2.3× baseline). • Scene graph construction : Edge-level confidence enables reliable graph extraction. At threshold 0.6, precision reaches 78.3% with 68.2% edge coverage (F1: 72.8%). • Vision-dominant features : Vision-based features contribute 87.4% of model importance versus 12.7% from VLM confidence—validating external verification over self-assessment. Our method adds only 46% computational overhead versus 500% for multi-prompt approaches. 2 Method We propose a vision-based confidence estimator for VLM spatial reasoning. Given an image \mathcal{I} , two objects (o_{1},o_{2}) , and a claimed spatial relation r (e.g., “left”, “right”), our goal is to estimate the probability that the VLM’s prediction is correct. Unlike text-based approaches relying on self-assessment Khan and others 2023 , we validate predictions through independent geometric verification using object detection. Figure 1: Overview of our vision-based confidence estimation pipeline. Given a VLM’s spatial prediction, we validate it through geometric verification using object detection. Four complementary confidence signals (geometric alignment, object separation, detection quality, VLM internal confidence) are fused via gradient boosting to produce a final confidence score. High-confidence predictions (green) are trusted; low-confidence predictions (red) are rejected or deferred to human review. Our pipeline (Figure 1 ) consists of four stages: ( 1 ) VLM spatial prediction, ( 2 ) geometric validation via object detection, ( 3 ) multi-signal feature extraction, and ( 4 ) learned confidence fusion. 2.1 VLM Spatial Prediction Given input image \mathcal{I} and object pair (o_{1},o_{2}) , we query the VLM with multiple prompt variations. For generative VLMs (BLIP-2 Li and others 2023 ), we use prompts like “ Where is the o_{1} relative to the o_{2} ? ” For classification-based models (CLIP Radford and others 2021 ), we score all possible relations r\in\{\text{left, right, above, below, near}\} by computing image-text similarity: s(r)=\text{sim}(\phi_{\text{img}}(\mathcal{I}),\phi_{\text{text}}(\text{``The }o_{1}\text{ is }r\text{ of the }o_{2}\text{''})) (1) The predicted relation is \hat{r}=\arg\max_{r}s(r) . 2.2 Geometric Validation via Object Detection We employ GroundingDINO Liu et al. 2023 to localize o_{1} and o_{2} , producing bounding boxes \mathbf{b}_{1}=(x_{1}^{\min},y_{1}^{\min},x_{1}^{\max},y_{1}^{\max}) and \mathbf{b}_{2} with detection confidences c_{1},c_{2}\in[0,1] . If either object is not detected (score below \tau=0.3 ), we assign zero confidence. From bounding boxes, we compute object centers: \mathbf{p}_{i}=\left(\frac{x_{i}^{\min}+x_{i}^{\max}}{2},\frac{y_{i}^{\min}+y_{i}^{\max}}{2}\right) (2) The geometric relation r_{\text{geo}} is determined by comparing displacements \Delta x=\mathbf{p}_{2}^{x}-\mathbf{p}_{1}^{x} and \Delta y=\mathbf{p}_{2}^{y}-\mathbf{p}_{1}^{y} : r_{\text{geo}}=\begin{cases}\text{left}&\text{if }|\Delta x|>|\Delta y|\text{ and }\Delta x>0\\ \text{right}&\text{if }|\Delta x|>|\Delta y|\text{ and }\Delta x<0\\ \text{above}&\text{if }|\Delta y|>|\Delta x|\text{ and }\Delta y>0\\ \text{below}&\text{if }|\Delta y|>|\Delta x|\text{ and }\Delta y<0\end{cases} (3) We compute geometric confidence \alpha_{\text{geo}}\in[0,1] measuring alignment between \hat{r} and r_{\text{geo}} : \alpha_{\text{geo}}=\begin{cases}0.2&\text{if }\hat{r}\text{ contradicts }r_{\text{geo}}\\ 0.5+0.5\cdot\min\left(1,\frac{d_{\text{primary}}}{100}\right)&\text{if }\hat{r}\text{ matches }r_{\text{geo}}\end{cases} (4) where d_{\text{primary}} is the displacement in the primary direction. We adjust by detection quality: \alpha_{\text{geo}}^{\text{adj}}=\alpha_{\text{geo}}\cdot\left(0.5+0.5\cdot\frac{1}{1+e^{-10(\bar{c}-0.3)}}\right) where \bar{c}=(c_{1}+c_{2})/2 . 2.3 Multi-Signal Feature Extraction We select four complementary features balancing model capacity and generalization: (1) Geometric Confidence ( \alpha_{\text{geo}} ). Alignment between VLM prediction and coordinate-based geometry (37.5% model importance). (2) Separation Confidence ( \alpha_{\text{sep}} ). Defined as \alpha_{\text{sep}}=1-\text{IoU} , where \text{IoU}=\frac{\text{Area}(\mathbf{b}_{1}\cap\mathbf{b}_{2})}{\text{Area}(\mathbf{b}_{1}\cup\mathbf{b}_{2})} . High overlap indicates ambiguous spatial relationships (32.7% importance). (3) Detection Quality ( \bar{c} ). Average detection confidence: \bar{c}=(c_{1}+c_{2})/2 . Low scores signal difficult visual conditions (17.2% importance). (4) VLM Token Confidence. For BLIP-2, maximum token probability from first generated token. For CLIP, maximum similarity score \max_{r}s(r) (12.7% importance). This minimal feature set reduces train-validation AUROC gap from 0.12 (10 features) to 0.10 (4 features), indicating better generalization. 2.4 Learned Confidence Fusion We aggregate signals into final confidence using gradient boosting Chen and Guestrin 2016 . Let \mathbf{f}=[\alpha_{\text{geo}},\alpha_{\text{sep}},\bar{c},\text{token\_conf}]^{\top} . We train XGBoost classifier

Read the complete text