Mechanistic Diagnostics of Spatial Lexical Bias in Multimodal Large Language Model Spatial Reasoning

Chuang Ma, Qianying Liu, Tomoyuki Obuchi, Fei Cheng, Wang Yang, Sudong Cai, Shuyuan Zheng, Akiko Aizawa, Sadao Kurohashi · 2026

Read Mechanistic Diagnostics of Spatial Lexical Bias in Multimodal Large Language Model Spatial Reasoning by Chuang Ma, Qianying Liu, Tomoyuki Obuchi, Fei Cheng, Wang Yang, Sudong Cai, Shuyuan Zheng, Akiko Aizawa, Sadao Kurohashi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01914.txt.

Introduction

Ma and co-authors diagnose a word-based bias in multimodal model reasoning. These models stay unreliable on spatial multiple-choice questions. Failures are often blamed on weak visual attention. The team find a different failure mode at work. A spatial word in the options can act as a lure.

Tests span nine open-weight multimodal models. The bias shows up widely across them. Some cases stay right on binary questions. Yet they flip when a third spatial option appears. Probes show the correct relation stays inside the model. The failure sits on the language side, not vision. Patching tests and sparse component checks support that view.

Catalog record

CREATORChuang Ma, Qianying Liu, Tomoyuki Obuchi, Fei Cheng, Wang Yang, Sudong Cai, Shuyuan Zheng, Akiko Aizawa, Sadao Kurohashi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01914

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mechanistic Diagnostics of Spatial Lexical Bias in Multimodal Large Language Model Spatial Reasoning Chuang Ma, Qianying Liu, Tomoyuki Obuchi, Fei Cheng, Wang Yang, Sudong Cai, Shuyuan Zheng, Akiko Aizawa, Sadao Kurohashi Abstract Multimodal large language models (MLLMs) remain unreliable on spatial multiple-choice questions, and their failures are often attributed to poorly attended visual information. In this work, we identify a complementary failure mode, spatial lexical bias : adding a spatial relation word to the answer options can attract the model’s decision and make the newly added option likely to be selected. Using nine open-weight MLLMs, we show that this phenomenon is widely observed. In particular, models can answer a binary spatial question correctly, yet consistently select an incorrect third spatial option once it is added to the answer set. We isolate such binary-stable but ternary-fragile cases as diagnostic examples and leverage mechanistic interpretability tools, revealing that a substantial part of the failure instead originates on the language side rather than the visual side: visual attention analyses and residual-stream probes show the correct spatial relation remains internally available on these failures, while irrelevant-option controls, activation patching, and sparse component interventions trace the bias to specific LLM-side channels and neurons. Based on this finding, we show that a lightweight LLM-only DPO update on tiny single-object-pair synthetic data mitigates the bias, lifting four-way robust accuracy by up to 100 points on synthetic data, and by 68.0, 32.6, and 20.1 points on broader evaluation datasets WhatsUp, SpatialMQA-Direct, and VSR. $\dagger$ $\dagger$ footnotetext: Corresponding author. 1 Introduction Figure 1: Failure pattern and evidence chain. (Top) One illustrative sample answers the binary {Left, Right} question correctly under both option orders, but chooses the added “Behind” option in all six ternary orderings. (Bottom) Our diagnostics test whether the relation remains recoverable from last-token residual-stream vectors, identify LLM-side residual-stream channels and MLP neurons whose intervention changes selected failures, and evaluate an LLM-only LoRA-DPO repair. Spatial reasoning evaluates whether multimodal large language models ground language in visual spatial structure, supporting applications that require models to understand object locations ( Liu et al. 2023 ; Kamath et al. 2023 ; Fu et al. 2024 ; Liu et al. 2025 ; Chen et al. 2024 ) . Recent studies have repeatedly shown that MLLMs remain unreliable on spatial reasoning tasks, and prior work often attributes these failures to weak visual grounding, missing visual evidence, or incorrect visual attention ( Tong et al. 2024 ; Leng et al. 2024 ; Huang et al. 2024 ; Chen et al. 2025 ) . In this study, we begin from a simple but revealing phenomenon that challenges such purely visual-information-loss explanation. As illustrated in Figure 1 , given the same image and the same question, a model can answer correctly when choosing between the binary options left and right . Yet when a new spatial option, behind , is added, the model can collapse and consistently choose the newly introduced option, regardless of answer order. For such specific option combinations, which we refer to as binary-stable but ternary-fragile (BSTF) cases, this collapse is systematic at the case level: a model can solve the binary classification nearly perfectly, yet once the particular spatial alternative is introduced, it fails across the ternary samples and consistently prefers the new option. This collapse raises a diagnostic question: when the model fails under the larger option set, has it truly lost the visual relation, or has the added spatial word changed the language-side decision process? We investigate this question in a controlled synthetic setting (Figure 1 ). If the added option disrupted visual grounding, we would expect weaker access to the relevant visual evidence. We observe the opposite. Attention maps show that binary-correct and ternary-wrong prompts attend to similar object regions, so the added option does not redirect the model away from the relevant visual evidence. Layer-wise probing of the last-token residual stream further recovers the correct spatial relation almost perfectly, even when the model selects the newly added wrong option. These results indicate that the collapse is not primarily caused by visual information loss: the correct relation is still present in the model’s internal representation, while the final answer is redirected by the changed option set. We therefore hypothesize the failure instead points to a language-side spatial lexical bias among spatial answer options. We argue that spatial answer words are not neutral labels: words such as left, right, front, and behind carry structured lexical-semantic relations inside the LLM, and a newly added spatial word can act as a lexical-semantic distractor that interferes with the LLM-side decision process. We compare spatial distractors with irrelevant options such as violin . Irrelevant options cause only modest degradation, whereas specific spatial alternatives can collapse the prediction and are consistently selected as the wrong answer. This contrast suggests that the failure is not merely caused by increasing the number of choices, but by the semantic structure among spatial words. We further show that the same distractors extend from ternary to four-way settings, indicating that the model has stable preferences over particular spatial options. To move beyond behavioral observation, we use interpretability tools to locate where the lexical-semantic attraction emerges. Activation patching shows that exchanging residual-stream states between binary-correct and ternary-wrong prompts can rescue or corrupt the answer, with the transition emerging around the middle-to-late layers. Sparse interventions further identify residual-stream channels and MLP neurons at the added-option position whose knockout partially restores correct predictions. These results provide causal evidence that specific LLM-side layers and components modulate the spatial-option bias. Finally, we test whether spatial lexical bias can be mitigated through a lightweight LLM-side update. Using a minimal synthetic preference dataset built from a single object-pair setting, we apply Direct Preference Optimization (DPO) only to the LLM decoder, with the vision encoder and projector kept frozen. Despite the simplicity of the training source, this LLM-only debiasing substantially improves spatial reasoning under four-way option sets, with gains of up to 100 percentage points on our controlled synthetic data. The improvement also transfers to broader and noisier public benchmarks, reaching gains of up to 68.0 points on WhatsUp, 32.6 points on SpatialMQA-Direct, and 20.1 points on VSR. This suggests that the bias is not limited to the specific binary-to-ternary cases but reflects a broader source of spatial reasoning failure, repairable by adjusting the language-side decision process. In summary, our contributions are threefold: • We identify binary-stable but ternary-fragile cases, a systematic failure pattern in which MLLMs solve a binary spatial contrast but collapse after a particular spatial alternative is added. Attention and probing analyses show the correct visual relation remains internally recoverable, indicating the failure is not simply caused by visual information loss. • We introduce spatial lexical bias as an LLM-side mechanism behind these failures. Irrelevant-option controls and ternary-to-four-way extensions show that spatial answer words interact in a structured way, while activation patching and sparse component interventions localize the bias to manipulable LLM-side states and components. • We show that this bias can be reduced with LLM-only DPO debiasing. A minimal single-object-pair synthetic preference dataset substantially improves spatial robustness and transfers to broader, noisier evaluation data. 2 Related Work We situate this work along three lines: spatial reasoning in MLLMs, MCQ position bias, and mechanistic interpretability.

Read the complete text