Can VLMs Reason Robustly? A Neuro-Symbolic Investigation

Weixin Chen [email protected], Antonio Vergari [email protected], Han Zhao [email protected] · 2026

Read Can VLMs Reason Robustly? A Neuro-Symbolic Investigation by Weixin Chen [email protected], Antonio Vergari [email protected], Han Zhao [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23867.txt.

Introduction

Chen and co-authors test whether vision-language models keep reasoning well when image inputs shift. The prediction rules stay fixed, but the look of the inputs changes. Fine-tuned models score high on familiar data yet fail under such shifts, they find. Their method, VLC, splits the work in two. A vision model names the objects in the image. A circuit then applies the task rules exactly to those concepts. On three visual deduction tasks, VLC beats other approaches on out-of-distribution data.

Catalog record

CREATORWeixin Chen [email protected], Antonio Vergari [email protected], Han Zhao [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51059
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.23867

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can VLMs Reason Robustly? A Neuro-Symbolic Investigation Weixin Chen [email protected], Antonio Vergari [email protected], Han Zhao [email protected] Abstract Vision-Language Models (VLMs) have been applied to a wide range of reasoning tasks, yet it remains unclear whether they can reason robustly under distribution shifts. In this paper, we study covariate shifts in which the perceptual input distribution changes while the underlying prediction rules do not. To investigate this question, we consider visual deductive reasoning tasks, where a model is required to answer a query given an image and logical rules defined over the object concepts in the image. Empirically, we find that VLMs fine-tuned through gradient-based end-to-end training can achieve high in-distribution accuracy but fail to generalize under such shifts, suggesting that fine-tuning does not reliably induce the underlying reasoning function. This motivates a neuro-symbolic perspective that decouples perception from reasoning. However, we further observe that recent neuro-symbolic approaches that rely on black-box components for reasoning can still exhibit inconsistent robustness across tasks. To address this issue, we propose VLC , a neuro-symbolic method that combines VLM-based concept recognition with circuit-based symbolic reasoning. In particular, task rules are compiled into a symbolic program, specifically a circuit, which executes the rules exactly over the object concepts recognized by the VLM. Experiments on three visual deductive reasoning tasks with distinct rule sets show that VLC consistently achieves strong performance under covariate shifts, highlighting its ability to support robust reasoning. 1 Introduction Vision-Language Models (VLMs) have recently been applied to a wide range of reasoning tasks, including abstract ( 17 ; 69 ) , temporal ( 42 ; 27 ) , and document reasoning ( 84 ; 73 ) , as well as relational, attributive, and order understanding ( 78 ; 83 ) . Despite these advances, it remains unclear whether VLMs can reason robustly under distribution shifts. Here, we focus on covariate shifts ( 61 ; 62 ; 63 ; 6 ) , where the distribution of the perceptual input changes, while the underlying rules for prediction do not. To study the robustness against covariate shifts, we use visual deductive reasoning tasks . In these tasks, an image containing multiple objects is provided, along with rules that define the reasoning function based on object concepts. The VLM is prompted to answer a query that requires reasoning over the object concepts using the given rules. See Fig. 1 (left) for an example. We focus on these tasks because they include samples with perceptual inputs of varying complexity and whose labels are generated by the same reasoning function. For instance, the reasoning function can be fixed as addition, while the perceptual input contains different digits (in shape and number), see Fig. 1 . End-to-end fine-tuning is the canonical approach for adapting VLMs to specific downstream tasks. Following this standard practice, we fine-tune VLMs on visual deductive reasoning tasks using the causal language modeling loss. As shown in Figure 1 (right), despite achieving high accuracy on in-distribution data, the fine-tuned models fail to generalize to out-of-distribution (OOD) data that involves a different number of objects but follows the same reasoning function. This result contrasts with the strong generalization often observed when VLMs are fine-tuned for other vision-language tasks like recognition ( 41 ) , and suggests that gradient-based fine-tuning does not necessarily enable the model to learn the underlying reasoning function. To see whether this limitation stems from end-to-end VLM reasoning, we evaluate two recent neuro-symbolic approaches that decouple perception from reasoning: Prism ( 58 ) and ViperGPT ( 64 ) . Prism delegates reasoning to a Large Language Model (LLM), whereas ViperGPT prompts an LLM to generate executable programs that call upon specialized pretrained models. However, as we shall see in Section 5.2 , even with this decoupling of perception and reasoning, both approaches still show limited robustness under covariate shifts. This leads to a natural question: How can we encode the reasoning function into VLMs to achieve robust reasoning? Inspired by recent studies ( 18 ; 4 ) showing that VLMs excel at recognition tasks but struggle with reasoning, and in line with 58 ; 64 ; 33 , we propose VLC , a neuro-symbolic reasoning paradigm for VLMs that decomposes the end-to-end reasoning process into two sequential phases: VLM-based concept recognition and circuit-based symbolic reasoning . In the first phase, a VLM serves as the neural module, leveraging its strong recognition capabilities to identify object concepts in the input image. In the second phase, we use circuits ( 54 ; 16 ; 71 ) as the symbolic module, which compile the provided rules into its structure ( 24 ; 37 ; 51 ) . At inference time, the VLM first recognizes object concepts, after which the circuit applies the compiled rules to derive the final answer. By explicitly encoding the true reasoning function into the circuit, VLC ensures interpretable and robust reasoning and could allow for principled neuro-symbolic integration where constraints can be embedded in the learning pipeline of deep learning architecture ( 44 ; 3 ; 12 ) . For our purposes, and striving for simplicity, we adopt the two-stage prediction scenario of 12 , where learning is decomposed into a first stage where the VLM is used to predict object concepts and then the circuit is used to make predictions, essentially firing the symbolic rules based on the observed concepts. Figure 1 : Left: Given an image containing multiple objects and natural-language rules about object concepts, the (fine-tuned) VLM, Prism, and ViperGPT are required to answer a query based on the image by reasoning with the given rules. Middle: VLC decouples perception from reasoning and infers the final answer by applying the symbolic rule compiled into the circuit. Right: Performance of different paradigms on datasets sharing the same reasoning function but differing in the number of objects per image. Here, the reasoning function is logical XOR (see Section 2 for the task definition), and the three datasets contain images with three, five, and seven handwritten binary digits, respectively. The models are required to output the XOR of these digits. To evaluate different reasoning paradigms, we advocate going back to benchmarking simple visual deductive reasoning tasks where generalization can be controlled in a rigorous way. Specifically, we use the rsbench benchmark suite ( 7 ) to generate datasets in which the perceptual input varies through the number of objects in the image, while labels are produced by the same reasoning function, i.e. , the covariate shift setting ( 61 ; 62 ; 63 ; 6 ) . We train on samples with fewer objects and test on samples with more objects. In particular, we consider three distinct reasoning functions: arithmetic addition , logical XOR , and a relational check , respectively. Note that these datasets differ from classic neuro-symbolic benchmarks, which often assume that each object is already perfectly segmented and provided as a separate image, and typically do not provide concept annotations or explicit rules specifying how object concepts map to the final label. Empirically, we find that even on these simple tasks, end-to-end fine-tuned VLMs fail to generalize to OOD samples, suggesting that end-to-end fine-tuning does not reliably learn the underlying reasoning function from data. In contrast, VLC consistently achieves competitive

Read the complete text