Visual serial processing deficits explain divergences in human and VLM reasoning
Nicholas Budny, Kia Ghods, Declan Campbell, Raja Marjieh, Amogh Joshi, Sreejan Kumar, Jonathan D. Cohen, Taylor W. Webb, Thomas L. Griffiths · 2025
Read Visual serial processing deficits explain divergences in human and VLM reasoning by Nicholas Budny, Kia Ghods, Declan Campbell, Raja Marjieh, Amogh Joshi, Sreejan Kumar, Jonathan D. Cohen, Taylor W. Webb, Thomas L. Griffiths (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25142.txt.
Introduction
Nicholas Budny and a team of co-authors traced AI vision fails to a lack of step by step sight. Their paper, from twenty twenty-five, starts with a puzzle: vision language models ace stock tests yet trip on plain sight tasks that people find easy. The team's guess is a flaw in serial sight work: the step wise scan people do in their heads. They set man and model side by side in three fields.
Shape thought, count by sight, and mind spin tasks each came in grades of serial load. As load rose, model scores fell off from human ones in a set way. Readers in AI mind science get a clean cause for a wide gap.
Catalog record
| CREATOR | Nicholas Budny, Kia Ghods, Declan Campbell, Raja Marjieh, Amogh Joshi, Sreejan Kumar, Jonathan D. Cohen, Taylor W. Webb, Thomas L. Griffiths |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-25142
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Visual serial processing deficits explain divergences in human and VLM reasoning Nicholas Budny, Kia Ghods, Declan Campbell, Raja Marjieh, Amogh Joshi, Sreejan Kumar, Jonathan D. Cohen, Taylor W. Webb, Thomas L. Griffiths Abstract Why do Vision Language Models (VLMs), despite success on standard benchmarks, often fail to match human performance on surprisingly simple visual reasoning tasks? While the underlying computational principles are still debated, we hypothesize that a crucial factor is a deficit in visually-grounded serial processing. To test this hypothesis, we compared human and VLM performance across tasks designed to vary serial processing demands in three distinct domains: geometric reasoning, perceptual enumeration, and mental rotation. Tasks within each domain varied serial processing load by manipulating factors such as geometric concept complexity, perceptual individuation load, and transformation difficulty. Across all domains, our results revealed a consistent pattern: decreased VLM accuracy was strongly correlated with increased human reaction time (used as a proxy for serial processing load). As tasks require more demanding serial processing—whether composing concepts, enumerating items, or performing mental transformations—the VLM-human performance gap widens reliably. These findings support our hypothesis, indicating that limitations in serial, visually grounded reasoning represent a fundamental bottleneck that distinguishes current VLMs from humans. 1 Introduction Large-scale vision-language models (VLMs), such as GPT-4o ( Achiam et al. 2023 ) , Claude-Sonnet 3.7 ( Anthropic 2023 ) , and Gemini 2.5 Pro ( Team et al. 2023 ) , demonstrate impressive capabilities, often matching or surpassing human-level accuracy on complex benchmarks like image captioning and visual question answering ( Fu et al. 2023 ) . Yet, paradoxically, they also exhibit surprising deficits in tasks that appear comparatively simple, such as counting and visual search ( Campbell et al. 2024b ) . Here, we hypothesize that these discrepancies, and a large amount of VLM-human divergence more broadly, stem from deficits in visually-grounded serial processing. To investigate this hypothesis and better evaluate the extent of these deficits, we use tasks from cognitive science ( Dehaene et al. 2006 ) to vary serial processing demands systematically. To provide a proxy for serial processing load, we use human reaction time (RT) — a well-established behavioral marker of serial cognitive engagement. Humans adapt to complexity in visual reasoning by trading time for accuracy, using sequential attention to focus on different elements when analyzing complex scenes ( Heitz 2014 ; Stewart et al. 2020 ) . We hypothesize that VLMs largely lack this capability, as their reasoning is tethered to sequential generation of text (their chain of thought), not sequential analysis of the image. We therefore predict that VLM accuracy will be inversely correlated with human RT, indicating that as tasks require more extensive serial processing by humans, the VLM-human performance gap widens. This method allows us to identify where VLM performance diverges most significantly from human performance and to evaluate whether these divergences align with visual concepts and operations hypothesized to demand more intensive serial processing. To test our hypothesis, we systematically manipulate the serial processing load in three distinct domains. First, in a geometric program induction task inspired by work modeling geometric concepts as programs ( Hsu et al. 2022 ) , we vary load by manipulating concept complexity, operationalized as the number of geometric primitives (Minimum Description Length; MDL) required to define a geometric concept ( Sablé-Meyer et al. 2022 ) . Second, for visual enumeration, drawing on work distinguishing subitizing from serial counting ( Trick & Pylyshyn 1994 ) , serial load is varied by manipulating color distinctiveness and spatial overlap, known to increase demands on serial attention ( Franconeri et al. 2013 ) . Finally, in a mental rotation task based on ( Shepard & Metzler 1971 ) , serial processing demands are modulated by varying the angular disparity between two stimuli, a factor shown to linearly increase human processing time ( Cooper & Shepard 1973 ) . Across these diverse tasks we demonstrate that as task conditions necessitate greater serial processing the VLM-human performance gap widens reliably, providing convergent evidence for a serial processing deficit. 2 Background and Related Work 2.1 Vision Language Models Vision Language Models (VLMs) are designed to jointly model visual and linguistic information, typically by learning multimodal representations that bridge inputs from both domains ( Radford et al. 2021 ; Alayrac et al. 2022 ) . These models commonly incorporate a vision encoder to extract features from images or videos, which are then interfaced with a large language model conditioned on these visual features alongside textual prompts ( Achiam et al. 2023 ; Team et al. 2023 ; Anil et al. 2023 ; AI 2024 ) . This architectural paradigm has enabled significant progress on a variety of downstream tasks, including image captioning, visual question answering (VQA), and, more recently, complex multimodal reasoning ( Liu et al. 2023 ; Zhu et al. 2023 ) . Despite these advances, the precise mechanisms by which these models integrate and reason about visual information remain an active area of study. Of particular interest is the extent to which the reasoning strategies learned by VLMs correspond to, or diverge from, human visual reasoning strategies. 2.2 Measuring correspondences between human visual processing and VLMs To gauge the alignment between human and VLM visual reasoning, recent studies have benchmarked VLMs against human performance on a wide range of visual reasoning tasks from cognitive science. These comparisons have uncovered both similarities and crucial differences in their behavioral biases ( Campbell et al. 2024b ; Dasgupta et al. 2022 ) . In some geometric reasoning tasks, VLMs exhibit remarkably human-like biases ( Campbell et al. 2023 ; Campbell et al. 2024a ) , successfully performing tasks involving perceptual grouping and identifying partially obscured objects ( Allen et al. 2025 ; Pothiraj et al. 2025 ) . The relative difficulty of spatial reasoning tasks also aligns between VLMs and humans ( Xu et al. 2025 ) , suggesting some commonalities. Despite these similarities, a significant gap persists between VLM and human performance across several core cognitive tasks ( Schulze Buschoff et al. 2025 ) . VLMs are known to have deficits in spatially grounding reasoning ( Tong et al. 2024 ; Kamath et al. 2023 ; Xu et al. 2025 ; Ramakrishnan et al. 2025 ) , feature integration ( Campbell et al. 2024b ) , and multi-step reasoning ( Campbell et al. 2024b ; Greff et al. 2020 ) . These manifest in specific tasks where VLM performance is near-chance: determining whether circles overlap, counting, identifying circled letters ( Rahmanzadehgervi et al. 2025 ) , estimating depth, establishing visual correspondence, performing multi-view reasoning ( Fu et al. 2024 ) , and even failing at simple visual analogies that are solvable by toddlers ( Goddard et al. 2025 ) —all tasks where human accuracy is at ceiling. What unites these seemingly diverse failures? This constellation of behavioral traits ( Ramakrishnan et al. 2025 ) currently lacks a unifying explanation. While various frameworks have catalogued VLM failures across different benchmarks ( Schulze Buschoff et al. 2025 ; Ramakrishnan et al. 2025 ; Huang et al. 2025 ) , effectively demonstrating when performance falters, the question of why they fail in these settings remains unresolved. Existing explanations range from training data constraints to