Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Models

Luca M. Schulze Buschoff, Konstantinos Voudouris, Elif Akata, Matthias Bethge, Joshua B. Tenenbaum, Eric Schulz · 2025

Read Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Models by Luca M. Schulze Buschoff, Konstantinos Voudouris, Elif Akata, Matthias Bethge, Joshua B. Tenenbaum, Eric Schulz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15678.txt.

Introduction

Luca Schulze Buschoff and five co-authors test if fine-tuning can make vision-language models see the world more like people do. Their twenty twenty-five paper notes that even strong pre-trained models lag behind human visual thought. The team gathered images and human judgments across many kinds of visual tasks.

They then fine-tuned models on true answers for two domains: intuitive physics and causal reasoning. Scores rose in the tuned domain, and the models also matched human answers more closely there. But the gains did not carry to new looks or new domains. The verdict is that narrow tuning helps locally, yet broad human-like sight needs more.

Catalog record

CREATORLuca M. Schulze Buschoff, Konstantinos Voudouris, Elif Akata, Matthias Bethge, Joshua B. Tenenbaum, Eric Schulz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-15678

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Models Luca M. Schulze Buschoff, Konstantinos Voudouris, Elif Akata, Matthias Bethge, Joshua B. Tenenbaum, Eric Schulz Abstract Pre-trained vision language models still fall short of human visual cognition. In an effort to improve visual cognition and align models with human behavior, we introduce visual stimuli and human judgments on visual cognition tasks, allowing us to systematically evaluate performance across cognitive domains under a consistent environment. We fine-tune models on ground truth data for intuitive physics and causal reasoning and find that this improves model performance in the respective fine-tuning domain. Furthermore, it can improve model alignment with human behavior. However, we find that task-specific fine-tuning does not contribute to robust human-like generalization to data with other visual characteristics or to tasks in other cognitive domains. † † affiliationnotice: Equal contribution 1 Introduction One of the main goals of machine learning research is to build machines that think and behave like humans. To meet this goal, Lake et al. (2017) proposed that human-like machine learning models must be capable of reasoning about their physical and social environment and its causal structure. These capabilities are sometimes summarized as intuitive theories —the cognitive expectations humans and other animals have about their environment from early on in development that they use to behave adaptively. In this paper, we focus on two classes of intuitive theory. Intuitive physics relates to the ability to predict and understand the physical properties and interactions of inanimate objects ( Battaglia et al., 2012 ; Piloto et al., 2022 ) , an ability that is present very early in development and does not require extensive learning or experience ( Baillargeon et al., 1995 ; Spelke, 1990 ; Spelke & Kinzler, 2007 ) . Causal reasoning describes the ability to infer cause-effect relationships ( Waldmann, 2017 ; Pearl, 2009 ) . There is growing evidence that humans possess an intuitive capacity to infer and predict causal relationships ( Griffiths & Tenenbaum, 2009 ) , and that this ability emerges early in development ( Kuhn, 2012 ; Sobel & Kirkham, 2006 ) . In the psychology literature, intuitive physics and causal reasoning have been studied most prominently in their relation to visual cognition—investigating how humans and other animals reason about their physical environment and its causal structure through the visual inputs they receive. Vision language models (VLMs), which receive visual and textual linguistic input and produce textual output, have received recent attention for their apparently sophisticated reasoning in visual and linguistic tasks ( Liu et al., 2025 ) . However, recent work has established that VLMs are still limited in their understanding of the physical world and its causal structure ( Jin et al., 2023 ; Balazadeh et al., 2024 ) , suggesting that they lack human-like intuitive physics and causal reasoning. While VLMs perform reasonably well on intuitive physics problems, such as predicting the stability of block towers, they do not show a good fit with human behavioral data. On tests of causal reasoning, such as predicting whether removing a block would cause a tower to fall, VLMs perform poorly and again do not fit well with human behavior ( Schulze Buschoff et al., 2025 ) . Beyond the domains of intuitive physics and causal reasoning, VLMs have also been shown to have a number of visual deficiencies, and they often struggle with simple visual tasks that would be trivial for a human observer ( Rahmanzadehgervi et al., 2024 ; Schulze Buschoff et al., 2025 ; Balazadeh et al., 2024 ) . VLMs are prone to hallucinations, where the corresponding output does not sensibly correspond to the input image ( Li et al., 2023 ; Liu et al., 2024 ) . Ullman (2024) shows that VLMs hallucinate visual illusions where there are none, if the visual stimuli resemble canonical illusions that were likely in their training data. Similarly, Zhang et al. (2023) show that while the general alignment to human perception is low, larger models are somewhat susceptible to the same visual illusions as humans. Additionally, VLMs are not adversarially robust and are therefore subject to manipulation of both textual and visual inputs ( Zhao et al., 2024 ) . Campbell et al. (2024) suggest that the failures of VLMs on tasks containing multiple objects can be explained by a binding problem , in which VLMs, like humans ( Frankland et al., 2021 ) , struggle to attend to, represent, and distinguish between multiple objects at the same time, because they share the same representational resources. Figure 1: Methodology overview. A: We study causal reasoning and intuitive physics using our Cubeworld fine-tuning and evaluation datasets and the Lerer et al. (2016) block tower evaluation dataset. B: Models are fine-tuned on the ground truth or human judgments for a domain of the Cubeworld dataset. C: We test whether fine-tuning improves model performance in four scenarios: new towers of the same heights as in training; new towers of different heights compared to training; naturalistic images from Lerer et al. (2016) ; and block towers from the other domain. We also test the alignment of these models to human responses. In pursuit of improving the performance of language models, fine-tuning is quickly distinguishing itself as the gold standard, enabling researchers to efficiently steer models towards better capabilities ( Han et al., 2024 ) as well as towards more human-aligned outputs ( Binz et al., 2024 ; Hussain et al., 2024 ) . In this paper, we explore whether fine-tuning VLMs on single tasks can improve their performance on intuitive physics and causal reasoning tasks in the visual domain, as well as steer them towards more human-aligned outputs. However, a hallmark of human cognition is not just the ability to reason about the physical environment and its causal structure, but also to robustly generalize from limited experience to solve new tasks ( Collins et al., 2022 ; Geirhos et al., 2018 ; Griffiths & Tenenbaum, 2009 ) . Therefore, we seek to evaluate whether task-specific fine-tuning not only improves performance on visual cognition tasks sampled from an identical distribution, but also whether it produces models that can generalize to new, but related, tasks in new domains. For example, we ask whether a model fine-tuned to accurately judge the stability of short tower blocks can generalize this knowledge to judge the stability of tall tower blocks, of tower blocks with different visual characteristics (from a different environment), or to causal reasoning problems about tower blocks. Our results allow us to appraise the limits of task-specific fine-tuning for building performant, human-like machine learning models that can generalize beyond the kinds of data on which they have been trained. Across a range of datasets and models, we do not find evidence that fine-tuning alone can achieve all these objectives. 1.1 Related Work Closest to our work is Balazadeh et al. (2024) , who fine-tune the VLM PaliGemma-3B ( Beyer et al., 2024 ) on a series of intuitive physics and visual reasoning tasks, asking questions about the height, color, and shape of tower blocks in an image, as well as whether the towers are stable or certain blocks are likely to move. They find that smaller

Read the complete text