Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It
Yulu Qin, Dheeraj Varghese, Adam Dahlgren Lindström, Lucia Donatelli, Kanishka Misra, Najoung Kim · 2025
Read Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It by Yulu Qin, Dheeraj Varghese, Adam Dahlgren Lindström, Lucia Donatelli, Kanishka Misra, Najoung Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13328.txt.
Introduction
Yulu Qin, Dheeraj Varghese, Adam Dahlgren Lindström, Lucia Donatelli, Kanishka Misra, and Najoung Kim ask whether adding vision training changes what language models know about word kinds, in work from 2025. Word kinds form a tree, such as animal over dog over poodle. Past results on vision training effects were mixed.
The team compared matched pairs of text-only models and their vision-trained twins. On a text quiz needing kind knowledge, the vision-trained models often scored better. Yet deep checks showed the kind knowledge itself barely differed. What changed was how the models handled questions that held kind-linked pairs. The authors conclude vision training helps models use knowledge, rather than rebuild it.
Catalog record
| CREATOR | Yulu Qin, Dheeraj Varghese, Adam Dahlgren Lindström, Lucia Donatelli, Kanishka Misra, Najoung Kim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-13328
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It Yulu Qin, Dheeraj Varghese, Adam Dahlgren Lindström, Lucia Donatelli, Kanishka Misra, Najoung Kim Abstract Does vision-and-language (VL) training change the linguistic representations of language models in meaningful ways? Most results in the literature have shown inconsistent or marginal differences, both behaviorally and representationally. In this work, we start from the hypothesis that the domain in which VL training could have a significant effect is lexical-conceptual knowledge, in particular its taxonomic organization. Through comparing minimal pairs of text-only LMs and their VL-trained counterparts, we first show that the VL models often outperform their text-only counterparts on a text-only question-answering task that requires taxonomic understanding of concepts mentioned in the questions. Using an array of targeted behavioral and representational analyses, we show that the LMs and VLMs do not differ significantly in terms of their taxonomic knowledge itself, but they differ in how they represent questions that contain concepts in a taxonomic relation vs. a non-taxonomic relation. This implies that the taxonomic knowledge itself does not change substantially through additional VL training, but VL training does improve the deployment of this knowledge in the context of a specific task, even when the presentation of the task is purely linguistic. 1 Introduction Humans readily integrate perceptual and linguistic signals to form generalizable mappings from semantic information to language, allowing them to reason about concepts beyond their immediate environment [ 56 , 17 ] . Approaches to concept grounding in AI, which traditionally relied on annotated datasets to specify how language links to people, objects, and events [ 70 , 25 ] , have rapidly shifted in light of the impressive capabilities of vision-language models (VLMs). † † footnotetext: {}^{*},^{\dagger} : Equal contribution; Code can be found at https://github.com/tinlaboratory/taxonomigqa Many standard VLMs [ 32 , 29 , i.a.] often build on top of a pretrained language model (LM) by adding visual conditioning to its next token prediction task, often also updating the parameters of the language model. Analyses of VLM capabilities often focus on the multimodal tasks this additional modality enables. But (how) does this vision-and-language (VL) training change the linguistic capacity of the model? Answering this question requires comparing VL-tuned LMs to their original LM counterparts. Empirical evidence in this literature is rather sparse, often comparing such “VLM-LM minimal pairs” on general benchmarks such as MMLU [ 16 ] and GLUE [ 63 ] . In this paper, we consider a more targeted investigation (like [ 71 ] ) of VLM-LM pairs in a particular domain: lexical-conceptual knowledge, specifically its taxonomic organization (e.g., a cat is an animal ). Evaluation of taxonomic knowledge has been of continued interest within the Natural Language Processing [ 14 , 31 , 42 , 44 ] and Computer Vision communities [ 2 , 61 , 47 ] —however, to the best of our knowledge no work so far has compared minimally differing VLM-LM pairs in terms of how well they can reason taxonomically. To this end, we develop TaxonomiGQA , a synthetically augmented text-only version of the popular visual-question answering (VQA) dataset GQA [ 18 ] , where a subset of WordNet [ 39 ] hierarchy is used to create questions that require taxonomic knowledge. On comparing 7 widely used VLM-LM minimal pairs, we find most VLMs to consistently outperform their LM counterparts, despite the fact that the QA task is text only. We put forth two hypotheses to explain these results. H1: VL training fundamentally alters the (task-agnostic) taxonomic knowledge in LMs; and H2: VL training improves the ability of the LM to deploy its (largely unchanged) taxonomic knowledge in tasks that require its usage. Through a series of controlled behavioral and representational analyses, we find evidence that supports H2 relative to H1. Finally, we conduct a preliminary investigation where we relate the successes of VLMs over LMs to the visual similarities between the hyponym-hypernym categories we have tested in our work. Here we find initial evidence that suggests that VLMs especially perform well at answering questions about hyponym-hypernym pairs that are visually similar, leaving open areas of interesting future research for a more precise characterization of the role of visual input. 2 Related Work Influence of vision on language in VLMs There are two main strands of empirical work measuring the influence of the additional visual modality on models’ linguistic behavior and representations. The first line of work compares VLM and LM performance on downstream text-only benchmarks. The results are mixed: for instance, FLAVA [ 59 ] noted around 8% point gains over the base masked language model on GLUE-style NLP tasks (although the evaluation setting involved finetuning). On the other hand, Molmo has been reported to be outperformed by its base LM, Qwen, on text-only benchmarks like MMLU [ 8 ] . Generally, more evidence exists in favor of multimodal training hurting text-only task performance [ 20 , 36 ] and this observation has been used to argue for freezing the language part of the model during multimodal training [ 12 ] . The second line of work conducts more targeted comparisons of VLMs and LMs, examining whether additional vision training leads to differences in representations of syntactic categories [ 64 ] and performance on tasks that require more “grounding” [ 71 ] , but the findings overall have indicated no substantial differences. Taxonomic knowledge and its deployment Taxonomic knowledge has long been a topic of interest in cognitive psychology [ 34 , 45 ] , and has also often been used to analyze conceptual organization in LMs [ 14 , 31 , 44 , 43 ] . Work that tests its functional consequences, such as property inheritance [ 42 , 57 , 55 ] and inductive generalization [ 41 , 13 ] , found strong evidence that while LMs do learn explicit taxonomic knowledge, they struggle to deploy it in taxonomically sensitive tasks [ 42 ] . Taxonomic knowledge has also been evaluated and analyzed in multimodal models. For instance, Pach et al. 2025 show that the internal structure of neurons in models such as CLIP are often in alignment with existing taxonomies. Our work contributes to this line of work by proposing a level-ground comparison between minimally differing LMs and VLMs, narrowing in on the precise ways in which additional VL training may or may not alter the nature of this knowledge. Figure 1 : The three-step pipeline to create TaxonomiGQA . 3 Behavioral testing of minimal pair VLMs and LMs with TaxonomiGQA The question we are interested in answering concerns the change that VL training introduces to the lexical-conceptual knowledge of a model. This requires a shared evaluation that can be applied to both VLMs and LMs. We discuss below how we designed this evaluation as well as our findings about a range of VLM-LM pairs from this evaluation. 3.1 Dataset design We created a QA dataset that requires taxonomic understanding based on GQA [ 18 ] , named TaxonomiGQA . A datapoint in GQA consists of an image of a scene, a question about this scene, and metadata that includes a scene graph of the objects, their attributes, and relations between the objects. We applied a three-step modification