Understanding Task Transfer in Vision-Language Models

Bhuvan Sachdeva, Karan Uppal, Abhinav Java, Vineeth N. Balasubramanian · 2025

Read Understanding Task Transfer in Vision-Language Models by Bhuvan Sachdeva, Karan Uppal, Abhinav Java, Vineeth N. Balasubramanian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18787.txt.

Introduction

Bhuvan Sachdeva and three co-authors study task transfer in vision language models: fine tuning a model on one perception task, such as depth or counting, can unpredictably help or hurt others. Specialized perception still lags behind humans, so knowing these interactions matters for efficient training.

They introduce a normalized metric for the change in performance after transfer and use it to score each task's breadth and strength of influence. Across three open models and thirteen perception tasks, a transfer graph reveals groups of tasks that aid each other and risks of negative interference. The metric can guide which data to train on next.

Catalog record

CREATORBhuvan Sachdeva, Karan Uppal, Abhinav Java, Vineeth N. Balasubramanian
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-18787

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understanding Task Transfer in Vision-Language Models Bhuvan Sachdeva, Karan Uppal, Abhinav Java, Vineeth N. Balasubramanian Abstract Vision-Language Models (VLMs) perform well on multimodal benchmarks but lag behind humans and specialized models on visual perception tasks like depth estimation or object counting. Finetuning on one task can unpredictably affect performance on others, making task-specific finetuning challenging. In this paper, we address this challenge through a systematic study of task transferability. We examine how finetuning a VLM on one perception task affects its zero-shot performance on others. To quantify these effects, we introduce Perfection Gap Factor (PGF), a metric that captures both the breadth and magnitude of transfer. Using three open-weight VLMs evaluated across 13 perception tasks, we construct a task-transfer graph that reveals previously unobserved relationships among perception tasks. Our analysis uncovers patterns of positive and negative transfer, identifies groups of tasks that mutually influence each other, organizes tasks into personas based on their transfer behavior and demonstrates how PGF can guide data selection for more efficient training. These findings highlight both opportunities for positive transfer & risks of negative interference, offering actionable guidance for advancing VLMs. Understanding Task Transfer in Vision-Language Models Figure 1 : One finetune, many fates: Finetuning Qwen-2.5-VL 32B on perception tasks creates a structured map of transfer capabilities. (The list of perception tasks considered can be found in Table 2 .) 1 Introduction Vision Language Models ( 23 ; 22 ; 18 ; 2 ; 19 ; 35 ) have demonstrated significant progress in understanding visual information in recent years, as reflected in their performance across well-known benchmarks such as MMMU ( 38 ) , DocVQA ( 27 ) , InfoVQA ( 26 ) , and TextVQA ( 32 ) . Visual instruction finetuning ( 23 ) has helped adapt Large Language Models (LLMs) to parse visual input by finetuning a small number of parameters to align a visual encoder (e.g., CLIP ( 29 ) ) with a given LLM backbone. Despite this progress, careful analysis has shown that VLMs fall short in many visual understanding tasks, most often due to their limitations in visual perception 10 ; 34 . Understanding the limits of VLMs on visual perception tasks, especially ones that are natural to humans and serve as building blocks that scaffold on to more complex visual tasks remains an urgent need, in order to provide foundational solutions to robust visual processing. Figure 2 : PGF Heatmaps for Qwen-2.5-VL model family (3B, 7B, 32B). VLMs lag behind humans and specialist models on basic perception tasks such as depth estimation, object detection, and counting. For example, on the BLINK ( 10 ) leaderboard, the top performing models (GPT-4o at 60.04% and GPT-4V at 51.14%) achieve modest average performance compared to humans (95%). This has motivated practitioners to finetune VLMs using parameter-efficient methods such as LoRA ( 15 ) on curated, task-specific datasets to improve performance on perception tasks. However, little is known about how such finetuning affects a model’s other perception capabilities, particularly in modern foundation models used predominantly in zero-shot settings. Understanding this phenomenon is essential for both improving robustness and characterizing how VLMs adapt and generalize. In this work, we address this gap by investigating the following question, which to our knowledge has not been previously studied: How does finetuning a VLM on one perception task transfer to its zero-shot performance on other perception tasks? Prior work on task relationships in vision has largely focused on transfer learning involving finetuning on both source and target tasks ( 39 ; 6 ) . Other efforts have explored pretraining strategies and their downstream effects ( 31 ; 33 ) . In contrast, we study zero-shot cross-task transfer in VLMs: how a single-task finetuning intervention reshapes performance across a diverse set of perception tasks. We introduce a new metric, the Perfection Gap Factor (PGF), to quantify task transferability—defined as the extent to which finetuning on a source task improves or degrades zero-shot performance on a target task. PGF captures both the breadth (how many tasks are affected) and magnitude (the strength of the effect) of transfer, enabling fair comparison across models with very different baselines. Using PGF, we conduct a systematic study across three model scales (3B, 7B, 32B) and thirteen perception tasks. Our analysis reveals several properties of tasks, including scale-dependent sharpening of transfer, distinct behaviors across perception granularity levels, and natural clustering among mutually beneficial tasks. We further extend our study to video models and observe similar trends across model sizes. Finally, we demonstrate that PGF can guide principled selection of training data subsets to improve finetuning efficiency and reduce performance regressions. Key Contributions. Our contributions are as follows: • Systematic Study : We are the first to analyze how finetuning on one visual perception task affects zero-shot performance on a broad suite of other perception tasks. • Perfection Gap Factor : PGF measures both the magnitude and breadth of transfer effects while normalizing for heterogeneous task difficulties and model baselines. • Task Properties : We uncover consistent structural properties of transfer, including scale-dependent sharpening, task type dependent transfer (granularity and perceptual), and the emergence of mutually beneficial task clusters. • Evaluation beyond images : We evaluate cross-task transfer in multimodal models trained on video, showing that our key findings generalize to the temporal domain. • Downstream use of PGF : We demonstrate that PGF can be used to identify beneficial source tasks and construct data subsets that improve finetuning efficiency while mitigating negative transfer. 2 Related Work In this section, we present a detailed description of adjacent literature. Benchmarks for Visual Perception. Several benchmarks ( 20 ; 13 ; 27 ; 26 ) have been introduced to evaluate the progress of VLMs on visual understanding. Notably, MMMU ( 38 ) is a large-scale benchmark assessing model capabilities across 30 subjects spanning technology, engineering, art, medicine, and more. Despite its broad coverage, MMMU largely focuses on examination-style question answering and lacks core perception tasks. Other benchmarks, such as DocVQA ( 27 ) and InfoGraphicsVQA ( 26 ) , combine OCR capabilities with visual understanding, while ChartQA ( 25 ) evaluates a model’s ability to parse complex charts and draw meaningful inferences. Visual Commonsense Reasoning ( 40 ) presents a challenging task in which models must reason about the intention or consequences of actions depicted in a scene. MathVista ( 24 ) is a multiple-choice mathematics dataset that requires interpreting figures or diagrams to answer questions, and NLVR ( 36 ) tests a broad range of linguistic phenomena through image captioning tasks. While each of these datasets highlights specific aspects of multimodal understanding, most do not explicitly measure the visual perception capabilities that humans perform naturally. BLINK ( 10 ) , in contrast, aggregates over 13 datasets spanning diverse tasks and serves as a central benchmark for our experiments. Transferability Studies. The Taskonomy framework ( 39 ) first introduced a framework for modeling the structure of computer vision tasks through transfer learning. 39 pretrain an encoder on a source task and then perform transfer learning on a task-specific decoder, estimating of transferability between tasks. However, this study is restricted to the pre foundation model era and is primarily conducted using CNN-based vision encoders and small decoders.

Read the complete text