Concepts or Skills? Rethinking Instruction Selection for Multi-modal Models

Andrew Bai, Justin Cui, Ruochen Wang, Cho-Jui Hsieh · 2025

Read Concepts or Skills? Rethinking Instruction Selection for Multi-modal Models by Andrew Bai, Justin Cui, Ruochen Wang, Cho-Jui Hsieh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10339.txt.

Introduction

Andrew Bai, Justin Cui, Ruochen Wang, and Cho-Jui Hsieh published this paper in 2025. It asks a practical question about training vision-language models: should the extra lessons match the test's ideas, or its skills? A benchmark may reward knowing the same objects, or doing the same kind of task.

The authors found that benchmarks split cleanly along that line. Their method first reads a target benchmark and judges which side it leans toward. It then picks training instructions that match on that side, either shared concepts or shared skills. Across more than ten benchmarks, the targeted picks beat the best existing selection method on average. The gain grew on skill-heavy tests. The lesson is that one selection rule cannot fit all: the trade-off must be set per target.

Catalog record

CREATORAndrew Bai, Justin Cui, Ruochen Wang, Cho-Jui Hsieh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-10339

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Concepts or Skills? Rethinking Instruction Selection for Multi-modal Models Andrew Bai, Justin Cui, Ruochen Wang, Cho-Jui Hsieh Abstract Vision-language instruction tuning achieves two main purposes: learning visual concepts and learning visual skills. In this paper, we found that vision-language benchmarks fall into the dichotomy of mainly benefiting from training on instructions with similar skills or visual concepts. Inspired by the discovery, we designed a simple targeted training data selection method to optimize the performance of a given benchmark. We first extract the concepts/skills from the benchmark, determine whether the benchmark predominantly benefits from similar concepts or skills, and finally select instructions with the most matching concepts/skills. Experiments on 10+ benchmarks validate the effectiveness of our targeted data selection method, showing +0.9% over the best existing baseline averaged over all benchmarks and +1.5% on the skill-focused subset. Our findings underscore the importance of recognizing the inherent trade-off within instruction selection, which requires balancing the acquisition of conceptual knowledge against visual skill. 1 Introduction Recent progress in vision-language modeling has been driven by the modular combination of large pretrained language models (LLMs) with powerful visual encoders, typically connected via a modality adapter that transforms visual features into a format compatible with the LLM’s input space. This architecture allows models to leverage the linguistic capabilities of LLMs while incorporating rich visual understanding from pretrained vision backbones. However, simply joining these components is insufficient for strong multimodal performance; extensive continual pretraining on paired vision-language data is often required to bridge the modality gap. In addition, vision-language instruction tuning—where the model is further trained on curated prompts that teach specific multimodal capabilities—has become essential for aligning the joint model’s behavior with desired visual-linguistic tasks. Vision-language instruction tuning has emerged as a powerful framework for aligning multimodal models with human-desired behaviors, especially in tasks requiring both visual perception and linguistic reasoning. This tuning process serves two major purposes: first, it helps models learn to associate visual representations with the corresponding textual concepts, and second, it enables the acquisition of new visual capabilities such as counting objects, reasoning about spatial relations, and inferring physical properties. These dual roles are critical for deploying vision-language models in real-world applications where generalization to both seen and unseen instructions is required. Yet, despite the widespread adoption of instruction tuning, the vision-language benchmarks used for evaluation often vary in the types of capabilities they measure. Some emphasize concept recognition (e.g., visual question answering), while others emphasize reasoning or skill-based tasks (e.g., counting, relational understanding, or commonsense inference). This discrepancy raises a fundamental question: Should instruction selection prioritize alignment with the skills required by a task or the concepts present in the visual content? More concretely, can performance on a benchmark be improved by tuning on a dataset that emphasizes similar skills versus one that emphasizes similar visual concepts? To investigate this, we conduct a systematic analysis across a diverse range of vision-language benchmarks, aiming to understand how different types of instruction similarity influence downstream performance. Our study reveals that existing benchmarks fall into two distinct categories: those that benefit more from instructions involving similar skills (e.g., object counting across different domains) and those that benefit more from instructions involving similar concepts (e.g., object or scene categories common to the benchmark). Notably, this pattern persists across architectures and training regimes, suggesting that the division is intrinsic to the nature of the tasks and not merely an artifact of the model design. Motivated by these findings, we propose a simple yet effective targeted instruction selection method that adapts to the nature of each benchmark. Specifically, we first extract the dominant concepts and skills present in a given evaluation set using automated instruction parsing and skill/concept taxonomy alignment. We then determine whether the benchmark is concept-dominant or skill-dominant using validation performance differentials, and finally, we select training instructions that most closely align with the benchmark’s dominant type. This targeted strategy allows the model to focus on the most relevant inductive biases, whether conceptual or procedural. We evaluate our approach across more than ten standard vision-language benchmarks, covering a wide range of task types and difficulty levels. Our results demonstrate consistent performance gains, with an average improvement of 0.9% over the strongest baseline using untargeted instruction tuning. These findings underscore the importance of benchmark-aware instruction selection and open new directions for task-adaptive multimodal learning. 2 Related Work 2.1 Vision Language Model Large language models (LLMs) Dubey et al. 2024 ; OpenAI ; Team et al. 2024 ; Chiang et al. 2023 have demonstrated remarkable performance across a wide variety of tasks. This success is largely attributed to pretraining on trillions of tokens, followed by post-training techniques such as reinforcement learning with human feedback (RLHF) Christiano et al. 2017 . Building on their capabilities in the text domain, recent research has extended LLMs to handle additional modalities such as images. MiniGPT-4 Zhu et al. 2023 integrates a pretrained Vision Transformer (ViT) backbone with a Q-Former and a single linear projection layer, combining it with the Vicuna language model to achieve strong performance on multimodal tasks. Concurrently, InstructBLIP Dai et al. 2023 employs a similar approach to generate instruction-following responses conditioned on both images and text prompts. LLaVA Liu et al. 2023b ; Liu et al. 2024 converts a large language model into a multimodal model by first encoding images with a CLIP Radford et al. 2021 encoder, then mapping the visual features into the text embedding space through a linear MLP, forming a simple yet effective integration strategy. 2.2 Visual Instruction Data Selection Dataset selection Har-Peled and Mazumdar 2004 ; Roux et al. 2012 ; Campbell and Broderick 2018 ; Mirzasoleiman et al. 2020 has been extensively explored to enhance the training efficiency of models. Recently, this line of research has been extended to vision-language models, aiming to reduce training costs while preserving performance. Coincide Lee et al. 2024 partitions the dataset into numerous subsets and retains samples based on the transferability of clusters. ICONS Wu et al. 2024 identifies important samples by measuring the influence of individual data points, defined via gradient similarity with a validation set. Prism Bi et al. 2025 introduces a training-free method that utilizes Pearson correlation analysis to measure the intrinsic visual encoding capabilities of MLLMs. In contrast, PreSel Safaei et al. 2025 approaches the problem differently: it first applies a filtering mechanism to identify high-quality images, and only then generates instructions for those selected samples. Despite significant progress in achieving strong performance with reduced data, there is limited research on how different tasks are affected by underlying concepts or skills. 2.3 Concepts vs Skills We distinguish sharply between the dichotomy: concepts , which refer to the visual entities, attributes, and objects present in an image, i.e., what appears, and skills , which encapsulate the reasoning operations or judgment strategies necessary for correctly interpreting or answering questions about those entities, i.e., how to analyze them). This distinction echoes the formal view of compositionality in VQA proposed by Whitehead et al. 2021 , who explicitly model a skill–concept decomposition to enable generalization across unseen combinations of skills (e.g. “color” judgment) and concepts (e.g. “car”). While concepts ground models in visual content, skills capture the latent reasoning patterns—such as counting, spatial inference, or trend

Read the complete text