ICONS: Influence Consensus for Vision-Language Data Selection

Xindi Wu, Mengzhou Xia, Rulin Shao, Zhiwei Deng, Pang Wei Koh, Olga Russakovsky · 2025

Read ICONS: Influence Consensus for Vision-Language Data Selection by Xindi Wu, Mengzhou Xia, Rulin Shao, Zhiwei Deng, Pang Wei Koh, Olga Russakovsky (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00654.txt.

Introduction

Xindi Wu and co-authors present ICONS, a method to pick the most useful training data for vision-language models. Large instruction mixtures carry much repeated material that raises cost with little gain. Past pickers use rough rules that ignore each task. ICONS scores each example by its influence on validation results, using first-order training dynamics.

Scores are then pooled across tasks by majority vote, which favours examples that help many tasks and softens outlier effects. Models trained on just twenty percent of the data kept about ninety-nine percent of full-data performance across three large mixtures. The picked sets also carried to new tasks and new model designs, and compact subsets were released.

Catalog record

CREATORXindi Wu, Mengzhou Xia, Rulin Shao, Zhiwei Deng, Pang Wei Koh, Olga Russakovsky
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-00654

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ICONS: Influence Consensus for Vision-Language Data Selection Xindi Wu, Mengzhou Xia, Rulin Shao, Zhiwei Deng, Pang Wei Koh, Olga Russakovsky Abstract Training vision-language models via instruction tuning relies on large data mixtures spanning diverse tasks and domains, yet these mixtures frequently include redundant information that increases computational costs without proportional gains. Existing methods typically rely on task-agnostic heuristics to estimate data importance, limiting their effectiveness across tasks. We introduce ICONS , a gradient-based I nfluence CON sensus approach for vision-language data S election. Our method leverages first-order training dynamics to estimate each example’s influence on validation performance, then aggregates these estimates across tasks via majority voting. This cross-task consensus identifies consistently valuable data points while mitigating score calibration and outlier sensitivity, enabling robust and scalable data selection for diverse multitask mixtures. Models trained on our selected 20% data subset from LLaVA-665K (respectively: from Cambrian-7M , from Vision-Flan-186K ) retain 98.6% (respectively: 98.8%, 99.8%) of full-dataset performance. We demonstrate that our selected data generalizes to unseen tasks and model architectures, and release three compact subsets LLaVA-ICONS-133K , Cambrian-ICONS-1.4M , and Vision-Flan-ICONS-37K for efficient vision-language model development. 1 Introduction Figure 1 : Influence consensus for vision-language data selection . ( Left ) Given a large-scale visual instruction tuning dataset ( LLaVA-665K ), our method uses majority voting across task-specific influence scores to identify training samples that are consistently influential across multiple tasks, forming a compact 20% subset ( LLaVA-ICONS-133K ) with data points achieving influence consensus. ( Right ) The radar plot compares performance between LLaVA-665K and our selected subset, showing the selected subset achieves comparable results to the full dataset. Visual instruction tuning (VIT) is a crucial step in training vision-language models (VLMs) [ 1 , 2 ] , enabling them to follow language instructions grounded in visual content. Recent approaches rely on large-scale instruction following datasets such as LLaVA-665K [ 1 ] and Cambrian-7M [ 3 ] . While effective, these datasets introduce significant barriers to model training and deployment: prolonged training times [ 4 , 5 ] , high storage demands [ 6 , 7 ] , and substantial compute costs [ 8 , 9 ] . Moreover, not all training samples contribute equally to diverse downstream tasks and naively scaling up diverse data mixtures can introduce redundancy and inefficiency. This raises a fundamental question: Can we identify a compact, multitask-effective subset of training data that preserves model capabilities across tasks while enabling faster experimentation? Prior work has explored various data selection strategies, including gradient-based approaches [ 10 , 11 ] , influence functions [ 12 , 13 ] , and diversity-based sampling [ 14 , 15 ] . However, many of these methods either optimize for single tasks in isolation or maximize source diversity without aligning to downstream needs. In the context of VIT data that supports diverse vision-language tasks, this is particularly limiting: optimizing for one task may hurt generalization, and task-agnostic diversity may dilute impact. Rather than selecting data based on per-task influence, we aim to identify samples that are broadly useful, with training examples that consistently contribute across tasks. To do this, we aggregate gradient-based influence scores using a simple yet effective majority voting approach to reach influence consensus. We introduce ICONS ( I nfluence CON sensus vision-language data S election), a method that builds upon LESS [ 10 ] , a gradient-based selection approach. Given access to validation data for each target task, our method: (1) computes first-order gradient influence scores to measure how each training sample impacts task-specific performance, and (2) reaches influence consensus via majority voting to identify training samples that show consistent positive impact across multiple tasks. This consensus-based mechanism identifies universally valuable training examples: while some samples might be highly influential for individual tasks, we prioritize those that demonstrate broad utility across the task spectrum. While the computational cost of influence estimation is high, this front-loaded, one-time investment yields a standardized, compact dataset that can significantly accelerate development of multimodal models, and enables reusable gradient datastores that amortize costs across iterations and deliver long-term savings. Using ICONS , we create LLaVA-ICONS-133K , Cambrian-ICONS-1.4M , and Vision-Flan-ICONS-37K , automatically selected 20% subsets of LLaVA-665K [ 1 ] , Cambrian-7M [ 3 ] , and Vision-Flan-186K [ 16 ] datasets, respectively. These compact datasets maintain 98.6%, 98.8%, and 99.8% of their original performance across multiple vision-language tasks, providing significant improvements over randomly selecting same-sized subsets (95.8%, 95.4%, and 91.6%) and eliminating approximately two-thirds of the performance drop from shrinking the training data. Moreover, our ICONS outperforms all baselines across different selection ratios, and remarkably achieves above-full-dataset performance, surpassing the original datasets at a 60% selection ratio for LLaVA-665K . Importantly, the selected subsets show strong transferability, e.g., LLaVA-ICONS-133K maintains 95.5-113.9% relative performance across unseen tasks, suggesting that ICONS identifies fundamentally valuable training data. We summarize our key contributions: 1. We propose ICONS , a simple yet effective method for multitask vision-language data selection that identifies broadly valuable training examples via majority voting over task-specific gradient influence scores. 2. Our consensus-based selection outperforms all baselines (§ 4.2 ) and we ablate influence aggregation strategies and show the advantage of voting-based consensus (§ 4.3 ). ICONS exceeds 102% of full-dataset performance at a 60% selection ratio on LLaVA-665K (§ 4.5 ). 3. We release subsets LLaVA-ICONS-133K , Cambrian-ICONS-1.4M and Vision-Flan-ICONS-37K for resource-efficient development, demonstrating competitive performance including generalization to unseen tasks (§ 4.4 ) despite a 5x reduction in training data. 2 Related work Visual instruction tuning. Vision-language models (VLMs), e.g., Flamingo [ 17 ] , LLaVA [ 2 ] , BLIP2 [ 18 ] , and Cambrian [ 3 ] , enable understanding and reasoning across visual and textual modalities for various multimodal tasks. A key component in advancing VLMs is visual instruction tuning (VIT) [ 2 ] , a training process that enables these models to interpret and follow instructions within a vision-language context, transforming them into versatile multimodal assistants. This tuning process not only improves the models’ instruction-following abilities but also aligns their outputs more closely with user expectations, thus enhancing their utility in practical applications [ 2 ] . To support this paradigm, recent VIT datasets have scaled to contain hundreds of thousands to millions of instruction-response pairs [ 1 , 3 ] . Data selection. Data selection methods [ 19 ] can be categorized based on the types of information they utilize for selection. Representation-based approaches [ 20 , 21 ] leverage embeddings to capture data representations. Loss trajectory-based methods [ 22 ] prioritize data points that contribute most significantly to reducing generalization error over training. Gradient-based techniques [ 23 , 10 , 11 ] select data based on gradient information. Recent work has explored various approaches to select optimal visual instruction tuning datasets. Concurrent work TIVE [ 24 ] employs gradient-based selection to identify representative instances. TIVE assumes that the number of specialist data should be proportional to task difficulty and thus samples specialist data based on an estimation of task difficulty. Our method does not rely on this assumption – we directly select samples that benefit the greatest number of tasks. COINCIDE [ 21 ] clusters data based on representations associated with concept-skill compositions. Our work follows

Read the complete text