Once-For-All: A Train-Once and Select-Anytime Framework for Multimodal Instruction Tuning

1 st Mingkang Dong *, 2 nd Hongyi Cai *, 3 rd Xiwen Lei, 4 th Jie Li, 5 th Tao Zhang, 6 th Muxin Pu † · 2026

Read Once-For-All: A Train-Once and Select-Anytime Framework for Multimodal Instruction Tuning by 1 st Mingkang Dong *, 2 nd Hongyi Cai *, 3 rd Xiwen Lei, 4 th Jie Li, 5 th Tao Zhang, 6 th Muxin Pu † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26761.txt.

Introduction

Dong and seven colleagues ask whether a data-selection signal for multimodal instruction tuning can be learned once and reused across datasets and models. Their OnceSelect selector encodes image-instruction pairs in a fixed joint space, groups them into coarse semantic clusters, and picks low-confidence samples as the most informative. Using just 15 percent of LLaVA-625K, it keeps 99.2 percent of full-data performance, and it transfers to unseen datasets and models without retraining.

Catalog record

CREATOR1 st Mingkang Dong *, 2 nd Hongyi Cai *, 3 rd Xiwen Lei, 4 th Jie Li, 5 th Tao Zhang, 6 th Muxin Pu †
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45633
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26761

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Once-For-All: A Train-Once and Select-Anytime Framework for Multimodal Instruction Tuning Thanks: * These authors contributed equally to this work. Thanks: † Corresponding author. 1 st Mingkang Dong *, 2 nd Hongyi Cai *, 3 rd Xiwen Lei, 4 th Jie Li, 5 th Tao Zhang, 6 th Muxin Pu † Abstract Multimodal instruction tuning is the de facto training for adapting vision language models (VLMs), yet instruction data are highly redundant, making data selection critical for training time efficiency and cost. Existing methods derive selection criteria from a specific model or dataset, therefore whenever the target model or candidate pool changes, the criteria must be recomputed from scratch at substantial cost. To address this, we propose Once-for-All (OFA) Data Filtering Framework, a data selection framework that trains a reusable selector once and applies it to any dataset or model without recomputation. OFA clusters multimodal instructions in a frozen text and image embedding space, taggs every aggregated cluster as pseudo-labels, serving to initiate the ability of selectors by training on it. Once finished, the frozen selector transfers directly across datasets and model scales. The selector is trained once on a seed large-scale image-text datasets and applied both to LLaVA-665K and, without any retraining, to the unseen Vision-Flan-186K. Selecting only 15% of the data, OFA achieves 98.3% of full data performance across 10 downstream benchmarks; As well as Vision-Flan-186K, the model fine-tuned by the transferred selector surpasses full data training by 10.6%, confirming that the learned signal generalizes to datasets never seen during selector training. To verify generalization, both Qwen2.5-VL-3B and LLaVA-v1.5-7B are also trained without per model recomputation, decoupling selection from the target model. These results demonstrate that a single, transferable selector provides an effective and reusable solution for efficient multimodal instruction tuning. I Introduction Vision Language Models (VLMs) have become the dominant paradigm for multimodal understanding, achieving strong performance across captioning, visual question answering, and instruction following [ 15 , 4 , 29 ] . Their instruction following capabilities are largely acquired by fine-tuning on large scale image text datasets such as LLaVA-665K [ 15 ] and Vision-Flan [ 31 ] . However, these datasets are massive and verbose: curating and training on them is computationally expensive, and the prevailing assumption that more data yields better models has increasingly been doubted. Recent work indicates that such large instruction datasets are highly redundant and, in some cases, even incorrectly annotated [ 5 ] . What ultimately drives consistent performance is the diversity and quality of the training set rather than its sheer quantity. A growing body of work therefore studies data efficiency through data selection , which can be broadly grouped into two types. Gradient based methods [ 20 , 30 ] estimate each sample’s influence on training by tracking gradient alignment or training dynamics, while scoring based methods [ 5 , 13 , 32 ] rank samples using signals such as model loss, perplexity, or learned quality scores. Complexity driven approaches such as Self-Filter [ 3 ] go further, training a scoring network jointly with the target VLM to rank instructions by difficulty. These methods share two practical drawbacks. Gradient based approaches are computationally costly, as they require per sample gradient computation over the full candidate pool; scoring based and complexity driven approaches are tightly coupled to the specific model or dataset on which their signals are computed, so the scores do not transfer once the setting changes. This exposes a question that is rarely asked: is data selection practical in real settings, where datasets and target models change frequently? For existing methods, this remains out of reach: whenever the dataset or the target model changes, the selection signal must be recomputed from scratch, often at a cost comparable to training itself. We instead pursue an once-for-all approach: a selector that is trained once and can then select valuable data at any time , across diverse multimodal instruction tuning datasets and across target models of different scales and architectures. Specifically, OFA encodes image-instruction pairs in a frozen joint multimodal space, clusters them to obtain pseudo labels that capture the data distribution, and trains a lightweight selector for only a few epochs; samples on which this deliberately under-trained selector remains least confident are retained as the most informative, while the rest are discarded. Since the selector is trained independently of any target VLM, the same frozen selector transfers seamlessly to new datasets and new models without any recomputation. Our contributions are summarized as follows: • We propose OFA, a data selection framework for multimodal instruction tuning that trains a selector once and applies it for all datasets and target models without recomputation, in contrast to prior methods whose selection signals are coupled to a specific model or dataset. • We studied what qualitative data that really benefit in VLMs by identifying low selector confidence samples. Since this depends only on the data and not on any target model, it generalizes to datasets and architectures never seen during selector training. • Extensive experiments demonstrate that data selected by the frozen OFA selector achieves strong performance on both unseen datasets and unseen model architectures while using only a fraction of the data, confirming its once-for-all transferability. II Related Work II-A Multimodal Instruction Tuning To adapt pre-trained multimodal models to downstream tasks and domain specific scenarios, instruction tuning has become the standard practice, fine-tuning a pre trained backbone on large collections of image instruction response pairs so that it learns to follow user intent [ 15 ] . As this paradigm has matured, however, the scale of instruction datasets has grown rapidly, with collections such as LLaVA-665K [ 15 ] and Vision-Flan [ 31 ] reaching hundreds of thousands of samples, and the training time and computational resources demanded by such corpora have become a pressing concern. Moreover, recent studies observe that these datasets contain a large portion of redundant and even incorrectly paired image text samples [ 5 ] , causing the marginal benefit of additional data to diminish as the dataset grows. Selecting a subset of the most valuable and diverse samples has therefore become crucial for efficient and effective instruction tuning. II-B Data Selection for Instruction Tuning To improve data efficiency, a growing body of work studies how to select a compact yet informative and diverse subset from large instruction datasets. Existing efforts can be broadly divided into gradient based and scoring based approaches. Gradient-based methods. Gradient-based methods estimate the importance of training samples by analyzing their influence on the model’s loss or parameter updates. Representative approaches, such as ICONS [ 30 ] and XMAS [ 20 ] , track gradient alignment or training dynamics to identify samples that maximize the model’s learning trajectory. While these methods are theoretically well-grounded and provide high-fidelity importance scores, they suffer from a severe practical limitation: the influence signal is inherently coupled to the specific model architecture and its current parameter state. Consequently, whenever the target model, the task objective, or even the initial checkpoint changes, these signals must be recomputed from scratch. Performing per-sample back-propagation over a massive candidate pool for every new training configuration is computationally prohibitive, rendering these methods difficult to scale to modern multimodal instruction tuning settings. Scoring-based methods. Scoring-based methods

Read the complete text