COMPACT: COMPositional Atomic-to-Complex Visual Capability Tuning
Xindi Wu, Hee Seung Hwang, Polina Kirichenko, Olga Russakovsky · 2025
Read COMPACT: COMPositional Atomic-to-Complex Visual Capability Tuning by Xindi Wu, Hee Seung Hwang, Polina Kirichenko, Olga Russakovsky (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21850.txt.
Introduction
Xindi Wu and co-authors present a recipe for training vision-language models on fewer, richer examples, in a 2025 paper. Training sets for visual instruction tuning have grown huge, yet few ask how much each sample teaches. Their recipe, COMPACT, builds one training item that stacks several basic visual skills into a single rich question about an image. Complexity, not raw count, is the lever.
Applied to a large standard set, the recipe cut the data budget by ninety percent while matching the full set's score across eight tests. On the hardest tests, the lean set even beat full training by clear margins. The authors argue that smart synthetic data can replace bulk in teaching models to see and reason.
Catalog record
| CREATOR | Xindi Wu, Hee Seung Hwang, Polina Kirichenko, Olga Russakovsky |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-21850
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
COMPACT: COMPositional Atomic-to-Complex Visual Capability Tuning Xindi Wu, Hee Seung Hwang, Polina Kirichenko, Olga Russakovsky Abstract Multimodal Large Language Models (MLLMs) excel at simple vision-language tasks but struggle when faced with complex tasks that require multiple capabilities, such as simultaneously recognizing objects, counting them, and understanding their spatial relationships. This might be partially the result of the fact that Visual Instruction Tuning (VIT), a critical training step for MLLMs, has traditionally focused on scaling data volume, but not the compositional complexity of training examples. We propose COMPACT ( COMP ositional A tomic-to-complex Visual C apability T uning), which generates a training dataset explicitly controlling for the compositional complexity of the training examples. The data from COMPACT allows MLLMs to train on combinations of atomic capabilities to learn complex capabilities more efficiently. Across all benchmarks, COMPACT achieves comparable performance to the LLaVA-665K VIT while using less than 10% of its data budget, and even outperforms it on several, especially those involving complex multi-capability tasks. For example, COMPACT achieves substantial 83.3% improvement on MMStar and 94.0% improvement on MM-Vet compared to the full-scale VIT on particularly complex questions that require four or more atomic capabilities. COMPACT offers a scalable, data-efficient, visual compositional tuning recipe to improve on complex visual-language tasks. 1 Introduction Multimodal Large Language Models (MLLMs) have shown impressive progress in handling a wide range of vision-language tasks [ 2 , 1 , 19 ] . Yet, from early diagnostic works [ 30 , 32 ] to recent state-of-the-art models like LLaVA [ 22 , 23 ] , Cambrian [ 37 ] , and Eagle [ 34 , 21 ] , the compositionality challenge remains persistent. Consider the following question: “Are there more blue squares or red circles on the image?” A model that is capable of recognizing shapes, colors and counting objects should be able to answer it correctly. Despite years of progress, state-of-the-art models still fail on such compositional questions, even though they can answer simpler ones correctly (e.g. “What color is the square?”). This has been a long‑standing issue and such failures suggest that current models do not systematically generalize to tasks with higher compositional complexity. To address this, recent efforts have primarily scaled up the amount of training data used for Visual Instruction Tuning (VIT) [ 37 , 24 , 25 , 34 , 21 ] , an essential but data- and compute-heavy step for MLLM training. However, such datasets (e.g. LLaVA-665K [ 24 ] ) are dominated by simple queries that require only one capability, lacking sufficient compositional complexity (Fig. 1 ). Even with large-scale instruction tuning, recent studies show that models still struggle with integrating capabilities and generalizing to complex visual tasks due to limitations in the compositional complexity of their training data [ 29 , 40 ] . Thus, we ask: Can we bridge the performance gap between different complexity regimes without simply increasing the data volume? Figure 1 : Compositional Complexity Comparison. Comparison between visual instruction tuning data ( LLaVA-665K [ 24 ] VIT) and our compositional tuning data ( COMPACT ). The VIT data is dominated by simple queries ( k=1 ), while our COMPACT data is balanced across compositional complexity levels ( k=1,2,3 ). Instead of treating compositionality as a byproduct of scale, we encourage compositional capabilities in MLLMs with hierarchically structured compositional training data. In this work, we introduce COMPACT ( COMP ositional A tomic-to-complex Visual C apability T uning), a data recipe that scales capabilities of MLLMs from atomic ( k=1 ) to composite ( k>1 ) complexity levels. We define a set of 10 atomic capabilities and then combine them to generate a compositional training dataset that can promote model’s internalization of the compositional structures of complex tasks in a compute-efficient manner. We summarize our key contributions: • We introduce COMPACT , a visual compositional tuning data recipe that builds complex capabilities from simple atomic capabilities. By systematically combining 10 atomic capabilities to control the complexity of training samples, COMPACT addresses a key limitation of conventional VIT methods that rely on incidental capability composition through data scaling. • We develop a structured data recipe that enforces a balanced distribution across different levels of compositional complexity ( k=1,2,3 ) to cover a wider range of task regimes. This approach flattens the complexity cliff in conventional VIT datasets [ 24 ] , where 90.1% of the questions require two or fewer capabilities. • We demonstrate the impressive effectiveness of COMPACT . With only 10% of the size of the VIT dataset (5% of the LLaVA-665K [ 24 ] VIT data augmented with 32K samples of our compositional tuning data), training with COMPACT matches the performance of full-scale VIT (100.18% relative score). Further, it demonstrates exceptional generalization to higher-complexity tasks, improving the score from 35.3 to 64.7 on MMStar [ 6 ] and from 32.5 to 57.5 on MMVet [ 43 ] for k=4 tasks. Figure 2 : COMPACT ’s Data Generation Pipeline. (Left): We sample k\in\{1,2,3\} atomic capabilities such as color, object recognition, and spatial relationship. (Center): We generate questions that integrate all k sampled capabilities. (Right): We verify the quality of generated conversations and combine them with instruction tuning data to maintain instruction following capability. This structured data recipe explicitly models atomic-to-complex learning procedure, in contrast to standard LLaVA-665K [ 24 ] VIT that promotes learning from simple queries. 2 Related Work Visual Instruction Tuning. Instruction following is an essential capability in language models [ 38 , 46 ] . Misalignment between a model’s response and the format requested by a question can hinder the precise evaluation of its performance and capabilities [ 11 , 12 , 31 , 3 ] . In order to adapt MLLMs to respond appropriately to diverse question formats (e.g., multiple-choice, short- and long-response questions), Visual instruction tuning [ 22 , 24 ] has been proposed. VIT involves training a model on a fixed set of instruction patterns that can be repeated during inference. Although VIT has shown performance improvements in general multimodal capabilities [ 14 ] , recent work [ 10 ] has shown that optimizing for response formatting potentially limits the quality of language model responses. While VIT [ 24 ] focuses on learning simple capabilities via instruction following, our data recipe explicitly models compositional capabilities in the training data. Our approach directly addresses the lack of exposure to compositional questions during training, enabling models to improve on tasks that are more complex in capability space. COMPACT leverages the learning potential of both compositional tuning and instruction tuning data to create a more optimal data recipe than conventional VIT. Compositionality in LLMs and MLLMs. Semantically, compositionality is the claim that the meaning of a complex statement is a result of the combination of its constituents. [ 8 ] . In the context of visual capabilities of MLLMs, compositional capability refers to a model’s ability to perform complex tasks by combining multiple capabilities [ 13 ] , where each capability is related to understanding basic visual concepts such as objects, attributes, relationships. Recent work has shown that compositional capability can be trained in LLMs [ 42 , 45 ] , but generalizations to the realms of MLLMs have been largely incomplete. Some studies