Multimodal LLMs Do Not Compose Skills Optimally Across Modalities
2025
Read Multimodal LLMs Do Not Compose Skills Optimally Across Modalities (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08113.txt.
Introduction
Paula Ontalvilla, Aitor Ormazabal, and Gorka Azkune test whether multimodal language models can combine skills learned in different channels. A task may need a reading skill plus an image skill used in sequence. The team built three evaluations that each require composing two modality dependent skills, then tried several open models in direct and cascaded modes.
Every model showed a clear cross modality composition gap, even when the two step structure was enforced by hand. Chain of thought prompting and a targeted fine tuning recipe improved scores, yet the gap stayed large. Better ways to join skills across text and vision remain an open need.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-08113
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal LLMs Do Not Compose Skills Optimally Across Modalities Abstract Skill composition is the ability to combine previously learned skills to solve new tasks. As neural networks acquire increasingly complex skills during their pretraining, it is not clear how successfully they can compose them. In this paper, we focus on Multimodal Large Language Models (MLLM), and study their ability to compose skills across modalities. To this end, we design three evaluation tasks which can be solved sequentially composing two modality-dependent skills, and evaluate several open MLLMs under two main settings: i) prompting the model to directly solve the task, and ii) using a two-step cascaded inference approach, which manually enforces the composition of the two skills for a given task. Even with these straightforward compositions, we find that all evaluated MLLMs exhibit a significant cross-modality skill composition gap . To mitigate the aforementioned gap, we explore two alternatives: i) use chain-of-thought prompting to explicitly instruct MLLMs for skill composition and ii) a specific fine-tuning recipe to promote skill composition. Although those strategies improve model performance, they still exhibit significant skill composition gaps, suggesting that more research is needed to improve cross-modal skill composition in MLLMs. Keywords: Evaluation Methodologies, Training, Fine-tuning, Adaptation, Alignment, and Representation Learning, Skill Composition, Cross-modality, Multimodality Multimodal LLMs Do Not Compose Skills Optimally Across Modalities Paula Ontalvilla, Aitor Ormazabal, Gorka Azkune HiTZ Center - Ixa, University of the Basque Country (UPV/EHU) {paula.ontalvilla, aitor.ormazabal, gorka.azcune}@ehu.eus Abstract content 1. Introduction Multimodal Large Language Models (MLLM) have become increasingly capable in recent years, receiving ever-growing interest from the research community and industry ( 1 ; 10 ; 24 ; 28 ) . However, even as these models become capable of solving more and more complex tasks, the mechanisms through which they do so remain unclear, spawning extensive research into their internal workings ( 7 ) . MLLMs are often trained using a capable LLM as a starting point, and visual or other multimodal capabilities are added by further fine-tuning it on modality-specific tasks. While some work has studied the effects of this type of training on the original model’s textual capabilities ( 20 ; 6 ) , the resulting MLLM’s ability to coherently combine its visual and textual skills is not well understood. In this work, we pose the following question: Can MLLMs successfully compose skills across modalities? More specifically, we devise 3 image-to-text tasks that can be solved by composing a visual skill with a textual one, in a way that would be trivial for a human: 1) Reasoning over questions rendered in an image (OCR + textual reasoning), 2) Object counting (object identification + counting), and 3) Card playing (Card identification + textual reasoning over the game state). By comparing the model’s natural performance on these tasks with a cascaded inference approach that forces skill composition by making two calls to the same model, we can measure a “cross-modality skill composition gap”. We measure this gap on a wide array of popular open-source MLLMs and datasets and find that, strikingly, all tested models exhibit a significant cross-modality skill composition gap , even when the required combination is straightforward 1 1 1 https://github.com/paulaonta/skillComposition . Having demonstrated the existence of this gap, we explore two possible mitigation strategies: one leveraging a prompt specifically constructed to induce the relevant skill composition and a second approach exploring fine-tuning to induce cross-modal skill composition. We show that the specially constructed prompt partially reduces the skill composition gap, but it suffers from lack of scalability as it requires a task-specific prompt. Fine-tuning also improves the results and it contributes to generalization to some extent, yet the gap remains. Our results show that MLLMs do not optimally combine their visual and textual skills even in very straightforward scenarios, prompting further research into the root cause and effective mitigation of this phenomenon. Figure 1: We define three tasks that can be solved with a trivial sequential composition of two modality-specific skills. We evaluate MLLMs on those tasks using direct inference, i.e. standard inference. We compare the performance with the cascaded inference, where we manually induce the required skill composition with two calls to the MLLM. 2. Related Work 2.1. Skill composition in neural networks There are different ways to understand and approach skills and skill composition in the literature, ranging from robotics 29 ; 25 to LLMs 40 . We will cover the literature of the domains that are closest to our work. In the artificial agent community, skills are defined as separated atomic tools that an agent can use at inference time via API calls 41 ; 26 . Thus, skill composition is the ability of those agents to compose the tools available to solve different tasks. To achieve skill composition 41 focus on self-improvement techniques, whereas 26 rely on an LLM to plan a sequence of tools to be used to approach a given task. The LLM community offers some other views on skill composition. 40 defines skill composition as the capacity to combine learned skills in novel ways that are not encountered during training. They used the SkillMix dataset 37 to show that LLMs can learn to compose skills to some extent through specific fine-tuning. In this framework, the skills they consider are literary, rhetorical, reasoning, logic, theory of mind, pragmatics, common sense and physical knowledge . We also understand skills as specific abilities learned by neural networks during training. However, rather than focusing on diffuse skill categories and complex compositions for text-only LLMs as in 37 , we focus on the multimodal scenario, where visual and textual skills can be sharply divided, and study straightforward scenarios that can be solved by combining one visual skill with one textual skill. This allows us to measure the cross-modality skill composition gap, and study how well MLLMs are able to compose skills across modalities. 2.2. Multimodal Large Language Models (MLLM) In this paper, we will focus on Multimodal Large Language Models (MLLMs) that receive visual and textual input and generate textual output. Following the current dominant paradigm, we consider open weights models that follow the late fusion paradigm, where a pretrained visual encoder and LLM are connected and further trained for vision-language instruction following. Examples of this paradigm are the LLaVA family 22 , Llama 3.2 13 , Molmo 9 and Qwen2.5-VL 2 . We consider open-source MLLMs to offer an ideal test bench to study skill composition, for two reasons. On the one hand, modality-specific skills can be easily distinguished and tracked to their training stages. For example, current MLLMs are usually trained for Optical Character Recognition (OCR), a visual skill (see § 3 ). Similarly, modern pretrained text-only LLMs are trained for arithmetic reasoning, a textual skill. On the other hand, the dominant late-fusion paradigm adds a modality specific (e.g. visual) encoder later in training, raising questions of how well the final model is able to combine skills from both modalities. As the acquisition of those skills depends on the training data, we do not consider closed models, since it is impossible to know what skills they were explicitly trained for. 3. Skill Composition and Multimodal Large Language Models In this section, we describe how we understand skills and skill composition, and how