Caption First, VQA Second: Knowledge Density, Not Task Format, Drives Multimodal Scaling
Hongjian Zou, Yue Ge, Qi Ding, Yixuan Liao, Xiaoxin Chen · 2026
Read Caption First, VQA Second: Knowledge Density, Not Task Format, Drives Multimodal Scaling by Hongjian Zou, Yue Ge, Qi Ding, Yixuan Liao, Xiaoxin Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13054.txt.
Introduction
Zou and four colleagues argue that multimodal scaling stalls not on task format but on thin knowledge in training data. Task-specific training such as visual question answering adds little beyond image captions, whose content can reconstruct it with tiny loss. Richer captions plus cross-modal knowledge instead lift performance across benchmarks, as semantic coverage predicts gains better than task diversity.
Catalog record
| CREATOR | Hongjian Zou, Yue Ge, Qi Ding, Yixuan Liao, Xiaoxin Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70419 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.13054
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Caption First, VQA Second: Knowledge Density, Not Task Format, Drives Multimodal Scaling Hongjian Zou, Yue Ge, Qi Ding, Yixuan Liao, Xiaoxin Chen Abstract Multimodal large language models (MLLMs) have achieved rapid progress, yet their scaling behavior remains less clearly characterized and often less predictable than that of text-only LLMs. Increasing model size and task diversity often yields diminishing returns. In this work, we argue that the primary bottleneck in multimodal scaling is not task format, but knowledge density in training data. We first show that task-specific supervision such as Visual Question Answering (VQA) contributes little incremental semantic information beyond image captions: VQA signals can be reconstructed from captions with negligible performance loss. We then demonstrate that increasing knowledge density—through structured caption enrichment and cross-modal knowledge injection—leads to consistent performance improvements across multimodal and downstream benchmarks. Across controlled experiments, performance correlates more strongly with semantic coverage than with task diversity. These findings suggest that current MLLMs fail to scale primarily because training data lacks sufficient knowledge coverage. We advocate for knowledge-centric multimodal training as a principled foundation for scalable multimodal models. † † affiliationnotice: Equal contribution 1 Introduction Multimodal large language models (MLLMs) have achieved rapid progress across vision–language tasks. However, unlike text-only large language models, their scaling behavior remains weaker and less predictable. Modern text LLMs are typically trained on tens of trillions of tokens, whereas current multimodal models rely on substantially smaller multimodal corpora, often on the order of only a few trillion multimodal tokens. Moreover, existing multimodal supervision frequently encodes limited semantic coverage per training sample. As a result, increasing model size, expanding task diversity, or incorporating additional supervision often yields diminishing returns. This raises a fundamental question: what drives scaling in multimodal systems? A common assumption is that stronger task supervision leads to stronger reasoning ability. In practice, Visual Question Answering (VQA) has become a default form of multimodal supervision for both pretraining and instruction tuning. Originally introduced as a benchmark for visual reasoning, VQA datasets are now widely treated as essential training signals. This implicitly suggests that to improve multimodal reasoning, one must increase task complexity or diversity. However, closer inspection reveals a different picture. Many VQA questions restate information already present in image captions, and the corresponding answers often require little more than extracting objects, attributes, or simple relations described in natural language. In other words, VQA supervision frequently contributes task formatting rather than new semantic content. This observation suggests that the limitation of current MLLMs may not lie in insufficient task diversity, but in insufficient knowledge coverage. This insight becomes particularly important when viewed through the lens of scaling. Despite the dominant role of scaling laws in advancing text-only LLMs, similar scaling behavior has not clearly emerged in multimodal models. Modern VLMs are trained on far fewer multimodal tokens than their text-only counterparts, and increasing multimodal data size does not consistently produce proportional performance gains. This discrepancy indicates that multimodal scaling may depend on a different governing factor. We argue that this factor is knowledge density —the amount and diversity of semantic information contained in training data. Much of today’s multimodal supervision, especially VQA annotations, provides limited incremental knowledge beyond captions. Captions naturally encode objects, relations, contextual cues, and commonsense information, whereas VQA primarily reshapes this knowledge into an interaction format. If knowledge content, rather than task format, determines scaling behavior, then improving multimodal models requires increasing knowledge density rather than increasing task diversity. This perspective motivates the central thesis of this paper: caption supervision is primary, while task-formatted supervision is secondary; the key bottleneck is knowledge density, not task format. We validate this thesis in two stages. First, we demonstrate that VQA supervision can be replaced or reconstructed from captions with negligible performance loss, indicating that captions subsume most task-relevant information. Second, we show that increasing knowledge density through cross-modal knowledge injection leads to consistent scaling improvements. Together, these results suggest that current MLLMs fail to scale not because tasks are insufficient, but because training data lacks sufficient knowledge coverage. We advocate for knowledge-centric multimodal training as a principled foundation for scalable multimodal models. 2 Related Work 2.1 Multimodal Large Language Models Multimodal Large Language Models extend text-only foundation models to jointly process visual and linguistic inputs, enabling unified reasoning across modalities ( 1 ; 12 ; 15 ) . Most modern MLLMs adopt a modular architecture that combines a pretrained vision encoder with a large language model through cross-modal alignment modules. Early systems ( 1 ; 12 ) demonstrated that frozen backbones with lightweight connectors can efficiently transfer linguistic priors to multimodal settings, while more recent approaches ( 9 ; 5 ; 4 ; 20 ) move toward large-scale joint pretraining and interleaved image–text modeling. Representative model families illustrate different design emphases in multimodal learning. The Qwen-VL series ( 3 ; 17 ; 5 ; 4 ) integrates strong language backbones with scalable multimodal alignment, emphasizing multilingual capability, instruction following, and deployment efficiency. In contrast, the InternVL family ( 8 ; 9 ; 7 ; 20 ; 18 ) focuses on high-fidelity visual perception and fine-grained cross-modal representation, prioritizing visual tokenization and perception-intensive reasoning. These efforts demonstrate rapid progress in multimodal architecture design and system capability. Despite these advances, existing MLLMs primarily emphasize architectural scaling and task-level supervision. Comparatively less attention has been devoted to the semantic properties of multimodal training data and their role in scaling behavior. In particular, current pipelines rely heavily on task-formatted supervision such as VQA, while the contribution of semantic coverage and knowledge density remains underexplored. Our work shifts the focus from architectural scaling to knowledge-centric scaling, examining how semantic density in training data is associated with multimodal model performance. 2.2 Supervision Paradigms: Captioning vs. Visual Question Answering Multimodal training commonly relies on task-structured supervision to align visual and linguistic representations. Among existing paradigms, image captioning and visual question answering (VQA) represent two fundamentally different forms of supervision. Image captioning provides declarative supervision in which visual content is described through natural language sentences that capture objects, attributes, relations, and scene context. Large-scale caption datasets such as MSCOCO Captions ( 13 ; 6 ) enabled early progress in vision–language learning, and caption supervision remains a core component of many modern multimodal training pipelines. VQA, originally introduced as a benchmark for visual reasoning ( 2 ) , frames supervision as question–answer interactions grounded in images. In recent years, VQA-style supervision has become a dominant training signal in many multimodal systems. Models such as LLaVA ( 15 ; 14 ) , Qwen-VL ( 3 ) , and InternVL ( 9 ) rely heavily on question–answer style multimodal instruction data, often constructed from VQA datasets or synthetic VQA generation. The widespread adoption of VQA-style supervision reflects a common assumption in multimodal learning: that stronger task-structured supervision can improve visual reasoning ability. By framing supervision as explicit questions requiring grounded answers, VQA training is often viewed as encouraging models to reason over visual content rather than merely describe it. However, the distinction between semantic content and interaction format has received limited attention. In particular, it remains unclear whether task-form supervision introduces additional semantic knowledge beyond declarative descriptions such as captions. In this work, we revisit this assumption from a data-centric perspective and