A Prescription for Better VLMs through Data Curation Alone
2026
Read A Prescription for Better VLMs through Data Curation Alone (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11405.txt.
Introduction
The DatologyAI team asks how far data curation alone can lift vision-language models, holding architecture, recipe and compute fixed while varying only training data. Applied to the MAmmoTH-VL single-image subset, their pipeline lifts scores by 11 point 7 points on average across twenty public benchmarks. At two billion parameters the curated model beats InternVL3.5-2B by 9 point 9 points at about seventeen times less training compute, and nearly matches Qwen3-VL-2B at about eighty-seven times less compute.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83226 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11405
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Prescription for Better VLMs through Data Curation Alone Abstract Data curation has shifted the quality–compute frontier for language-model and contrastive image-text pretraining, but its role for vision-language models (VLMs) is far less established. Here, we ask how far data curation alone can take VLM performance, holding architecture, training recipe, and compute fixed and varying only the training data. Our pipeline, applied to the MAmmoTH-VL single-image subset, lifts performance by + 11.7 +11.7 pp on average across 20 public VLM benchmarks (spanning grounding, VQA, OCR and documents, captioning, spatial and 3D, counting, charts, math, brand-ID, and multi-image reasoning) and by + 11.3 +11.3 pp on average across all nine capability axes of DatBench ( Joshi et al. 2026 ) , our high-fidelity, comprehensive VLM eval suite. At 2B, our curated model surpasses InternVL3.5-2B by 9.9 9.9 pp at ∼ 17 × \sim 17\times less training compute and closes the gap to Qwen3-VL-2B to within 1.8 1.8 pp at ∼ 87 × \sim 87\times less compute, all from pretraining alone. Beyond strong gains in accuracy, our curation pipeline delivers four further properties: (1) Reliability : per-capability standard deviation across training seeds drops by ∼ 67 % \sim 67\% and the curated-vs-baseline lift survives a context-length sweep from 4k to 16k tokens; (2) OOD generalization : the 9-eval OOD average rises by + 7.2 +7.2 pp, and multi-image reasoning on BLINK rises by + 3.09 +3.09 pp despite single-image-only training, with Visual Correspondence gaining + 11.8 +11.8 pp despite demanding cross-image reasoning; (3) Behavioral gains beyond benchmarks : across ∼ 1,100 {\sim}1{,}100 open-ended queries the curated 2B is more honest and more specific than the matched-compute baseline , and answers more concisely and refuses fewer benign queries than a frontier 2B reference; (4) Pareto-dominance on inference cost : at every scale tested (1B, 2B, 4B) the curated model raises accuracy while lowering response FLOPs against the matched-compute baseline , and the curated 4B matches near-frontier accuracy at 3.3 × 3.3\times lower response FLOPs than Qwen3-VL-4B. Our pipeline shows that data curation is a high-leverage tool for building better VLMs, reaching near-frontier accuracy at up to ∼ 150 × \sim 150\times less training compute. \projectname 20/20 Vision Language Models: \projectlogo figures/VLM_logo.png \teambyline DatologyAI Team 1 1 1 See Contributions and Acknowledgements (§ 9 ) for full author list. Figure 1 : Curation matches frontier 2B/4B VLMs at a fraction of the training compute. Pareto frontier of error rate ( 1- avg. accuracy) vs. training FLOPs on (left) the 20-eval public VLM benchmark suite and (right) DatBench . DatologyAI-curated models (blue) at 1B, 2B, and 4B Pareto-dominate the matched-compute MAmmoTH-VL baseline (gray) at every scale, and approach Qwen3-VL, InternVL3.5, and Qwen3.5 (extensively post-trained, up to \sim 150\times more compute) on both suites. 1 Introduction In the era of deep learning, curating training data has been a key lever for improvements across modalities. The efficacy of training data curation has been extensively studied in language-model pretraining: curated open corpora (DCLM, FineWeb, RedPajama, Dolma, Nemotron-CC) can shift the quality–compute Pareto frontier by as much as architectural or scale choices ( Li et al. 2024a ; Penedo et al. 2024 ; Weber et al. 2024 ; Soldaini et al. 2024 ; Su et al. 2024a ; DatologyAI 2024b ; Maini and DatologyAI Team 2025 ) . Data filtering and mixture design have also become first-class axes in contrastive image-text training ( Gadre et al. 2023 ; Xu et al. 2023 ; Fang et al. 2024 ; DatologyAI 2024a ) . In vision-language model research, by contrast, the training mixture is typically characterized only at the level of “X billion image–text pairs” rather than treated as a primary design variable, with most attention paid to non-data variables such as encoder choice, connector design, resolution handling, visual instruction tuning, and reinforcement learning ( Liu et al. 2023a ; Chen et al. 2024d ; Wang et al. 2024a ; Tong et al. 2024a ; McKinzie et al. 2024 ; Deitke et al. 2024 ; Yu et al. 2024 ; Shen et al. 2025 ; Liu et al. 2025b ) . Holding architecture, training recipe, and compute fixed, we measure the impact of pretraining data curation alone on VLM capability. Prior work on VLM data curation has typically isolated a single axis at a time: multimodal deduplication ( Slyman et al. 2024 ; Liu et al. 2025a ) , image-text quality filtering ( Wang et al. 2024b ; Wang et al. 2025b ; Chen et al. 2024b ) , data-mixture design ( McKinzie et al. 2024 ; Tong et al. 2024a ; Laurençon et al. 2024b ; Deitke et al. 2024 ; Shi et al. 2024 ; Chen et al. 2024c ; Wang et al. 2025a ) , recaptioning of paired data ( Chen et al. 2023 ; Li et al. 2024b ) , task-agnostic synthetic data ( Liu et al. 2024c ; Su et al. 2024b ; Liao et al. 2025 ) , or task-specific synthetic data ( Joshi et al. 2025b ; Yang et al. 2025 ) . Others release curated corpora without a matched-compute uncurated baseline ( Guo et al. 2024 ) . We instead compose joint image-and-text deduplication, joint image-and-text filtering, target-distribution matching for mixture design, and both task-agnostic and task-specific synthetic data into a single end-to-end pipeline, and ask how far data curation alone can take VLMs. Concretely, we run our end-to-end data curation pipeline on the single-image subset (10M) of MAmmoTH-VL-12M ( Guo et al. 2024 ) and train models across 1B–4B language backbones (Qwen3 LM ( Qwen Team 2025a ) with SigLIP2 vision encoder ( Tschannen et al. 2025 ) ). We compare against a baseline trained on the same uncurated MAmmoTH-VL subset under identical training recipe and compute, and against SOTA public models at similar scales: the Qwen3-VL family (2B/4B) ( Qwen Team 2025b ) , the InternVL3 and InternVL3.5 families (2B/4B) ( Chen et al. 2024d ) , and Qwen3.5 (2B/4B) ( Qwen Team 2026 ) . These are strong reference points: trained on up to \sim\!150\times more compute and including extensive post-training (instruction tuning, RLHF, RLVR) on top of their pretraining stage. At 2B, our curation lifts the average across 20 public VLM benchmarks by +11.7 pp , spanning grounding, VQA, OCR and documents, captioning, spatial and 3D reasoning, counting, charts, math, brand-ID, and multi-image reasoning, and the average across all nine capability axes of DatBench by +11.3 pp . Figure 1 previews the cost-quality Pareto: the curated models Pareto-dominate the matched-compute baseline at every scale tested (1B, 2B, 4B), and approach extensively post-trained references at up to \sim 150\times less training compute. All results below are post-decontamination, applied to baseline and curated runs alike; the multimodal decontamination methodology is detailed in Appendix A . We organize the paper around five key findings on the impact of data curation for VLMs: • Curation shifts the cost-quality Pareto frontier across model scales. (§ 4 ) Across the 20-eval public suite, curation improves accuracy at every tested scale: +13.0 pp at 1B, +11.7 pp at 2B, and +14.0 pp at 4B. Across DatBench ’s nine capability axes, the gains are similarly large, reaching +11.3 pp at 2B and increasing with scale. Moreover, these