Should VLMs be Pre-trained with Image Data?
Sedrick Keh* Jean Mercat* Samir Yitzhak Gadre Kushal Arora Igor Vasiljevic Benjamin Burchfiel Shuran Song Russ Tedrake Thomas Kollar Ludwig Schmidt Achal Dave, Toyota Research Institute, Columbia University, Stanford, MIT, {sedrick.keh,jean.mercat}@tri.global · 2025
Read Should VLMs be Pre-trained with Image Data? by Sedrick Keh* Jean Mercat* Samir Yitzhak Gadre Kushal Arora Igor Vasiljevic Benjamin Burchfiel Shuran Song Russ Tedrake Thomas Kollar Ludwig Schmidt Achal Dave, Toyota Research Institute, Columbia University, Stanford, MIT, {sedrick.keh,jean.mercat}@tri.global (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07603.txt.
Introduction
Sedrick Keh and ten co-authors ask whether vision-language models should meet images early in training rather than only after text training is done. Their paper came out in 2025. The usual recipe trains a language model fully, then adds image training in a second phase.
The team trained models with different mixes of image and text data, at different scales, and introduced vision at different points. Mixing images into pre-training from the start gave better vision-language results while keeping text skills strong. For a mid-size model, adding vision late but not at the very end still beat adding it only after full text training. The gains were modest but consistent across six varied tasks. The study helps builders decide when images should enter a model's training diet.
Catalog record
| CREATOR | Sedrick Keh* Jean Mercat* Samir Yitzhak Gadre Kushal Arora Igor Vasiljevic Benjamin Burchfiel Shuran Song Russ Tedrake Thomas Kollar Ludwig Schmidt Achal Dave, Toyota Research Institute, Columbia University, Stanford, MIT, {sedrick.keh,jean.mercat}@tri.global |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-07603
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Should VLMs be Pre-trained with Image Data? Sedrick Keh* Jean Mercat* Samir Yitzhak Gadre Kushal Arora Igor Vasiljevic Benjamin Burchfiel Shuran Song Russ Tedrake Thomas Kollar Ludwig Schmidt Achal Dave, Toyota Research Institute, Columbia University, Stanford, MIT, {sedrick.keh,jean.mercat}@tri.global Abstract Pre-trained LLMs that are further trained with image data perform well on vision-language tasks. While adding images during a second training phase effectively unlocks this capability, it is unclear how much of a gain or loss this two-step pipeline gives over VLMs which integrate images earlier into the training process. To investigate this, we train models spanning various datasets, scales, image-text ratios, and amount of pre-training done before introducing vision tokens. We then fine-tune these models and evaluate their downstream performance on a suite of vision-language and text-only tasks. We find that pre-training with a mixture of image and text data allows models to perform better on vision-language tasks while maintaining strong performance on text-only evaluations. On an average of 6 diverse tasks, we find that for a 1B model, introducing visual tokens 80% of the way through pre-training results in a 2% average improvement over introducing visual tokens to a fully pre-trained model. Figure 1: An overview of our VLM pre-training data recipe. We investigate data mixes and design choices for text-only pre-training, image-text pre-training, and fine-tuning. Note that while we depict "LLM Pre-training" and "Image-text Pre-training" as two separate steps in this diagram, in practice, we continuously transition from the first stage to the second. 1 Introduction Numerous work have extensively studied how to train a VLM from a pre-trained LLM ( Liu et al., 2024 ; Alayrac et al., 2022 ; Bavishi et al., 2023 ; Li et al., 2024a ; Karamcheti et al., 2024 ; Tong et al., 2024 ; Beyer et al., 2024 ; Laurençon et al., 2024 ; Lu et al., 2024 ; Wu et al., 2024a ; Wu et al., 2024b ; Chen et al., 2025 ) . Most existing VLMs follow roughly the same high-level recipe: 1) Start with a pre-trained LLM, 2) continue training intermediate layers on image-text tokens (e.g., Flamingo, Pali-Gemma, Idefics, DeepSeek-VL) or align an image projection layer (e.g., LLaVA, Cambrian, Prismatic), and 3) fine-tune the model on multi-task instructions or chat templates. Notably, training with images included in the initial pre-training stage (i.e., step 1) is largely undocumented in these papers, even though a number of large models (with undocumented training procedures) are known to be natively multimodal (e.g., Pixtral, Gemini, Fuyu). In this paper, we investigate the significance of each step in the pipeline for downstream performance on vision-language and text benchmarks. Additionally, we examine whether increasing the amount of image data in one step can reduce the data requirements in other steps. To understand when image data should be introduced to VLM training, we train a suite of 300 models over various numbers of parameters, varying the amount of text-only pre-training data, as well as the amount, type, and ratio of image pre-training data (Figure 1 ). Our experiments suggest several key findings: First, incorporating image data during pre-training generally helps, especially after a model has seen many text tokens. However, the timing and method of introducing visual information are also important. Specifically, we find that it is beneficial to add image data before the LLM is fully pretrained, during the cooldown phase. Our strategy to interrupt the text pre-training at 80% completion and add image-text data outperforms the popular alternative of fully training the LLM, then re-warming and cooling the model with image data (Section 3.1 ). Second, the fraction of visual data introduced during cooldown is another key parameter for strong performance across domains. At the 1B parameter regime, our experiments reveal that 10% to 20% of tokens should be visual. Going above or below this ratio results in worse downstream performance. However, this fraction appears to be a function of scale. At the smaller 79M parameter regime, larger fractions of visual data are preferred (Sections 3.2 and 3.3 ). Third, the timing of when to introduce instruction fine-tuning image tokens is crucial for downstream performance, both for pure text and vision-language tasks. We observe that mixing together instruction fine-tuning during the image-text pre-training process actively hurts the model (Section 3.4 ). Meanwhile, adding instruction tokens during fine-tuning improves vision-language task performance up to 4 epochs at the 1B parameter scale, at the cost of slightly hurting performance on text-only tasks (Section 3.5 ). 2 Experimental Setup Figure 2: The commonly used framework we apply to add vision capabilities to a transformer model. Our experimental setup is designed to study the impact of image-text data on the model’s downstream results on both text and vision tasks. To this end, we implement a commonly used model architecture ( Liu et al., 2023 ; Tong et al., 2024 ; McKinzie et al., 2024 ; Beyer et al., 2024 ; Bai et al., 2023 ; Li et al., 2023a ; Laurençon et al., 2024 ) consisting of a pre-trained image encoder, a projection block, and a decoder-only transformer. Unless stated otherwise, the image encoder is SigLIP 400M ( Zhai et al., 2023 ) , the projection block is a two-layer MLP, and the transformer is the 1.4B parameter model described in Gadre et al. (2024b) . Our training procedures (Section 2.1 ) are applied to the model prior to its downstream evaluation (Section 2.2 ). 2.1 Training Procedure We train all our models with a three-step process illustrated in Figure 1 : 1. Partial text-only pre-training 2. Image-text pairs mixed with text-only continued pre-training 3. Image-text pairs multi-task fine-tuning We can view the first and second steps as a single pre-training stage, where we introduce images before the model has finished pre-training, as opposed to previous methods where images are introduced to a model which had already been pre-trained. In practice, this is implemented by resuming from the same learning rate schedule (Section 2.1.2 ). A more detailed comparison overview of different models can be found in Table 1 . Table 1: The conventional approach to VLM training first fully trains a language model on text-only data, and adds images in a second or third stage of training. In this work, we instead introduce image data earlier in pre-training. To do this efficiently, we resume a language model during the course of its pre-training at various stages (e.g., 20% of the way through training) and introduce images midway through training. Model Text-Only Pre-training Image-Text Pre-training Multitask Fine-Tuning BLIP3 ( Xue et al., 2024 ) Fully pre-trained (Phi3-mini) Re-warmup Caption and interleaved text-image data; no pure text Re-warmup Flamingo ( Alayrac et al., 2022 ) Fully pre-trained (closed model) Re-warmup Caption and interleaved text-image data; no pure text (Skipped) IDEFICS ( Laurençon et al., 2024 ) Fully pre-trained (Mistral-7B-v0.1) Re-warmup Interleaved text-image data; no pure text Re-warmup MM1 ( McKinzie et al., 2024 ) Fully pre-trained (closed model) Re-warmup Various image-text ratios (100::0, 91::9, 86::14, 66::33) Re-warmup DeepSeek-VL / DeepSeek-VL2 ( Lu et al., 2024 ; Wu et al., 2024b ) Fully pre-trained (DeepSeek) Two-stage (re-warmup then re-warmup) Multitask; 30::70 image-text ratio Re-warmup Qwen-VL ( Bai et al., 2023 ) Fully pre-trained (Qwen) Two-stage (re-warmup then re-warmup) Caption then