Instructify: Demystifying Metadata to Visual Instruction Tuning Data Conversion
Jacob Hansen, Wei Lin, Junmo Kang, Muhammad Jehanzeb Mirza, Hongyin Luo, Rogerio Feris, Alan Ritter, James Glass, Leonid Karlinsky · 2025
Read Instructify: Demystifying Metadata to Visual Instruction Tuning Data Conversion by Jacob Hansen, Wei Lin, Junmo Kang, Muhammad Jehanzeb Mirza, Hongyin Luo, Rogerio Feris, Alan Ritter, James Glass, Leonid Karlinsky (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18115.txt.
Introduction
Jacob Hansen and co-authors open up a data making pipeline in a 2025 paper. Training vision language models takes huge sets of image chats, yet most sets are built ad hoc. Makers often pay for closed model services to turn image labels into chat form, which costs money and blocks sharing. Their tool, Instructify, converts plain image metadata into tuning data with open methods.
The paper lays out how the conversion works step by step, where earlier sets stay vague. Open code and clear records let other teams rebuild or extend the data. Cheaper, shared pipelines could widen who can train strong vision models. The work treats data craft itself as a research product worth publishing.
Catalog record
| CREATOR | Jacob Hansen, Wei Lin, Junmo Kang, Muhammad Jehanzeb Mirza, Hongyin Luo, Rogerio Feris, Alan Ritter, James Glass, Leonid Karlinsky |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18115
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Instructify: Demystifying Metadata to Visual Instruction Tuning Data Conversion Jacob Hansen, Wei Lin, Junmo Kang, Muhammad Jehanzeb Mirza, Hongyin Luo, Rogerio Feris, Alan Ritter, James Glass, Leonid Karlinsky Abstract Visual Instruction Tuning (VisIT) data, commonly available as human-assistant conversations with images interleaved in the human turns, are currently the most widespread vehicle for aligning strong LLMs to understand visual inputs, converting them to strong LMMs. While many VisIT datasets are available, most are constructed using ad-hoc techniques developed independently by different groups. They are often poorly documented, lack reproducible code, and rely on paid, closed-source model APIs such as GPT-4, Gemini, or Claude to convert image metadata (labels) into VisIT instructions. This leads to high costs and makes it challenging to scale, enhance quality, or generate VisIT data for new datasets. In this work, we address these challenges and propose an open and unified recipe and approach, Instructify , for converting available metadata to VisIT instructions using open LLMs. Our multi-stage Instructify features an efficient framework for metadata grouping, quality control, data and prompt organization, and conversation sampling. We show that our approach can reproduce or enhance the data quality of available VisIT datasets when applied to the same image data and metadata sources, improving GPT-4 generated VisIT instructions by 3% on average and up to 12% on individual benchmarks using open models, such as Gemma 2 27B and LLaMa 3.1 70B. Additionally, our approach enables effective performance scaling - both in quantity and quality - by enhancing the resulting LMM performance across a wide range of benchmarks. We also analyze the impact of various factors, including conversation format, base model selection, and resampling strategies. Our code, which supports the reproduction of equal or higher-quality VisIT datasets and facilities future metadata-to-VisIT data conversion for niche domains, is released at https://github.com/jacob-hansen/Instructify . Figure 1 : Overview of Instructify. We present a unified framework that automatically transforms diverse metadata from publicly available datasets into multimodal instruction-tuning conversations. Our approach merges metadata from multiple sources, categorizing it into captions, bounding boxes, and question-answer (QA) pairs. While LLMs effectively convert captions and QA into natural language, we find that grounded annotations - such as bounding boxes - benefit significantly from a hierarchical representation using ASCII tree, capturing the geometric and semantic structure of the image. The serialized metadata then undergoes an iterative instruction-generation process, where context is refined and quality is rigorously controlled, with an LLM in the loop. 0 0 footnotetext: \dagger Work done during internship at MIT-IBM Watson AI Lab 0 0 footnotetext: \ddagger Correspondence: [email protected] 1 Introduction Large Multimodal Models (LMMs) have enabled significant advancements, allowing billion-parameter-scale language models to effectively perform multimodal tasks such as comprehensive image captioning [ 29 , 71 ] , visual question answering [ 14 , 39 , 27 ] and complex visual reasoning [ 39 , 27 , 91 , 79 , 13 ] . The advancement of LMMs, however, has increasingly relied on a distillation paradigm, wherein high-performing proprietary systems, such as GPT-4 [ 51 ] , are leveraged to generate training data through carefully designed prompts. While these systems enable powerful visual understanding and instruction-following capabilities, the field’s reliance on closed-source models introduces substantial limitations. The top-performing open-weight LMMs have increasingly used data that is either closed-source [ 85 , 4 ] or collected by commercial models [ 39 , 89 , 9 , 10 , 35 ] . This creates a landscape where transparency and reproducibility are restricted. This reliance on proprietary data pipelines leads to several core challenges: (1) Repeatability suffers due to the lack of open, accessible code and documentation for data collection, making it difficult to replicate results accurately. (2) Generalizability is limited as existing methods often rely on dataset-specific configurations, reducing their applicability across diverse domains. (3) Scalability is hindered by the high costs and restrictive access associated with proprietary APIs. Existing approaches to Visual Instruction Tuning (VisIT) data generation are largely ad-hoc, developed by individual groups with minimal standardization. These methods often rely on closed-source model APIs like GPT-4 [ 51 ] , Gemini [ 68 ] , or Claude [ 1 ] to convert image metadata into VisIT instructions [ 9 , 10 , 89 , 70 , 7 ] , leading to high costs and scalability issues. In this paper, we present a complete, open-source platform which we name Instructify, designed to overcome these limitations, enabling high-quality, scalable, and reproducible data generation for visual instruction tuning. An overview is provided in Figure 1 . Specifically, our Instructify consists of four primary steps. First, Data Loading and Organization manages a wide range of datasets and groups metadata by image source. Metadata Conversion convert the metadata of different types into textual descriptions. Information Management and Quality Control enhances data quality and diversity by filtering redundant information and employing LLM-based quality checks. Eventually Prompt Management dynamically selects prompt templates, adjusting for different instruction styles and task requirements based on the metadata context. Our contributions are threefold: (1) we introduce a unified, open-source framework that consolidates metadata across diverse datasets and standardizes prompt engineering strategies, thereby enhancing replication and accessibility . (2) Our system is architected for scalability , achieving vertical scaling through a high-throughput backend and horizontal scaling using a file-based control system for distributed processing. (3) We reduce reliance on closed-source models by demonstrating that high-quality synthetic data can be generated with open models, such as Gemma 2 27B [ 67 ] and LLaMA 3.1 70B [ 48 ] , achieving comparable or superior quality to data generated by proprietary systems like GPT-4. Our extensive experiments show that our Instructify not only matches but often enhances the data quality found in existing VisIT datasets, improving GPT-4-generated instructions by 3% on average (over 12 benchmarks) and up to 12% on individual benchmarks. Furthermore, we explore the impact of various factors, including conversation format, base model selection, and resampling strategies, on data quality and performance across benchmarks. By providing a reproducible, adaptable and scalable solution, our work lays a foundation for advancements in visual instruction tuning. Our codebase, allowing for the reproducibility of existing datasets as well as expansion to niche domains, will be open-sourced for open, transparent research in LMM development. 2 Related Work Large Multimodal Models (LMMs). Both academia and industry have made significant strides in developing Large Multimodal Models (LMMs) that aim to match the multimodal reasoning capabilities of top-performing proprietary models such as GPT-4o [ 50 ] , Gemini [ 68 ] , and Claude [ 1 ] . The LLaVA series [ 39 , 37 , 38 ] introduces a framework for multimodal instruction tuning, leveraging vision-language alignment through instruction-following data. These models incrementally expand the number of visual tokens during training to enhance vision-related reasoning and image comprehension. Math-LLaVA [ 62 ] extends this approach to mathematical reasoning, integrating a specialized dataset, MathV360K, and fine-tuning a LLaVA-1.5-based model [ 37 ] to improve performance on mathematical benchmarks. VILA2 [ 16 ] focuses on iterative self-refinement, where the model continuously re-annotates its own pretraining data, retraining from scratch with these refined annotations to enhance performance. This method contrasts with traditional approaches that rely on external proprietary models or large-scale web crawled data. Molmo [