All in One: A Unified Synthetic Data Pipeline for Multimodal Video Understanding
Tanzila Rahman, Renjie Liao, Leonid Sigal, [0.5em] University of British Columbia · 2026
Read All in One: A Unified Synthetic Data Pipeline for Multimodal Video Understanding by Tanzila Rahman, Renjie Liao, Leonid Sigal, [0.5em] University of British Columbia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12335.txt.
Introduction
Rahman and two colleagues propose SynMulti, a single pipeline that auto-generates unlimited multimodal video data with rich labels for training video-understanding models. Real video annotation is costly, slow, and narrow in scope, so the pipeline spans counting, question answering, and segmentation in one system. A question-answering fine-tuning plan pushes models past captions toward deeper visual grounding. Models trained mostly on synthetic data generalize well to real datasets, often beating rivals trained the old way.
Catalog record
| CREATOR | Tanzila Rahman, Renjie Liao, Leonid Sigal, [0.5em] University of British Columbia |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35069 |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.12335
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
All in One: A Unified Synthetic Data Pipeline for Multimodal Video Understanding Tanzila Rahman, Renjie Liao, Leonid Sigal, [0.5em] University of British Columbia Abstract Training multimodal large language models (MLLMs) for video understanding requires large-scale annotated data spanning diverse tasks such as object counting, question answering, and segmentation. However, collecting and annotating multimodal video data in real-world is costly, slow, and inherently limited in diversity and coverage. To address this challenge, we propose a unified synthetic data generation pipeline capable of automatically producing unlimited multimodal video data with rich and diverse supervision. Our framework supports multiple task formats within a single pipeline, enabling scalable and consistent data creation across tasks. To further enhance reasoning ability, we introduce a VQA-based fine-tuning strategy that trains models to answer structured questions about visual content rather than relying solely on captions or simple instructions. This formulation encourages deeper visual grounding and reasoning. We evaluate our approach in three challenging tasks: video object counting, video-based visual question answering, and video object segmentation. Experimental results demonstrate that models trained predominantly on synthetic data generalize effectively to real-world datasets, often outperforming traditionally trained counterparts. Our findings highlight the potential of unified synthetic data pipelines as a scalable alternative to expensive real-world annotation for multimodal video understanding. 1 Introduction Figure 1 : Comparison of real and synthetic data for training video foundation models. Real data (left) requires manual annotation, making it costly and labor-intensive, while synthetic data (right) is automatically generated, offering a scalable and efficient alternative for creating high-quality training datasets. Multimodal large language models (MLLMs) have demonstrated remarkable progress in understanding and reasoning over visual and textual data. Recent advances in vlm models have enabled a wide range of applications, including image captioning 69 ; 44 ; 42 , visual question answering (VQA) 49 ; 82 ; 16 , and video understanding 59 ; 11 ; 80 . These successes are largely driven by the scale of the models and the training data. However, the acquisition of large-scale, high-quality multimodal annotations remains a fundamental bottleneck. This challenge is particularly acute for video-based tasks, which require dense supervision across both spatial and temporal dimensions, including frame-level annotations, object tracking, and temporally consistent labeling 65 ; 47 . Such requirements significantly increase the cost, complexity, and time associated with dataset construction, making it difficult to scale real-world data collection. Moreover, real-world datasets often suffer from biases, uneven coverage of reasoning patterns, and limited diversity, which restrict the ability of models to learn generalizable multimodal reasoning skills 73 ; 90 . Data quality is a critical factor in determining the effectiveness of modern machine learning systems. Although real-world data are ubiquitous, they are often noisy, incomplete, biased, or expensive to annotate. In addition, when sensitive information is involved (e.g., medical or personal data), privacy constraints further limit the availability and usability of data 20 ; 94 . These challenges have motivated growing interest in synthetic data, artificially generated data designed to mimic the statistical properties of real data are a powerful alternative to traditional data collection. Unlike real data, synthetic data can be generated at scale with precise control over quality, balance, and diversity while also avoiding privacy concerns. However, for synthetic data to be effective, it must be plausible and capture the underlying distributions and structural patterns of real-world data. Early approaches such as Random OverSampling and SMOTE 17 attempted to address data imbalance through simple transformations, but their ability to model complex dependencies was limited. More recent advances in deep generative models, including Variational Autoencoders (VAEs) 21 and Generative Adversarial Networks (GANs) 30 , have enabled the generation of high-fidelity synthetic data across a wide range of domains, particularly in computer vision. Despite these advances, most existing multimodal LLMs still rely heavily on real-world datasets with manual annotations in the form of captions, instructions, or aligned image–text pairs 63 ; 55 ; 60 ; 79 ; 93 ; 19 . Although effective to some extent, these annotation strategies are limited in diversity and often fail to capture complex spatial–temporal relationships or higher-order reasoning patterns. Furthermore, the availability of such annotated datasets is uneven across tasks, modalities, and domains, creating a major bottleneck for scaling and generalizing multimodal models, especially in video-centric settings. To address these challenges, we propose a synthetic data generation pipeline that enables the creation of unlimited multimodal data with rich and structured annotations. By leveraging controllable simulation environments and automated labeling mechanisms, our pipeline produces high-quality synthetic datasets without the need for costly human annotations. This approach allows precise control over object configurations, scene layouts, temporal dynamics, and reasoning complexity, while ensuring balanced coverage across diverse scenarios. The generated data supports a wide range of vision language tasks, including object counting 72 ; 86 ; 10 , VQA 1 ; 31 ; 77 , and object segmentation 98 ; 29 ; 12 , which makes it suitable for training robust and generalizable multimodal models (see figure 1 ). In addition to the data generation pipeline, we introduce a novel fine-tuning strategy based on Visual Question Answering (VQA), rather than conventional caption-based or instruction-following supervision 87 ; 78 ; 105 . Unlike captions, which often emphasize surface-level descriptions, VQA explicitly requires models to attend to specific visual evidence and reason over objects, attributes, and temporal relationships to produce correct answers. By conditioning learning on targeted questions, this paradigm enforces tighter alignment between visual perception and language understanding, encouraging the model to focus on task-relevant regions and frames while suppressing irrelevant context. In addition, VQA naturally supports various forms of reasoning, including counting, comparison, temporal inference, and causal reasoning—capabilities that are critical for complex video understanding tasks. We evaluate our approach on three challenging video-based tasks: video object counting, video-based VQA, and video object segmentation. Extensive experiments demonstrate that models trained on our synthetic data generalize effectively and achieve competitive or improved performance on real-world benchmarks. This strong transfer performance can be attributed to the structured, controllable, and diverse nature of synthetic supervision, which exposes models to a wide range of object interactions, temporal dynamics, and reasoning patterns that are difficult to obtain at scale from real-world videos. Consequently, the model learns general multimodal reasoning principles rather than overfitting to dataset specific biases, while significantly reducing the cost and effort associated with manual data annotation. In summary, we make the following key contributions in this paper: 1. We introduce a synthetic data generation pipeline capable of producing unlimited data with diverse types of annotations including texts, images, videos, audios and so on. This enables training of multimodal large language models (LLMs) with reduced dependency on real-world data. 2. Unlike most existing LLMs, which are fine-tuned using instructions or image captions, we propose a novel fine-tuning approach based on Visual Question Answering (VQA) to enhance the reasoning abilities of multimodal LLMs. 3. We evaluate our approach on three tasks, video object counting, video-based VQA, and video object segmentation; and demonstrate that models trained with our synthetic data show improved performance on real-world benchmarks. 2 Related work 2.1 Synthetic Data and Its Evaluation: Synthetic data has attracted increasing attention due to its scalability and reduced annotation cost. In safety-critical applications, research has focused primarily on synthetic data detection