Beyond Language Modeling: An Exploration of Multimodal Pretraining

Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys, Junlin Han, Rob Fergus, Naila Murray, Marjan Ghazvininejad, Mike Lewis, Nicolas Ballas, Amir Bar, Michael Rabbat, Jakob Verbeek, Luke Zettlemoyer, Koustuv Sinha, Yann LeCun, Saining Xie · 2026

Read Beyond Language Modeling: An Exploration of Multimodal Pretraining by Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys, Junlin Han, Rob Fergus, Naila Murray, Marjan Ghazvininejad, Mike Lewis, Nicolas Ballas, Amir Bar, Michael Rabbat, Jakob Verbeek, Luke Zettlemoyer, Koustuv Sinha, Yann LeCun, Saining Xie (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03276.txt.

Introduction

Shengbang Tong, David Fan, John Nguyen, and a large team train multimodal models from scratch to learn what actually makes them work. Native models that mix language and vision are usually adapted from language-first systems, muddying every design lesson. Their 2026 study runs controlled pretraining experiments with no language pretraining in the way.

Four findings stand out. One visual representation excelled at both understanding and generation. Vision and language data proved complementary, each boosting the other. World modeling emerged naturally from unified training rather than special design. Mixture-of-experts scaling worked efficiently and induced modality specialization on its own. Scaling analysis also shows vision is far more data-hungry than language, an asymmetry the expert architecture helps balance.

Catalog record

CREATORShengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys, Junlin Han, Rob Fergus, Naila Murray, Marjan Ghazvininejad, Mike Lewis, Nicolas Ballas, Amir Bar, Michael Rabbat, Jakob Verbeek, Luke Zettlemoyer, Koustuv Sinha, Yann LeCun, Saining Xie
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94343
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.03276

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Language Modeling: An Exploration of Multimodal Pretraining Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys, Junlin Han, Rob Fergus, Naila Murray, Marjan Ghazvininejad, Mike Lewis, Nicolas Ballas, Amir Bar, Michael Rabbat, Jakob Verbeek, Luke Zettlemoyer, Koustuv Sinha, Yann LeCun, Saining Xie Abstract The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models. † † affiliation: FAIR, Meta † † affiliation: New York University † † contribution: equal contribution † † contribution: equal advising † † date: March 3, 2026 \titlespacing * 0pt1.25ex plus 1ex minus .2ex0.75em \project https://beyond-llms.github.io/ Suggestions at a Glance S1. Unify visual representations : RAE bridges visual understanding and generation. § 3 S2. Embrace diverse data : multimodal co-training yields cross-modal synergy. § 4 S3. Unlock world modeling : capabilities naturally emerge from unified multimodal pretraining. § 5 S4. Use MoE : MoE scales multimodal training efficiently and harmonizes capacity across modalities. § 6 , 7 1 Introduction The foundation model era has been defined largely by the success of language pretraining ( OpenAI 2022 ; Google 2023 ) . By scaling autoregressive models on trillions of text tokens, we have created systems with remarkable reasoning capabilities. Yet, fundamentally, text is a human abstraction—a lossy compression of reality. To borrow the allegory of Plato’s Cave ( Plato 375 BCE ) , language models have mastered the description of shadows on the wall without ever seeing the objects casting them. They capture symbols well but miss the high-fidelity physics, geometry, and causality of the physical world. Beyond this philosophical limitation lies a hard, pragmatic ceiling: high-quality text data is finite and approaching exhaustion ( Sutskever 2025 ) . In contrast, the visual world possesses an endless stream of signal “outside the cave”, capturing the raw dynamics of reality that language misses. As a result, the path forward requires moving beyond the shadows to model the source directly. We turn to unified multimodal pretraining, treating the visual signal not as an auxiliary input, but as a first-class citizen alongside language. The scientific landscape of unified multimodal pretraining remains largely opaque. While recent efforts ( Wang et al. 2024 ; Zhou et al. 2025a ; Cui et al. 2025 ) have begun to move beyond language-only pretraining, the design space is rife with confounding variables. Rather than jointly learning from vision and language from scratch, most current methodologies ( Liao et al. 2025 ; Deng et al. 2025 ) rely on initialization from pretrained language models ( Grattafiori et al. 2024 ; Yang et al. 2025a ) . This paradigm prioritizes preserving existing language capabilities while adapting the model to become multimodal. Moreover, the knowledge already embedded in these pretrained backbones confounds any conclusions drawn about the multimodal training itself, making it difficult to disentangle what is learned from unified training versus what is inherited from language pretraining. Consequently, the fundamental dynamics and scaling relationships between vision and language remain poorly understood. In this work, we seek to provide empirical clarity to this landscape. We focus squarely on pretraining, as a model’s core capabilities are largely acquired during this phase ( Zhou et al. 2024 ) . We train a single model from scratch using the Transfusion framework, with next-token prediction for language and diffusion for vision, on diverse data including text, video, image-text pairs, and action-conditioned video. We conduct controlled experiments to isolate key variables and evaluate across a comprehensive spectrum of tasks, ranging from language evaluation and visual understanding/generation, to planning capabilities in world modeling. Specifically, we investigate the following axes: • Visual Representation: We evaluate a broad spectrum of visual representations, ranging from Variational Autoencoders (VAE) and semantic representations to raw pixels. We identify Representation Autoencoders (RAE) as the optimal representation. ( Section 3 ) • Data: We study a wide range of data mixtures, from pure text and video to paired image-text and action-conditioned video. We find minimal interference between modalities and even observe positive synergy in certain cases. ( Section 4 ) • World Modeling: We extend our evaluation to the Navigation World Model (NWM) setting, formatting actions directly as text tokens. We demonstrate that physical prediction capabilities emerge primarily from general multimodal pretraining (e.g. video) rather than domain-specific data. ( Section 5 ) • Architecture: We study design choices for Mixture-of-Experts (MoE) in this unified multimodal setting and observe natural formation of modality separation and unification. ( Section 6 ) • Scaling Properties: We conduct IsoFLOP experiments to derive scaling laws for vision and language during unified pretraining. We uncover a scaling asymmetry where vision is significantly more data-hungry than language, and find that MoE architectures effectively bridge this gap. ( Section 7 ) Figure 1: Overview of our study. Top: High-level model architecture. We train a single autoregressive model with next-token prediction for text and next-visual state prediction with flow matching. Bottom: We study five axes: visual representations, data, world modeling, architecture, and scaling. 2 Experiment Setup Our goal is to study the science and design choices around unified multimodal pretraining. Across experiments, we keep the compute budget and training hyperparameters controlled within each ablation family. Unless otherwise specified, we evaluate at the end of pretraining (without any instruction tuning). We provide an overview of our training and inference protocols here, and more details in Appendix B . 2.1 Training and Inference Language modeling. Given a text sequence (x_{1},\dots,x_{n}) , we minimize standard autoregressive cross-entropy: \mathcal{L}_{\text{LM}}=-\sum_{i=1}^{n}\log p_{\theta}(x_{i}\mid x_{<i}). (1) Image-wise flow matching for visual tokens. Let z_{0} denote the clean latents for an image or a video frame (flattened into a sequence), and let \epsilon\sim\mathcal{N}(0,I) . Following flow matching formulations ( Lipman et al. 2023 ; Liu et al. 2023 ) , we sample t\sim\mathcal{U}[0,1] and construct an interpolated latent z_{t}=(1-t)\,\epsilon+t\,z_{0} . The model predicts a velocity field v_{\theta}(z_{t},t,\text{context}) and we minimize the squared error \mathcal{L}_{\text{flow}}\;=\;\mathbb{E}_{t,z_{0},\epsilon}\left[\,\lVert v_{\theta}(z_{t},t,\cdot)-(z_{0}-\epsilon)\rVert_{2}^{2}\,\right]. (2) To facilitate diffusing through high-dimensional visual representations, we shift the noise schedule towards the noisier end of the spectrum ( Esser et al. 2024 ) , following RAE ( Zheng et al. 2026 ; Tong et

Read the complete text