The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer
Weixian Lei, Jiacong Wang, Haochen WangXiangtai LiJun Hao LiewJiashi Feng, Zilong Huang, * Equal contribution, Project LeadBytedance Seed · 2025
Read The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer by Weixian Lei, Jiacong Wang, Haochen WangXiangtai LiJun Hao LiewJiashi Feng, Zilong Huang, * Equal contribution, Project LeadBytedance Seed (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10462.txt.
Introduction
Weixian Lei and six co-authors tested how far a single transformer can go as the whole engine of a vision-language model, in a 2025 paper. Most such models bolt a pre-trained vision tower onto a language model. Their model, SAIL, drops the separate tower and handles raw pixels and text decoding inside one architecture.
SAIL adapts mixed attention and position codes to suit the two input kinds. The team compared its growth with size, its cross-modal flow, and its vision skill against modular rivals. Scaled up, SAIL matched modular models overall. Removing the pre-trained tower changed how information flows between sight and text, and freed the design to scale. On vision tasks such as image segmentation, SAIL scored on par with a giant specialist vision model.
Catalog record
| CREATOR | Weixian Lei, Jiacong Wang, Haochen WangXiangtai LiJun Hao LiewJiashi Feng, Zilong Huang, * Equal contribution, Project LeadBytedance Seed |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10462
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer Weixian Lei, Jiacong Wang, Haochen WangXiangtai LiJun Hao LiewJiashi Feng, Zilong Huang, * Equal contribution, Project LeadBytedance Seed Abstract This paper introduces Sail , a single transformer unified multimodal large language model (MLLM) that integrates raw pixel encoding and language decoding within a singular architecture. Unlike existing modular MLLMs, which rely on a pre-trained vision transformer (ViT), Sail eliminates the need for a separate vision encoder, presenting a more minimalist architecture design. Instead of introducing novel architectural components, Sail adapts mix-attention mechanisms and multimodal positional encodings to better align with the distinct characteristics of visual and textual modalities. We systematically compare Sail ’s properties—including scalability, cross-modal information flow patterns, and visual representation capabilities—with those of modular MLLMs. By scaling both training data and model size, Sail achieves performance comparable to modular MLLMs. Notably, the removal of pretrained ViT components enhances Sail ’s scalability and results in significantly different cross-modal information flow patterns. Moreover, Sail demonstrates strong visual representation capabilities, achieving results on par with ViT-22B in vision tasks such as semantic segmentation. Code and models are available 1 1 1 https://github.com/bytedance/SAIL . 1 Introduction The pursuit of multimodal intelligence has driven the development of Multimodal Large Language Models (MLLMs) [ 61 , 68 , 49 ] , which typically adopt a modular design: a pre-trained vision encoder ( e.g . , CLIP-ViT [ 62 , 15 ] ) extracts image features, a Large Language Model (LLM) [ 59 , 60 , 2 , 58 , 16 , 38 ] processes text, and a lightweight projector aligns the two modalities. This framework achieves strong performance through multi-stage pretraining, supervised fine-tuning (SFT), and post-training on multimodal datasets [ 49 , 91 , 18 , 76 , 3 , 77 ] . While effective, this modular MLLM paradigm inherently fragments multimodal processing, reinforces reliance on pretrained visual encoders, which may limit deployment flexibility and scalability [ 21 , 11 , 54 ] . Figure 1 : (A) Data scaling curve for Modular Multimodal Large Language Model (MLLM) and Sail , our Single Transformer-based MLLM. As pretraining data increases, the single transformer Sail shows a sharper performance gain, demonstrating its superior data scalability. (B) Comparison to existing Single Transformer-based MLLMs : our Sail pushes the performance boundaries on both vision tasks and vision-language tasks. A promising alternative is to eliminate the visual encoder entirely and process raw image patches and text tokens within a single Transformer. This unified architecture removes modality-specific modules, enabling parameter sharing and end-to-end learning of vision-language interactions. Previous works [ 21 , 11 , 54 ] have primarily explored the architecture design, training data, and methods of Single Transformer-based MLLMs. However, little exploration has been given to their fundamental properties, such as scalability , cross-modal information flow patterns , and visual representation capabilities . A deeper understanding of these properties is crucial for unlocking the full potential of Single Transformer-based MLLMs. In this work, we present an experimental analysis of the fundamental properties of Single Transformer-based MLLMs and compare them to modular MLLMs ( e.g . LLaVA [ 49 ] ). Addtionally, in the absence of a pre-trained visual encoder, Single Transformers have to learn visual representations from scratch. Thus, an intriguing question arises: can a trained Single Transformer emerge as a strong vision encoder? We conduct a series of experiments to train and study our S ingle tr A nsformer model for v I sion and L anguage ( Sail ). While we do not propose novel architecture designs, we introduce necessary modifications to enable the model to process different modalities in a unified architecture. In its micro architecture design, we address the different spatial characteristics of 2D images and 1D text data by employing a mixed attention mechanism: bidirectional attention for image patches and causal attention for text tokens, combined with multimodal rotary position embedding. Through model and data scaling, Sail achieves performance on vision-language benchmarks comparable to modular MLLMs, while also functioning as a high-performing vision backbone, as shown in Figure 1 . More concretely, our empirical analysis uncovers following striking advantages of Single Transformer architectures: (i) Superior Data Scaling: In controlled experiments, Sail exhibits steeper performance gains as pretraining data scales. While LLaVA-style modular MLLMs initially perform well, our model’s performance becomes very close to theirs when pretrained on 512M samples, as shown in Figure 1 (A). This suggests that unified architectures can effectively leverage large-scale data and potentially match the performance of modular MLLMs. (ii) Vision Centric Information Flow Pattern: Through analysis of attention distributions, we observe that Single Transformers assign significantly higher attention scores to image tokens during token prediction compared to modular MLLMs. This indicates that the information flow in Single Transformer MLLMs is more direct, with visual tokens influencing prediction tokens more prominently, highlighting a vision-centric approach to decision-making. (iii) Vision Encoder Functioning: Our experiments further demonstrate that the pretrained Single Transformer inherently serves as a powerful vision encoder. Comprehensive evaluations on vision-centric tasks, such as image classification and semantic segmentation, show that the model learns rich visual representations during multimodal pretraining. These representations enhance its capacity for both semantic-level comprehension ( e.g . , object categorization) and pixel-level understanding ( e.g . , fine-grained segmentation masks), bridging high-level abstraction and low-level visual reasoning within a unified architecture. In summary, our findings indicate that Single Transformer-based MLLMs hold great promise in surpassing modular MLLMs in terms of leveraging large-scale data, forming direct vision-centric information pathways, and functioning as effective vision encoders. We hope our empirical findings inspire further research to refine and enhance Single Transformer architecture, ultimately driving advancements in multimodal intelligence from a new perspective. 2 Related Work 2.1 Paradigms in Vision-Language Model Design Modular MLLMs with Visual Encoders. The prevailing approach in MLLM design employs modular architectures [ 61 , 68 , 49 , 3 , 77 ] that rely on pretrained vision encoders ( e.g . , CLIP-ViT [ 62 , 15 ] , InternViT [ 14 ] ) to process visual inputs. The visual features extracted from these frozen encoders are then aligned with LLM input spaces via linear [ 49 , 91 , 7 , 75 ] or cross-attention layers [ 1 , 33 ] . While this module design enables effective transfer of pretrained visual-language knowledge, it also introduces several limitations. First, incorporating a separate ViT encoder significantly slows down both training and inference, increasing deployment complexity and requiring costly infrastructure—especially when compared to a single transformer unified model. Second, common strategies for integrating visual features, such as direct mapping into LLM inputs [ 49 , 91 , 7 ] or sharing them across LLM layers [ 33 , 1 ] , often struggle to reconcile the inherent differences between images and text representations. Finally, as model scale, balancing the interactions between the the encoder, LLM, and alignment layers becomes increasingly challenging [ 11 , 54 ] . Thus, in this work, we explore a single transformer-based MLLM architecture that eliminates the ViT encoder and alignment components to overcome these challenges. Single Transformer-based MLLMs Without Visual Encoders. Emerging research explores end-to-end architectures that process raw image patches and text tokens through a single