Trillion 7B Technical Report
Trillion Labs [email protected] · 2025
Read Trillion 7B Technical Report by Trillion Labs [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15431.txt.
Introduction
Trillion Labs introduces a seven-billion-parameter language model built to be unusually cheap to train across languages. Their 2025 technical report presents Trillion-7B, a Korean-centric multilingual model. Its signature trick is a cross-lingual document attention mechanism that transfers knowledge from English text into target languages such as Korean and Japanese more efficiently than standard training.
Only about a tenth of the two trillion training tokens were multilingual data, tuned with language-specific filtering and a custom tokenizer. The full run took a modest GPU budget by frontier standards. Across twenty-seven benchmarks in four languages, the model performed competitively with strong cross-lingual consistency. The report's broader claim is economic: careful data design can substitute for brute compute when a model must serve several languages well.
Catalog record
| CREATOR | Trillion Labs [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-15431
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Trillion 7B Technical Report Trillion Labs [email protected] Abstract We introduce Trillion-7B, the most token-efficient Korean-centric multilingual LLM available. Our novel Cross-lingual Document Attention (XLDA) mechanism enables highly efficient and effective knowledge transfer from English to target languages like Korean and Japanese. Combined with optimized data mixtures, language-specific filtering, and tailored tokenizer construction, Trillion-7B achieves competitive performance while dedicating only 10% of its 2T training tokens to multilingual data and requiring just 59.4K H100 GPU hours ($148K) for full training. Comprehensive evaluations across 27 benchmarks in four languages demonstrate Trillion-7B’s robust multilingual performance and exceptional cross-lingual consistency. Preview: Trillion-7B Trillion-LLaVA-7B \reportnumber 001 Figure 1: Trillion-7B significantly advances the Pareto-frontier across all aspects. 1 Introduction Recently, significant advancements have been made in developing massively multilingual large language models (LLMs) ( Touvron et al., 2023 ; Grattafiori et al., 2024 ; Team et al., 2024 ; Qwen et al., 2025 ; DeepSeek-AI et al., 2025 ) expanding their coverage to hundreds of languages. However, substantial performance disparities still persist between high-resource languages like English and less resourced languages like Korean. The fundamental reason for this discrepancy is data scale. Despite being spoken by over 81 million people worldwide, Korean content represents less than 2% of English data volume in common web-crawled datasets like mC4 ( Xue et al., 2021 ) . This severe imbalance means that languages other than English and Chinese simply cannot follow the same scaling trajectory that has proven successful for extremely high-resource languages (Figure 3 ). Addressing this limitation is critical to developing AI systems beneficial to all of humanity, as achieving true artificial general intelligence (AGI) inherently necessitates robust performance across diverse linguistic contexts. In this technical report, we tackle the critical issue of data imbalance in multilingual training. We introduce Trillion-7B, a Korean-targeted multilingual model, which relies on our novel Cross-lingual Document Attention (XLDA) mechanism. XLDA functions as a form of architectural code-switching, efficiently transferring linguistic knowledge from a resource-rich language (like English) to a target less-resourced language (like Korean) without degrading English capabilities. Combined with other innovative training methodologies, we achieve exceptional multilingual performance, especially in Korean, while using significantly less multilingual tokens than prominent Korean-dominant approaches ( Kim et al., 2021 ; Ko et al., 2023 ; Yoo et al., 2024b ; Research et al., 2024a ; Research et al., 2024b ) . Our comprehensive description of Trillion-7B focuses on its multilingual pre-training and post-training, detailing the architecture, methodologies, and transfer strategies that enable robust performance across diverse benchmarks. Our main contributions are as follows: • Cross-lingual Document Attention (XLDA) : A novel mechanism designed explicitly for efficient cross-lingual knowledge integration. • Multilingual Token Efficiency Training : Achieving competitive multilingual performance while dedicating approximately 10% (< 220B tokens, with less than 180B in Korean) of the total 2T training tokens to multilingual data. • Complementary Technical Innovations : Including optimized multilingual data mixtures, language-specific data filtering, customized tokenizer construction, and establishing empirical scaling laws to guide effective model scaling and training efficiency. Training Costs Pre-Training Post-Training Total in H100 GPU Hours 59K 0.36K 59.4K in USD $147K $896 $148K Table 1: Training costs of Trillion 7B on 2T tokens, given H100 price as $2.49 per GPU hour. Figure 2: Cross-Lingual Document Attention. A multilingual batch (left) is packed so that each sequence contains contiguous spans from at least two languages (e.g. English + Korean). The XLDA mask (centre) keeps full self - attention across language blocks (blue cells) while standard causal mask (right) blocks attention across document boundaries (grey cells). 2 Cross-lingual Document Attention (XLDA) During language model pre-training, it is common to pack several short documents into a single long sequence to maximize GPU utilization. Typically, training pipelines introduce segmentation masks at each document boundary to prevent tokens from attending to tokens in previous documents, thus avoiding cross-attention contamination ( Zhao et al., 2024a ) . However, in multilingual pre-training, this intra-document packing may unintentionally block beneficial cross-lingual correspondences that could naturally emerge when documents from different languages share the same context window. To address this limitation, we introduce a novel technique called Cross-Lingual Document Attention (XLDA), specifically designed to manage cross-lingual interactions effectively and improve token-level training efficiency. 2.1 Strategic Batch-level Document Packing The first mechanism of XLDA focuses on how documents are packed into training batches. As illustrated in Figure 2 (left), strategic batch-level document packing ensures that each sequence contains documents from multiple language sources—specifically, contiguous spans from at least two languages (English + non-English). We employ a controlled sampling strategy that combines documents from diverse languages at a predetermined rate, creating opportunities for cross-lingual learning. This intentional interleaving of linguistic content creates a rich training environment where the model can identify cross-lingual patterns and correspondences. Formally, let \mathcal{L}=\{l_{1},l_{2},\ldots,l_{m}\} be the set of m languages in our corpus. For each training sequence S , we construct it as: S=[d_{1}^{l_{i}},d_{2}^{l_{j}},\ldots,d_{k}^{l_{n}}] (1) where d_{t}^{l_{x}} represents the t -th document in language l_{x} . We enforce the constraint that for each sequence S , there exist at least two documents d_{i}^{l_{p}} and d_{j}^{l_{q}} such that l_{p}\neq l_{q} , with a mixing probability \rho that controls the likelihood of cross-lingual document adjacency. The sampling probability for constructing a batch with language l_{i} is given by: P(l_{i})=\alpha\cdot\frac{|D_{l_{i}}|}{\sum_{j=1}^{m}|D_{l_{j}}|}+(1-\alpha)\cdot\beta_{l_{i}} (2) where |D_{l_{i}}| is the corpus size for language l_{i} , \alpha is a temperature parameter that controls sampling smoothness, and \beta_{l_{i}} is a language-specific upsampling factor for low-resource languages. Figure 3: Discrepancy in scaling curves of Llama. The above plots suggest that brute-force scaling (by Llama 2 & 3) results in huge performance gaps between English and Korean, whereas Trillion-7B shows more desirable scaling laws for Korean performance closing the wide gap. 2.2 Selective Attention Masking The second mechanism of XLDA introduces a novel approach to attention masking. As depicted in Figure 2 , the XLDA mask keeps full self-attention across language blocks which allows tokens from different language documents to attend to each other. This contrasts with the standard causal mask that blocks attention across document boundaries. The visualization in Figure 2 clearly illustrates how XLDA fundamentally differs from conventional masking approaches by permitting beneficial cross-document and cross-language attention flows. Note that this attention masking is effective because of the above enforced cross-lingual document packing. 2.3 Design Rationale for XLDA Cross-lingual In-context Pretraining. We draw on insights from recent studies that proper document sequence packing can significantly enhance knowledge integration between the sequences being trained together in-context ( Levine et al., 2022 ) . Through XLDA, our non-English data is consistently meta-learned ( Ortega et al., 2019 ; Lampinen et al., 2024 ) within a linguistic context that includes English, thereby strengthening the model’s ability to generalize learned concepts across languages. Synthetic Code Switching Recent works on code-switching ( Yoo et al., 2024a ; Wang et al., 2025 ) demonstrates substantial improvements in language alignment when models are exposed to mixed-language contexts. XLDA builds on this insight by enabling cross-document attention between languages during pretraining, creating natural opportunities for cross-lingual knowledge transfer without explicit code-switching data. 3 Pre-training 3.1 Pre-training Data The pretraining corpus for Trillion comprises approximately 2T tokens spanning English, multilingual, mathematical, and coding domains. The token distribution follows an 8.5 : 1 : 0.5