Multilingual Pretraining for Pixel Language Models
Ilker Kesen † Jonas F. Lotz Ingo Ziegler † Phillip Rust † Desmond Elliott † † Department of Computer Science, University of Copenhagen ‡ ROCKWOOL Foundation Research Unit · 2025
Read Multilingual Pretraining for Pixel Language Models by Ilker Kesen † Jonas F. Lotz Ingo Ziegler † Phillip Rust † Desmond Elliott † † Department of Computer Science, University of Copenhagen ‡ ROCKWOOL Foundation Research Unit (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21265.txt.
Introduction
Five authors pretrain a pixel-based language model on four languages at once and call it PIXEL-M4. Pixel models read text as pictures of rendered words, so they need no fixed vocabulary, a trait that should help across scripts. Until now, training had stayed mostly English. The new model learned from English, Hindi, Ukrainian, and Simplified Chinese, a spread of scripts and language families.
On meaning and grammar tests, PIXEL-M4 beat its English-only sibling on the non-Latin scripts. Word-level probes showed it had picked up rich language features, even for languages outside its training set. Its inner space also lined up the trained languages closely by meaning. The authors say the result proves multilingual pretraining, routine for token models, pays off for pixel models too.
Catalog record
| CREATOR | Ilker Kesen † Jonas F. Lotz Ingo Ziegler † Phillip Rust † Desmond Elliott † † Department of Computer Science, University of Copenhagen ‡ ROCKWOOL Foundation Research Unit |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21265
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multilingual Pretraining for Pixel Language Models Ilker Kesen † Jonas F. Lotz Ingo Ziegler † Phillip Rust † Desmond Elliott † † Department of Computer Science, University of Copenhagen ‡ ROCKWOOL Foundation Research Unit Abstract Pixel language models operate directly on images of rendered text, eliminating the need for a fixed vocabulary. While these models have demonstrated strong capabilities for downstream cross-lingual transfer, multilingual pretraining remains underexplored. We introduce pixel-m 4 , a model pretrained on four visually and linguistically diverse languages: English, Hindi, Ukrainian, and Simplified Chinese. Multilingual evaluations on semantic and syntactic tasks show that pixel-m 4 outperforms an English-only counterpart on non-Latin scripts. Word-level probing analyses confirm that pixel-m 4 captures rich linguistic features, even in languages not seen during pretraining. Furthermore, an analysis of its hidden representations shows that multilingual pretraining yields a semantic embedding space closely aligned across the languages used for pretraining. This work demonstrates that multilingual pretraining substantially enhances the capability of pixel language models to effectively support a diverse set of languages. 1 Introduction Figure 1: Average performance across tasks comparing pixel-m 4 and pixel-bigrams grouped by scripts: Arabic, Brahmic, Chinese-Japanese-Korean, Cyrillic, Latin, and others. Both models share the same architecture and hyperparameters, but pixel-m 4 is pretrained in four visually and linguistically diverse languages: English, Hindi, Ukrainian and Simplified Chinese. pixel-m 4 performs better in almost all non-Latin script languages without sacrificing Latin-script performance. Visually-rendered text has emerged as an alternative to sub-word tokenization for language models ( Salesky et al., 2021 ; Rust et al., 2023 ) . In comparison to sub-word tokenization, processing visually-rendered text enables models to transfer to unseen languages without needing to initialize new embeddings Dobler and de Melo (2023) , or relying on back-off mechanisms based on bytes Xue et al. (2022) or characters Clark et al. (2022) . Previous work on pixel-based language models has predominantly focused on monolingual pretraining on English data Rust et al. (2023) ; Lotz et al. (2023) , with related efforts extending to multilingual pretraining for machine translation Salesky et al. (2023) . Given evidence that pixel-based models facilitate positive transfer through visual similarity ( Lotz et al., 2025 ; Muñoz-Ortiz et al., 2025 ) , we investigate multilingual pretraining for general-purpose representation learning specifically by selecting only one language per script. This approach is particularly valuable for low-resource languages that can benefit from transfer via visually similar, high-resource languages. We present pixel-m 4 : a multilingual version of pixel ( Rust et al., 2023 ) . pixel-m 4 is pretrained on four equally-sized amounts of visually diverse scripts sourced from mC4 Xue et al. (2021) : English (Latin script), Hindi (Devanagari script), Simplified Chinese (Han script), and Ukrainian (Cyrillic script). These scripts were chosen to represent abugida, alphabetic, logographic/logosyllabic writing systems, covering billions of speakers. Furthermore, not only do these scripts represent visual diversity, they also represent grammatical diversity, covering Balto-Slavic, Indo-Iranian, Germanic, and Sino-Tibetan languages. In downstream task experiments, we investigate the ability of pixel-m 4 to transfer to new languages in three conditions (i) same-script; (ii) related-script; and (iii) unrelated scripts to better understand what is gained by multilingual pretraining. 1 1 1 The downstream task languages also cut across different language families, e.g. Indo-European, Sino-Tibetan, and Turkic. However, we focus on script transfer, given the visual nature of the data processed by pixel-m 4 . The same-script experiments focus on Simplified Chinese (Han), Hindi (Devanagari), and various Latin and Cyrillic script languages. The related-script experiments include Japanese and Brahmic script languages; while the unrelated-script experiments focus on Armenian, Greek, Korean and languages using the abjad writing system (e.g. Arabic and Hebrew). Compared to its monolingually-pretrained equivalent, pixel-bigrams ( Lotz et al., 2023 ) , we find consistent improvements in performance for almost all non-Latin script languages on text classification, dependency parsing and named entity recognition. We conduct word-level probing experiments using linspector Şahin et al. (2020) to compare differences in linguistic understanding across 15 languages from multilingual versus monolingual pretraining. We find that pixel-m 4 captures linguistic features more effectively than pixel-bigrams , both for seen scripts (e.g., Russian and Macedonian) and unseen scripts (Arabic, Armenian, Greek). Additionally, an exploration of pixel-m 4 ’s embedding space reveals that earlier layers primarily encode visual information, while deeper layers shift toward semantic understanding, corroborating earlier observations by Tatariya et al. (2024) . Through cross-lingual retrieval experiments, we find that pixel-m 4 has learned a semantic representation space that is shared across the pretraining languages. In short, the main contributions of this paper are: • We present the first multilingually-pretrained general-purpose pixel language model, 2 2 2 Code and models: ilkerkesen/pixel-m4 trained over four visually and linguistically diverse languages. • Experiments on syntactic and semantic tasks show consistent improvements for non-Latin script languages compared to previous pixel language models. • Word-level probing analyses show that multilingual pretraining produces representations that capture more linguistic features across languages, such as case marking, part-of-speech tags, and verb tense. • Sentence-level analyses of the learned hidden representations reveal that pixel-m 4 has learned a representation space highly aligned between a subset of its pretraining languages. 2 pixel-m 4 2.1 Pretraining Data Following our motivation to explore multilingual pretraining through a diverse selection of scripts rather than a large range of languages, pixel-m 4 is pretrained on text written in Latin (English), Cyrillic (Ukrainian), Simplified Chinese characters (Chinese), and Devanagari (Hindi). For each script, a corresponding subset of the mC4 ( Xue et al., 2021 ) corpus is rendered into images, following the strategy of rendering two characters per image patch from Lotz et al. (2023) . With a sequence length of 529 image patches and a batch size of 256, the model observes approximately 135 billion image patches over 1 million pretraining steps – this is the same total amount of data as the original pixel and pixel-bigrams models. However, pixel-m 4 is trained on an order-of-magnitude more unique samples than pixel-bigrams . This difference is due to the fact that pixel-bigrams was trained by iterating 10 times over the English-only Wikipedia + BookCorpus datasets ( Zhu et al., 2015 ) , whereas pixel-m 4 processes each sample in our subset of mC4 only once across the four pretraining languages. processes each sample only once while still adhering to the same number of update steps. 2.2 Pretraining Procedure Both pixel-m 4 and pixel-bigrams follow the pixel pretraining recipe from Rust et al. (2023) , including hyperparameter values. Based on the Masked Autoencoding Vision Transformer ( He et al., 2022 ) , the models render each input sequence to a 529-patch image using the PangoCairo rendering library, 3 3 3 https://docs.gtk.org/PangoCairo where each image patch is 16\times 16 pixels. We use the Google Noto Sans fonts collection to ensure that the majority of Unicode codepoints can be accurately rendered. 4 4 4 https://fonts.google.com/noto pixel-m 4 is trained by mixing the four languages within each batch; however, each individual sample consists of only one language. The image patches are first embedded through a linear projection, 25% of them are masked (in spans of up to 6 consecutive patches), and only the unmasked patches plus a CLS token are passed to the encoder. After the