MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts
Chen Hu, Yintao Tai, Antonio Vergari, Frank Keller, Alessandro SugliaSchool of Informatics, University of Edinburgh {yintao.tai, avergari, asuglia}@[email protected]@inf.ed.ac.uk · 2026
Read MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts by Chen Hu, Yintao Tai, Antonio Vergari, Frank Keller, Alessandro SugliaSchool of Informatics, University of Edinburgh {yintao.tai, avergari, asuglia}@[email protected]@inf.ed.ac.uk (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11575.txt.
Introduction
Hu and four colleagues introduce MIXAR, the first generative pixel-based language model. Pretrained on eight major languages across Latin and CJK scripts and scaled to 0.5B parameters, it beats a tokenizer-based model on the same corpus for unseen low-resource languages, with the largest gains on the most distant scripts. MIXAR replaces adversarial training with a GRPO RL stage that rewards its own free-running generations, multiplying free-generation accuracy by up to 2.3x. It matches tokenizer-based models on test benchmarks and resists orthographic attacks better.
Catalog record
| CREATOR | Chen Hu, Yintao Tai, Antonio Vergari, Frank Keller, Alessandro SugliaSchool of Informatics, University of Edinburgh {yintao.tai, avergari, asuglia}@[email protected]@inf.ed.ac.uk |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53285 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.11575
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts Chen Hu, Yintao Tai, Antonio Vergari, Frank Keller, Alessandro SugliaSchool of Informatics, University of Edinburgh {yintao.tai, avergari, asuglia}@[email protected]@inf.ed.ac.uk Abstract Pixel-based language models are gaining momentum as alternatives to traditional token-based approaches, promising to circumvent tokenization challenges. However, the inherent perceptual diversity across languages poses a significant hurdle for multilingual generalization in pixel space. This paper introduces Mixar , the first generative pixel-based language model trained on eight different languages utilizing a range of different scripts. We empirically evaluate Mixar against previous pixel-based models as well as comparable tokenizer-based models, demonstrating substantial performance improvement on discriminative and generative multilingual tasks. Additionally, we show how Mixar is robust to languages never seen during the training. These results are further strengthened when scaling the model to 0.5B parameters which not only improves its capabilities in generative tasks like LAMBADA but also its robustness when challenged with input perturbations such as orthographic attacks. † † footnotetext: \bigstar Equal contribution. \blacklozenge Joint supervision, correspondence to: Antonio Vergari, Frank Keller, Alessandro Suglia. 1 Introduction A fundamental component in the design of mainstream large language models (LLMs) is the tokenizer , which creates the fundamental input units, i.e., tokens that represent the input of the model ( 46 ) . These elementary units are generally sub-words ( 52 ) , characters ( 6 ) , sentence pieces ( 21 ) , or bytes ( 39 ) . However, creating and maintaining the vocabularies that are associated with these tokenizers is a time-consuming task and is typically done only for a handful of dominant languages ( 19 ) . To represent tokens as part of an LLM, an embedding matrix, whose size grows linearly as the number of tokens in the reference vocabulary, is required. For modern sub-word LLMs, representing tokens requires millions of parameters, and this is further exacerbated when dealing with multiple languages . To see why, consider the steady increase of parameters starting from \sim 23M parameters for a BERT-like model trained only on English , raising to \sim 81M for a multilingual BERT model , and \sim 620M parameters for Qwen3-8B ( 53 ) . Additionally, fixed tokenizers disproportionately harm low-resource languages by degrading performance ( 48 ) and increasing costs ( 2 ) —a “dual penalty” that widens the global AI equity gap. Figure 1: Mixar : a Transformer-based decoder-only architecture that uses rendered text as input to learn across multiple languages. Encoding language as pixels, enables Mixar to be robust to visual attacks as well. Furthermore, token-based LLMs exhibit significant structural vulnerabilities, including the presence of under-trained tokens ( 22 ) , the phenomenon of unargmaxable tokens ( 13 ; 15 ) and a general susceptibility to adversarial attacks ( 40 ) . Recently, pixel-based models , i.e., LLMs operating directly (and only) on text rendered as images, such as Pixel ( 34 ) and Pixar ( 45 ) , have been proposed as a compelling alternative to token-based models with the promise of ameliorating the aforementioned issues. In fact, a single pixel-based model should be able to handle multiple languages and scripts without the need to increase its embedding size, as pixels are able to accommodate different writing systems while being more robust to visual changes ( 34 ) . However, both Pixel and Pixar were trained only on English , ignoring the possible benefits pixel-based representations across languages and scripts. This is important if we want to build more inclusive NLP systems that can generalise across languages and cultures ( 24 ) . More recently, 18 introduced Pixel-M4 as a multilingual variant of Pixel , trained on English (Latin script), Hindi (Devanagari script), Simplified Chinese (Han script), and Ukrainian (Cyrillic script). However, Pixel-M4 , like Pixel , is a discriminative BERT-like model and as such is not able to generate new text (in pixel space). In this paper, address this limitation and propose the first generative and multilingual pixel-based LLM. By doing so, we pave the way towards more realistic NLP systems that dispense with tokenizers and instead use other modalities to represent and generate language. Contributions. We advance the literature on pixel-based language models in the following ways: ( 1 ) We present Mixar , a multilingual alternative to Pixar , able to scale to up to 0.5B parameters, higher patch resolution and being pre-trained on eight languages, including German, English, Spanish, Italian, French, Korean, Chinese and Japanese. These languages uses a variety of scripts, including Latin and CJK (East Asian) Scripts as shown in Fig. 1 ; ( 2 ) We conduct evaluations on both discriminative (e.g., natural language inference) as well as generative NLP tasks (e.g., language modelling), demonstrating performance competitive with tokenizer-based models like BERT ( 8 ) and superior to the current state-of-the-art pixel-based models. 2 Beyond English-only and token-based LLMs Subword tokenization (e.g., 38 ; 37 ; 21 ) is the mainstay of most popular LLMs. However, its effectiveness has been limited to English and a few high-resource languages ( 46 ) . Additionally, logographic writing systems (e.g., Chinese) become highly challenging for sub-word tokenizers due to their graphic nature. In early work, 25 used CNNs to capture the visual features of Chinese text at the character level. On this basis, 43 used a dataset of images rendering Chinese characters to train a text classifier. Using standard symbolic tokenizers to extract Chinese graphic information often miss inherent information of the text, as mentioned by 30 , who used Tianzige to extract visual features from Chinese characters. Moreover, 7 and ChineseBERT ( 44 ) integrated character-level visual information into the embedding vectors of BERT-like models. However, they have a fixed vocabulary list and lack graphical information of the words. Therefore, pixel-based models are designed to tackle these problem. Using visual information as input, 35 designed a machine translation model, but its output layer still depends on embeddings over a fixed vocabulary. To overcome this limitation, Pixel ( 34 ) pretrained a masked auto-encoder (MAE) ( 16 ) using a large corpus containing rendered text using a masked reconstruction objective. This model is considered the first pure pixel-based LLM that can handle typical downstream tasks such as extracting question answering and POS tagging. This study was subsequently expanded to handle multilingual tasks by 36 , and further advanced by 18 , who introduced PIXEL-M4 to achieve robust cross-script transfer through pretraining on visually diverse languages. Additionally, 45 created Pixar , the first autoregressive pixel-based model, which showed that it is possible to achieve results that are competitive with traditional tokenizer-based models through a purely autoregressive architecture. Pixel-based models have the advantage of representing language directly from rendered text, overcoming the weaknesses of tokenizer-based language models, such as being prone to vocabulary attacks ( 23 ) . This benefit was leveraged by 28 who expanded recent LLMs with pixel-based information, demonstrating more robust performance even with logographic writing systems. However, this approach requires ad-hoc expansion techniques, which make training more cumbersome. Designing and training a scalable and effective multilingual pixel-based generative model is an open challenge. 3 Mixar We introduce Mixar by building upon Pixar , extending its capabilities through four key contributions: (1) increasing the