Unified Pix Token And Word Token Generative Language Model
Haun Leung ZiNan Wang · 2026
Read Unified Pix Token And Word Token Generative Language Model by Haun Leung ZiNan Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14028.txt.
Introduction
Leung and Wang join picture tokens and word tokens in one generative language model. Vision Transformers borrowed from CLIP or SigLIP power most open multimodal models, yet they blur fine detail such as small text or digits in images. The new model gives each picture pixel its own token embedding, and adds color folding, a global conditional attention approximation, and unsupervised image pretraining. Pretraining trials test the design.
Catalog record
| CREATOR | Haun Leung ZiNan Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39175 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14028
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unified Pix Token And Word Token Generative Language Model Haun Leung ZiNan Wang Abstract Since the emergence of Vision Transformer (ViT), it has been widely used in generative language model and generative visual model. Especially in the current state-of-art open source multimodal models, ViT obtained by CLIP or SigLIP method serves as the vision encoder backbone to help them acquire visual understanding capabilities. But this method leads to limitations in visual understanding for details, such as difficulty in recognizing small text or numbers in images. To address these issues, we propose a new model to unify pix token and word token into the generative language model. The new model also features with each pix of image having its own token embedding, color folding, global conditional attention approximation and image unsupervised pretraining. We conducted image unsupervised pretraining experiments using our new model to explore its potential. The experimental results show that it has good performance even in small model and with limited training data. We believe our model also conforms to the scaling law, as long as model parameters and training data increased, its performance will continue to improve. Keywords Pix Token Embedding \cdot Color Folding \cdot Global Attention Approximation \cdot Image Unsupervised Pretraining 1 Introduction Since the emergence of Vision Transformer (ViT), it has been widely used in generative language model and generative visual model. Especially in the current state-of-art open source multimodal models, just like Llava, ViT obtained by CLIP or SigLIP method serves as the vision encoder backbone to help them acquire visual understanding capabilities. But this method leads to some issues. Either there are limitations in visual understanding for details, such as difficulty in recognizing small text or numbers in image. Either there are visual understanding drifts in multi-turns of conversations, the first turn of conversation is correct, but errors are prone to occur later on. Although these limitations can be alleviated by adding more training data related to the issues or optimization CLIP method, there will still be other challenges in the future. We believe that these limitations are likely due to the CLIP method, including the following reasons. 1) The encoded features cannot represent all the details of the image ViT encodes images into a small features that is similar to a text features. But what if this text does not fully describe other small details of the image? Or can the text describe all the details of the image? In this case, the encoded features of the image cannot represent all the details of the image. 2) The slightest nudge causes the widest chain reaction If a small change is made for the image, for example the license plate number of car in the image changes from 1 to 6, the all floating-point numbers in the encoded features will change. The slightest nudge causes the widest chain reaction, which is not good. 3) ViT use a fake token embedding ViT forms a fake token embedding by patching the pixels in the image. But this fake token embedding does not have truly adjustable parameters. So the expressive ability of ViT is questionable. We may need to propose a new architecture to address its shortcomings. 4) Limitations of supervised training CLIP uses supervised training, but the supervised training data is always limited. It is easy to cause deficiencies outside of distribution. We creatively proposed a new architecture to unify pix token and word token into the generative language model, no longer relying on ViT encoder. We believe that each pix of image should have its own unique token and true token embedding that we call Pix Token Embedding. We also proposed the concept of Color Folding, which reduces the huge amount of computation and makes Pix Token Embedding feasible. Furthermore, we have proposed Global Conditional Attention Approximation that enables the entire lengthy Pix Token Embedding sequence of image to participate in self-attention operations effectively and feasibly. Now the new architecture make generative language model can predict pix token like predicting word token. We also proposed image unsupervised pretraining, which enable model to learn the patterns of the pixes from huge image datas. And our method is lossless in image data preprocessing, but models like ViT suffer losses in image data preprocessing. We conducted experiments on image unsupervised pretraining, using our unified pix Token and word token model. We found that it has the potential to learn the patterns of the pixes. The experimental results show that it has good performance even in small model and with limited training data. We believe our architecture also conforms to the scaling law, as long as model parameters and training data increased, its performance will continue to improve. However, due to our limited computing resources, we are unable to conduct experiments with large-scale model and data. We cannot predict the upper limit of its potential. God that creates colorful, please please blesses us with some computing resources. We hope our research results 1 1 1 https://github.com/HaunLeung/upw.git can benefit the whole humanity you created. 2 Related work Table 1: Model Configuration Information dim layers heads kv heads image dim image layers fold factor image size window size 768 12 12 6 768 5 16 224 16 2.1 ViT (Vision Transformer) and application ViT (Vision Transformer) and application ( 6 ) is a computer vision model based on Transformer architecture ( 15 ) . It transforms image processing tasks into sequence processing problems, making breakthrough progress in multiple visual tasks. The core idea of ViT is to view an image as a sequence of image patchs. Specifically, it devides the input image into fixed sized non overlapping small patchs, and then linearly projects each small patch into a token embedding. These token embedding sequences, along with an additional learnable classification token embedding, form the input of the Transformer encoder. CLIP (Contrastive Language-Image Pre-training) ( 11 ) is a multimodal visual language pre training model proposed by OpenAI in 2021. Its core goal is to map images and text to a unified semantic space through contrastive learning, achieving cross modal semantic alignment. SigLIP (Sigmoid Loss for Language-Image Pre-training) ( 18 ) is a multimodal pre training model proposed by Google DeepMind in 2023. It is an improved version of CLIP, with the core innovation of using Sigmoid loss function instead of traditional Softmax contrastive loss, significantly improving training efficiency and cross modal alignment performance. CLIP and SigLIP are multimodal contrastive learning models based on the Vision Transformer (ViT) architecture, aimed at mapping images and text to the same semantic space for cross modal understanding. In LLM, ViT obtained through CLIP or SigLIP methods is used as the vision encoder backbone to assist large language models in acquiring visual understanding capabilities. For example, LLAVA ( 7 ) . Currently, excellent open-source multimodal LLMs are further optimized based on the SigLIP method to achieve better image understanding capabilities, such as Kimi-2.5 ( 14 ) and QWen-2.5-VL ( 1 ) . 2.2 Unsupervised pretraining Research work in this area includes GPT-3 ( 2 ) and so on. GPT-3 builds language understanding ability by learning to predict the next word through unsupervised pretraining on massive Internet texts (about 300B tokens). After unsupervised pretraining, the GPT-3 model can