Rethinking generative image pretraining: How far are we from scaling up next-pixel prediction?
Xinchen Yan, Chen Liang, Lijun Yu, Adams Wei Yu, Yifeng Lu, Quoc V. Le · 2025
Read Rethinking generative image pretraining: How far are we from scaling up next-pixel prediction? by Xinchen Yan, Chen Liang, Lijun Yu, Adams Wei Yu, Yifeng Lu, Quoc V. Le (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08704.txt.
Introduction
Xinchen Yan and five co-authors study how far a strikingly simple idea can scale: training a vision model to predict an image one pixel at a time. They train a family of transformers on small images across many compute budgets and judge them by prediction loss, classification accuracy, and image completion quality.
The best way to grow depends on the goal. Generation wants data to grow three to five times faster than classification does at the same budget. As resolution rises, model size must grow much faster than dataset size. Projecting the trends, the authors argue the true bottleneck is compute rather than data. With compute multiplying each year, they forecast that full pixel-by-pixel image modeling becomes feasible within about five years.
Catalog record
| CREATOR | Xinchen Yan, Chen Liang, Lijun Yu, Adams Wei Yu, Yifeng Lu, Quoc V. Le |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-08704
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rethinking generative image pretraining: How far are we from scaling up next-pixel prediction? Xinchen Yan, Chen Liang, Lijun Yu, Adams Wei Yu, Yifeng Lu, Quoc V. Le Abstract This paper investigates the scaling properties of autoregressive next-pixel prediction, a simple, end-to-end yet under-explored framework for unified vision models. Starting with images at resolutions of 32 × \times 32, we train a family of Transformers using IsoFlops profiles across compute budgets up to 7e19 FLOPs and evaluate three distinct target metrics: next-pixel prediction objective, ImageNet classification accuracy, and generation quality measured by Fréchet Distance. First, optimal scaling strategy is critically task-dependent. At a fixed 32 × \times 32 resolution alone, the optimal scaling properties for image classification and image generation diverge, where generation optimal setup requires the data size grow three to five times faster than for the classification optimal setup. Second, as image resolution increases, the optimal scaling strategy indicates that the model size must grow much faster than data size. Surprisingly, by projecting our findings, we discover that the primary bottleneck is compute rather than the amount of training data. As compute continues to grow four to five times annually, we forecast the feasibility of pixel-by-pixel modeling of images within the next five years. † † affiliationnotice: \icmlCoreContribution 1 Introduction Figure 1 : Key findings on the scaling properties of next-pixel prediction, based on training Transformers on 32 \times 32 images. (a) Learning on raw pixels (blue line) requires 10-20\times higher optimal token-to-parameter ratio than learning on language tokens (yellow line). (b) The optimal scaling strategy varies: generation quality (Fréchet Distance, green) requires more training data optimally than classification (Top-1 accuracy, red) or the next-pixel prediction loss (blue). (c)-(d) The optimal-token and optimal-parameter setup is further verified by training models following the scaling prediction. Given 3.5e20 training FLOPs (with 5 \times more compute), we project to reach 46.39% accuracy (vs. 46.41% in reality) and 244 Fréchet Distance (vs. 240 Fréchet Distance in reality). Inspired by the success in language models where the data is in the form of sequence of words ( 42 ; 50 ) or characters ( 41 ) , a parallel line of research has explored the similar paradigm for vision. Representing each image as a sequence of pixels, prior work ( 48 ; 7 ) demonstrated the possibility of learning visual recognition and generation in a simple and end-to-end fashion, through next-pixel prediction. Conceptually, next-pixel prediction is highly scalable due to its unsupervised nature that requires no human-annotated labels. Indeed, representing each image as pixel sequence enforces the least inductive bias on the structure of images. However, the general paradigm of end-to-end pixel-level modeling ( 48 ; 7 ; 17 ; 36 ) has become less popular over the years. This is largely due to the rise of compute-efficient methods that perform multi-stage pixel generation ( 18 ; 30 ; 22 ) or patch-level learning with vision tokenizers ( 49 ; 27 ; 12 ; 29 ; 6 ; 53 ) . Despite this shift in focus, a simple question remains unanswered: “ How far are we from scaling up next-pixel prediction? ” Admittedly, next-pixel prediction poses many challenges compared to next-token prediction in natural languages. First, pixels have very low semantic content compared to words in a sentence that usually contain rich semantic meaning. Second, pixels have complex spatial dependencies that are non-trivial to represent through sequential modeling. The color of a pixel is not only influenced by its spatial neighboring pixels, but also objects and structures in the scene that are not locally connected. Third, the complexity of next-pixel prediction grows rapidly with the image resolution. To generate a 128 \times 128 image, an autoregressive model must predict 16,384 pixels one after another. In this paper, we present an analysis of the scaling properties of next-pixel prediction across both image recognition and generation. We begin our study at a fixed resolution of 32 \times 32 pixels due to its computational tractability that enables extensive experimentation. At such resolution, images start to show distinct structures and clear object interactions, and hence provides a meaningful approximation of the situation at the native image resolution. We conduct initial scaling experiments based on the next-pixel prediction loss . As shown in Figure 1 (a), results reveal that learning on raw pixels requires substantially ( 10-20\times ) higher optimal token-to-parameter ratio than on text tokens. More concretely, achieving a compute-optimal balance requires at least an order of magnitude more tokens per parameter compared to language models (a ratio of over 400 vs. 20 ). This initial finding motivates a deeper investigation into three core questions. First, how do we reliably assess the performance of these models, especially at the low resolutions where extensive study is more feasible? Second, do the scaling trends derived from the simple next-pixel prediction loss align with the scaling behavior of more meaningful downstream tasks like classification and image completion? Third, how do the scaling trends shift at different resolutions? To answer these questions, we conduct a series of controlled experiments to map the scaling frontiers for three distinct metrics: next-pixel prediction loss, linear probing accuracy, and generation quality (completion-based Fréchet Distance). At a fixed 32 \times 32 resolution, the results reveal (see Figure 1 (b)) that the optimal scaling strategy depends heavily on the target metric, with generation quality requiring a larger token-to-parameter ratio than classification or the next-pixel prediction loss. Furthermore, these scaling dynamics are not static; our study across image resolutions including 16 \times 16 and 64 \times 64 shows that model size must grow much faster than data size, as resolution increases. Finally, we emphasize the forward-looking nature of this work. Based on our scaling trend projection, a 10^{4}\times FLOPs increase (from 10^{20} to 10^{24} ) suggests that Transformers trained via next-pixel prediction could reach 80% ImageNet classification accuracy and compelling generation results. This computational requirement is currently impractical, given that well-established methods such as MAE ( 14 ) (with 4.6\times 10^{20} FLOPs) and DINO ( 5 ) (with 2.1\times 10^{20} FLOPs) achieve comparable performance with significantly less compute. On the other hand, given the sheer volume of visual data available and unsupervised nature of raw pixel learning, we highlight that scaling pixel-by-pixel modeling is far from data bound, but compute bound. We anticipate it will become feasible within the next five years based on the projection that the training compute for frontier AI models grows 4-5 times annually ( 32 ) . 2 Related Work We situate our work within the established literature on pixel prediction models, image tokenization, and scaling laws. Pixel prediction models. Autoregressive next-pixel prediction aims to model the image distribution in a tractable and scalable way. This generative learning objective has been studied on various types of neural network architectures, including RNNs ( 48 ) , CNNs ( 47 ; 31 ) , and Transformers ( 7 ) . Autoregressive Transformers have demonstrated remarkable capabilities at language modeling ( 26 ) through years of improvement, achieving the unification of comprehension and synthesis in a scalable manner. However, autoregressive next-pixel prediction for visual modeling falls far behind and outside mainstream approaches since the early attempt of iGPT ( 7 ) . Modern Transformers are designed to