Residual Decoder Adapter: ID-Preserving Tokenizer Adaption for Autoregressive Text Rendering
Dongxing Mao, Alex Jinpeng Wang, Jiahao Tang, Kevin Qinghong Lin, Linjie Li, Zhengyuan Yang, Lijuan Wang, Min Li, Jingru Tan · 2026
Read Residual Decoder Adapter: ID-Preserving Tokenizer Adaption for Autoregressive Text Rendering by Dongxing Mao, Alex Jinpeng Wang, Jiahao Tang, Kevin Qinghong Lin, Linjie Li, Zhengyuan Yang, Lijuan Wang, Min Li, Jingru Tan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01911.txt.
Introduction
Mao and co-authors fix blurry text in images from autoregressive models. Such models predict discrete tokens decoded by a tokenizer. Overall image quality is strong in these models. Text rendering still lags, with blurred strokes. Letter shapes often break or warp as well.
The team traces the flaw to the visual tokenizer. It struggles to rebuild fine-grained detail faithfully. Retraining both parts would fix it but costs a lot. Their Residual Decoder Adapter upgrades the tokenizer after the fact. It leaves the token space unchanged. A paired codebook shares the old token spread. A parallel branch learns the tiny output differences. Text sharpness improves with no full retraining.
Catalog record
| CREATOR | Dongxing Mao, Alex Jinpeng Wang, Jiahao Tang, Kevin Qinghong Lin, Linjie Li, Zhengyuan Yang, Lijuan Wang, Min Li, Jingru Tan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01911
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Residual Decoder Adapter: ID-Preserving Tokenizer Adaption for Autoregressive Text Rendering Dongxing Mao, Alex Jinpeng Wang, Jiahao Tang, Kevin Qinghong Lin, Linjie Li, Zhengyuan Yang, Lijuan Wang, Min Li, Jingru Tan Abstract Visual Autoregressive (AR) models generate images by predicting discrete tokens that are decoded by a visual tokenizer. Despite demonstrating strong overall image generation ability, they still underperform on text rendering with blur strokes and disrupt letter shapes. In this work, we trace this limitation to the visual tokenizer, which struggles to reconstruct fine-grained detail. Improving the tokenizer is straightforward but expensive, as it necessitates retraining both the tokenizer and the AR model. Can we improve text rendering performance of AR models without retraining the existing tokenizer and AR model? To achieve this, we propose the Residual Decoder Adapter(RDA) that upgrades an existing tokenizer post-hoc without changing its token space. Specifically, it refines the decoder output of the visual tokenizer by introducing two novel components: (i) a paired codebook that shares the token distribution with the original one; (ii) a parallel branch to learn the tiny differences (residual) between the reconstructed image and the ground-truth images in the pixel space. This residual design allows us to enhance the tokenizer non-invasively while preserving compatibility with prior AR models. RDA substantially improves text rendering significantly by a large margin. For instance, we boost finetuned Janus-Pro OCR accuracy rises from 24.52% to 58.26% (TextVisionBlend), from 12.75% to 36.81% (StyledTextSynth) on competitive TextAtlas benchmark. The code is available at github.com/CSU-JPG/RDA . † † footnotetext: ⋆ Equal contribution. 2 2 footnotetext: \dagger Corresponding author 1 Introduction Visual Autoregressive (AR) models 43 ; 6 ; 44 have emerged as a powerful paradigm for image generation, predicting discrete visual tokens that are decoded by a visual tokenizer 47 ; 12 . Despite competitive performance on standard benchmarks, AR models face a significant challenge that has recently drawn considerable attention in the community: text rendering. This task has become widely recognized as one of the most demanding tests for fine-grained generation capability 51 ; 9 ; 31 ; 55 , yet AR models substantially lag behind diffusion models 38 ; 20 , producing blurred strokes and distorted shapes ( Fig. 1 (a)). Figure 1 : (a) Comparison between AR model ( Janus-Pro ) and Diffusion model ( FLUX 1.0-dev ) . They exhibit similar performance on the general text-to-image benchmark GenEval, with accuracy of 0.80 and 0.82, respectively. (b) Comparison between VQ-VAE (from Janus-Pro) and VAE (from FLUX) at 512 resolution. Their reconstruction metrics on ImageNet with rFID scores of 9.63 and 7.92, respectively. The weak text rendering of AR models largely stems from the visual tokenizer, whose limited reconstruction capability results in the loss of fine-grained textual details ( Fig. 1 (b)), as also observed in recent analyses 25 ; 49 . As AR models decode solely through discrete visual tokens, details omitted during tokenization are inherently difficult to recover in subsequent generation. Because the tokenizer defines the entire visual language that the AR model can speak, any advance in text rendering must begin with the tokenizer. One strightforward approach is to curate more high-quality data or degsin a sophisticated model to obtain a stronger tokenizer 35 ; 62 ; 29 . However, this process incurs significant training costs, since changing token IDs invalidates prior training and requires retraining the entire AR model( Fig. 2 (b)). Can we improve AR models’ performance in text rendering without modifying the trained tokenizer and AR model? Here comes our solution ( Fig. 2 (c)): Instead of change the tokenizer, we attach an plug-in module that refines the outputs of decoder without altering its ID space. In this work, we propose Residual Decoder Adapter (RDA) , a plug-and-play refinement framework that upgrades any pretrained discrete visual tokenizer post-hoc. RDA keeps the entire visual tokenizer frozen and learns two lightweight components: (1) a learnable Hint Codebook that shares token IDs with the frozen one but provides complementary high-frequency embeddings; (2) a Residual Decoder that reconstructs fine residual in the pixel space based on the original decoder outputs. Together, they act as a “visual fidelity booster”, enhancing text clarity while preserving compatibility with downstream AR models. Figure 2 : The intuition behind our method. (a) The modern text-to-image generation ecosystem consists of an AR model and a visual tokenizer. (b) Existing methods develop an improved tokenizer so inevitably re-train AR models. (c) Our approach enhances the tokenizer with an adopter while preserving the compatibility with original AR model(AR-Training Free). We conducted extensive experiments. Our method consistently improves both general AR models (Janus-Pro 6 , TAR 15 ) and finetuned text-specific AR models (Janus-Pro 6 , Lumina-mGPT 26 ) on text rendering performance across multiple benchmarks. For instance, applying our method to the text-specific Janus Pro 1B model increases its text accuracy from 24.52% to 58.26% on StyledTextVisionBlend 50 . Meanwhile, RDA enhances text reconstruction capability of the tokenizer while preserving overall image fidelity. On AnyText-Benchmark 46 , it improves LlamaGen-VQ 42 text accuracy from 21.26% to 36.79% and reduces LPIPS from 0.1912 to 0.1832. Our design enables scalable visual enhancement without touching the base AR model or retraining the tokenizer. By reinterpreting the tokenizer not as a static bottleneck but as an extendable interface, RDA transforms a long-standing limitation into a pathway for the continual evolution of visual tokenizers. Our contributions are summarized as follows: i. We propose a novel paradigm that bridges the long-standing gap between discrete VQ tokenizers and the continuous image space in difficult text rendering. RDA works as an external tool that can improve the base AR model without ever ”seeing” it. ii. Our method extend to mainstream AR models, including Janus-Pro, TAR, and Lumina-mgpt, which employ discrete VQ-based decoders. We further show the consistent performance gains over popular benchmarks 46 ; 4 ; 14 ; 11 ; 50 . 2 Related Work Visual Autoregressive Models formulate image synthesis as a next-token prediction task. Pioneering works 47 ; 12 discretized pixel space into visual tokens, enabling transformers 48 to treat images as sequences of tokens. 36 ; 10 scaled this approach, showing that large-scale AR transformers could generate high-quality, semantically consistent images from text prompts. Recent advancements 42 ; 36 ; 43 ; 56 ; 6 ; 33 ; 64 ; 53 have scaled AR models to billion-parameter vision-language models, demonstrating improved compositional reasoning and cross-modal understanding 65 ; 70 . Compared to diffusion models 39 ; 38 ; 19 ; 41 ; 59 ; 60 , AR models align naturally with large language models 45 , providing a unified multimodal framework with strong prompt-following and in-context learning 21 ; 58 , reasoning and planning abilities 18 ; 32 ; 9 ; 30 , making them ideal for complicate applications like multimodal dialogue 26 ; 61 . However, their visual fidelity remains limited 23 ; 28 , as performance is bottlenecked by the discrete tokenizer: even with perfect token prediction, reconstruction quality depends on the decoder. This leads to notable degradation in tokenizer-sensitive tasks such as text rendering. Finally, several recent approaches replace the decoder entirely with a diffusion-based decoder or a DiT head 13 ; 54 ; 51 ; 5 ; 34 , achieving stronger decoding capacity. Yet