DualTSR: Unified Dual-Diffusion Transformer for Scene Text Image Super-Resolution
Axi Niu, Kang Zhang, Qingsen Yan, Hao Jin, Jinqiu Sun, Yanning Zhang · 2026
Read DualTSR: Unified Dual-Diffusion Transformer for Scene Text Image Super-Resolution by Axi Niu, Kang Zhang, Qingsen Yan, Hao Jin, Jinqiu Sun, Yanning Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14207.txt.
Introduction
Niu and co-authors present DualTSR, a single transformer that restores low-resolution scene text images. Older methods lean on outside OCR models for text hints or stack hard-to-train parts. DualTSR trains one backbone with two diffusion goals at once. One goal models the sharp image itself, and the other models the text content as discrete data. The shared design lets sight and text inform each other at every layer. Tests on synthetic Chinese sets and a real-world suite show strong clarity and text fidelity.
Catalog record
| CREATOR | Axi Niu, Kang Zhang, Qingsen Yan, Hao Jin, Jinqiu Sun, Yanning Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50436 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.14207
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DualTSR: Unified Dual-Diffusion Transformer for Scene Text Image Super-Resolution Axi Niu, Kang Zhang, Qingsen Yan, Hao Jin, Jinqiu Sun, Yanning Zhang Abstract Scene Text Image Super-Resolution (STISR) aims to restore high-resolution details in low-resolution text images, which is crucial for both human readability and machine recognition. Existing methods, however, often depend on external Optical Character Recognition (OCR) models for textual priors or rely on complex multi-component architectures that are difficult to train and reproduce. In this paper, we introduce DualTSR, a unified end-to-end framework that addresses both issues. DualTSR employs a single multimodal transformer backbone trained with a dual diffusion objective. It simultaneously models the continuous distribution of high-resolution images via Conditional Flow Matching and the discrete distribution of textual content via discrete diffusion. This shared design enables visual and textual information to interact at every layer, allowing the model to infer text priors internally instead of relying on an external OCR module. Compared with prior multi-branch diffusion systems, DualTSR offers a simpler end-to-end formulation with fewer hand-crafted components. Experiments on synthetic Chinese benchmarks and a curated real-world evaluation protocol show that DualTSR achieves strong perceptual quality and text fidelity. 1 Introduction Scene Text Image Super-Resolution (STISR) aims to reconstruct high-quality scene text images from low-resolution inputs. The task is especially challenging because character structures are sensitive to degradation: small distortions, missing strokes, or blending with background clutter can alter the underlying semantics. Unlike general image super-resolution, which primarily enhances textures, STISR must simultaneously guarantee textual fidelity and stylistic realism . Any structural error directly causes recognition failure, while inconsistencies in font, color, or orientation reduce visual plausibility. These difficulties are further amplified in languages with large character sets such as Chinese. To reduce the inherent ambiguity of STISR, recent work 2 ; 23 ; 24 ; 36 incorporates textual priors extracted from a pre-trained OCR model. Knowing the expected text sequence significantly constrains the solution space and often improves reconstruction quality. This OCR-guided paradigm has therefore become dominant, and several recent methods 29 ; 44 explicitly condition the SR process on predicted text tokens to enforce semantic correctness. However, its reliability is fundamentally limited by the accuracy of the external OCR: erroneous predictions propagate into the SR network, leading to hallucinated strokes, incorrect glyphs, and overall degraded results. To mitigate this dependency, subsequent works introduced stronger low-level priors. MARCONet 13 learns a discrete codebook of character structures to provide robust structural guidance, while GlyphSR 39 goes further by generating explicit glyph masks using SAM to supervise fine-grained stroke reconstruction. Although these structural priors alleviate the brittleness of OCR guidance, they require increasingly complex and multi-stage pipelines for prior extraction. This growing reliance on external modules highlights a core limitation of existing approaches and motivates a unified framework that can learn both semantic and structural cues internally without handcrafted or externally produced priors. To this end, diffusion-based generative models 33 ; 17 have recently been adopted for STISR, particularly for complex scripts 29 ; 44 . DiffTSR 44 reports strong results by modeling text and image with separate modules connected through a multi-modality fusion block. While effective, this multi-stage architecture is cumbersome: it increases system size, complicates training, and limits the depth of cross-modal interaction because textual and visual information are only exchanged at specific fusion points. To address the reliance on external OCRs and the complexity of multi-module designs, we propose DualTSR , an end-to-end STISR framework built upon a single multimodal transformer. DualTSR unifies super-resolution and text recognition within a dual diffusion objective: a Conditional Flow Matching model generates the high-resolution image, while a discrete diffusion model predicts the character sequence. These two processes operate within a shared architecture, enabling the model to infer its own text priors directly from visual cues and to exploit bidirectional interactions between the evolving text and image representations. This yields a single-model, deeply fused alternative to prior modular pipelines. Our contributions are threefold: • We present DualTSR , a unified STISR framework that jointly learns image super-resolution and text recognition within a single multimodal transformer. • We design a dual diffusion objective, continuous (flow matching) for images and discrete diffusion for text, that removes the need for external OCR priors and avoids complex multi-stage architectures. • Experiments on synthetic Chinese benchmarks and a curated real-world evaluation protocol show that DualTSR delivers strong perceptual quality and text reconstruction accuracy while preserving a unified end-to-end design. Figure 1 : A comparison of text super-resolution architectures. Our proposed method unifies image generation and text recognition within a single model. (a) Baseline: Directly models the conditional distribution of high-resolution images from low-resolution inputs, without a text-based prior. (b) DiffTSR ( 44 ) : Independently models the image and text priors, subsequently fusing their representations using a Multi-modality Module (MoM). (c) Proposed Method: A unified multimodal transformer jointly optimizes for image generation and text recognition through two distinct but interconnected diffusion processes. 2 Related Work 2.1 Text Image Super-Resolution Research in Text Image Super-Resolution (TISR) addresses the unique challenges of simultaneously enhancing image details and improving text legibility. While general-purpose blind super-resolution methods such as BSRGAN 40 and Real-ESRGAN 37 restore natural images well, they often fail to preserve the fine-grained structures and appearance cues required for text. This has motivated specialized STISR methods that introduce text-specific priors or auxiliary supervision. Early and mid-stage STISR work explored complementary cues for restoration. Text Gestalt ( 3 ) emphasizes stroke-aware reconstruction, TATT ( 25 ) introduces text attention to improve robustness to spatial deformation, and C3-STISR ( 45 ) combines multiple clues to recover both semantics and structure. A large line of work then adopts OCR-guided priors, where a pre-trained recognizer predicts text that is fed into the SR network to enforce semantic correctness ( 36 ; 23 ; 29 ; 44 ) . These methods can substantially improve recognition-oriented metrics, but their performance remains tied to the quality of the external OCR prior. More recent methods instead emphasize structural priors. MARCONet ( 13 ) learns a character-structure codebook, MARCONet++ ( 14 ) further strengthens this line with improved character priors, and GlyphSR ( 39 ) uses explicit glyph masks generated from an auxiliary pipeline. These approaches reduce dependence on raw OCR predictions, but they also introduce additional stages, modules, or prior-generation procedures. Our method differs from both families: rather than injecting a separately predicted text prior or a handcrafted structural prior, DualTSR learns image restoration and text prediction jointly within one multimodal backbone. 2.2 Diffusion Models in Super-Resolution Diffusion probabilistic models have demonstrated state-of-the-art performance in image synthesis and restoration, owing to their powerful ability to model complex data distributions 9 ; 17 . This success has naturally led to their application in both natural image super-resolution 28 ; 31 ; 27 and TISR 46 ; 48 ; 18 . Furthermore, research has shown that diffusion models are not only suitable for continuous data like images but can also effectively model discrete data distributions, such as text sequences 1 ; 26 ; 21 ; 30 . In the context of TISR, diffusion models have been used in several ways to mitigate unreliable priors and ambiguous degradations. PEAN ( 46 ) refines text-aware features with diffusion-based priors, TextDiff (