Text-Aware Image Restoration with Diffusion Models
Jaewon Min, Jin Hyeon Kim, Paul Hyunbin Cho, Jaeeun Lee, Jihye Park, Minkyu Park, Sangpil Kim, Hyunhee Park, Seungryong Kim · 2025
Read Text-Aware Image Restoration with Diffusion Models by Jaewon Min, Jin Hyeon Kim, Paul Hyunbin Cho, Jaeeun Lee, Jihye Park, Minkyu Park, Sangpil Kim, Hyunhee Park, Seungryong Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-09993.txt.
Introduction
Jaewon Min and eight co-authors tackle a failure mode of AI image repair: invented text, in a 2025 paper. Diffusion-based restorers clean up blurry or damaged photos well, but in regions with words they often paint plausible-looking gibberish instead of the real letters. The authors name this text-image hallucination and define a task, text-aware image restoration, that demands visual and textual fidelity together. They built a benchmark of one hundred thousand scene images richly labeled with text.
Their repair framework, TeReDiff, feeds the diffusion model's inner features into a text-spotting module so the two help each other. Text regions get read while the image is rebuilt, not after. Sharper handling of words matters for documents, signs, and archives, where a wrong letter is a wrong fact.
Catalog record
| CREATOR | Jaewon Min, Jin Hyeon Kim, Paul Hyunbin Cho, Jaeeun Lee, Jihye Park, Minkyu Park, Sangpil Kim, Hyunhee Park, Seungryong Kim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-09993
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Text-Aware Image Restoration with Diffusion Models Jaewon Min, Jin Hyeon Kim, Paul Hyunbin Cho, Jaeeun Lee, Jihye Park, Minkyu Park, Sangpil Kim, Hyunhee Park, Seungryong Kim Abstract Image restoration aims to recover degraded images. However, existing diffusion-based restoration methods, despite great success in natural image restoration, often struggle to faithfully reconstruct textual regions in degraded images. Those methods frequently generate plausible but incorrect text-like patterns, a phenomenon we refer to as text–image hallucination. In this paper, we introduce Text-Aware Image Restoration (TAIR) , a novel restoration task that requires the simultaneous recovery of visual contents and textual fidelity. To tackle this task, we present SA-Text , a large-scale benchmark of 100K high-quality scene images densely annotated with diverse and complex text instances. Furthermore, we propose a multi-task diffusion framework, called TeReDiff , that integrates internal features from diffusion models into a text-spotting module, enabling both components to benefit from joint training. This allows for the extraction of rich text representations, which are utilized as prompts in subsequent denoising steps. Extensive experiments demonstrate that our approach consistently outperforms state-of-the-art restoration methods, achieving significant gains in text recognition accuracy. See our project page: https://cvlab-kaist.github.io/TAIR/ † † footnotetext: ∗ : Equal contribution † † footnotetext: † : Corresponding authors Figure 1: Text-Aware Image Restoration (TAIR). Given a low-quality (LQ) image containing degraded text, our method faithfully restores the original textual content with high legibility and fidelity, whereas previous diffusion-based models [ 5 , 94 , 51 , 9 ] often fail to recover the text regions. 1 Introduction Image restoration is a fundamental task in computer vision, aiming to recover high-quality images from degraded observations. This task is crucial for applications ranging from photography enhancement to medical and autonomous - vision systems. Recent advances in generative modeling—particularly diffusion models [ 25 , 67 ] —have demonstrated remarkable capabilities in image restoration by leveraging powerful generative priors, achieving superior perceptual quality across various degradation scenarios [ 95 , 78 , 51 , 86 , 94 , 9 ] . However, previous models still struggle to recover text regions, as shown in Fig. 1 . Since these models rely on powerful generative priors of diffusion models [ 51 , 86 , 94 , 9 ] , they often synthesize plausible textures instead of reconstructing the precise characters, leading to a text-image hallucination . Yet, textual content provides semantic cues that are essential for scenarios such as document digitization [ 59 , 3 , 42 , 19 , 20 ] , street sign understanding [ 105 , 73 , 17 ] , or AR navigation [ 41 ] , where even slight distortions can compound into significant information loss. Most image restoration studies have focused on the overall perceptual quality and have not explicitly addressed text readability. On the other hand, image super - resolution methods for text restoration have attempted to improve the perceptual quality and legibility of cropped text regions [ 15 , 81 , 88 , 39 ] . However, their patch - level focus introduces fundamental limitations. The global context is discarded by focusing solely on cropped text regions, thereby ignoring information crucial for overall visual coherence. Additionally, most methods train the model from scratch without leveraging large-scale generative priors, restricting their ability to generate high - quality images. To address these limitations, we propose a new task: Text-Aware Image Restoration (TAIR) . In contrast to conventional image restoration approaches [ 82 , 45 , 95 , 78 , 51 , 86 , 94 , 9 ] , TAIR necessitates the integration of textual semantics within the restoration process by operating on full natural images that contain text of varying sizes and spatial contexts—unlike prior methods [ 15 , 81 , 88 , 39 ] that focused solely on cropped text regions. However, a primary challenge lies in the absence of suitable datasets. Existing image restoration benchmarks [ 1 , 6 , 44 , 85 ] are not designed to address text reconstruction, making it difficult to train models that align visual restoration with text readability. While some datasets [ 72 , 76 , 22 , 32 , 96 , 10 ] provide image-text pairs for text-spotting, they are suboptimal for our purpose due to their limitations in scale and quality. These datasets, typically created through synthetic generation [ 22 ] or manual annotations [ 72 , 76 , 96 , 10 ] , often suffer from low resolution. To overcome this, we introduce a large-scale dataset, SA-Text , specifically curated for TAIR. Our dataset curation pipeline begins with automatic text detection to identify candidate images containing text regions. These candidates are then validated using vision-language models [ 4 , 57 ] (VLMs), which align semantic labels with corresponding textual regions. After filtering out low-quality images using VLMs, we obtain high-quality crops of scene images paired with accurate text annotations. Based on SA-1B [ 36 ] , SA-Text comprises 100K images densely annotated with rich textual content. The text in this dataset encompasses a diverse range of font styles, sizes, orientations, and complex visual contexts, offering a robust benchmark for evaluating TAIR. To the best of our knowledge, this is the first benchmark to jointly evaluate perceptual restoration quality and text fidelity. The primary objective of TAIR is to restore full scene images while faithfully preserving the original textual content. To achieve this, we propose a new model for TAIR, named TeReDiff ( Te xt Re storation Diff usion model) that combines a diffusion-based image restoration model with a text-spotting module. Inspired by recent studies demonstrating the effectiveness of diffusion features in vision downstream tasks [ 87 , 33 , 75 ] , we directly use diffusion U - Net features as input to the text - spotting module. This multi-task diffusion framework improves the text spotting performance by leveraging semantically rich and text-aware representations, while also enhancing restoration performance through shared features. Furthermore, at inference time, the output of the text spotting module can be leveraged to generate input prompts for subsequent denoising steps, thereby jointly enhancing visual quality and text readability. In summary, our key contributions are as follows: • We introduce Text - Aware Image Restoration (TAIR), the first restoration task that explicitly requires simultaneous recovery of scene appearance and textual fidelity . • We release SA-Text, a dataset of 100K high-quality images densely annotated with diverse and VLM-verified text, enabling rigorous evaluation and further research on text - conditioned restoration. • We propose TeReDiff, a model trained within a multi - task diffusion framework where U - Net features are forwarded to a text - spotting model during training, while the spotted text is provided as a prompt at inference time, yielding mutual gains in perceptual quality and character legibility. Figure 2: SA-Text curation pipeline. First, a text-spotting model such as DG-Bridge Spotter [ 31 ] is applied to the entire image to detect text regions. Since detection at the full-image scale may fail to capture certain text instances, we further extract image patches corresponding to the detected regions and reapply the same model to each patch in order to detect potential false negatives. Next, two vision-language models (VLMs) [ 4 , 57 ] transcribe the text within