NCAP: Scene Text Image Super-Resolution with Non-CAtegorical Prior

Dongwoo Park, SUK PIL KO · 2025

Read NCAP: Scene Text Image Super-Resolution with Non-CAtegorical Prior by Dongwoo Park, SUK PIL KO (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00410.txt.

Introduction

Dongwoo Park and Suk Pil Ko sharpen blurry photos of signs and labels by trusting the text recognizer less, not more. Super-resolution tools rebuild fine detail from small images. A hint from a text recognizer helps, but a wrong hint poisons the rebuild.

Their fix swaps the recognizer's final answer for a softer signal from its next-to-last layer. That signal guides without dictating letters. They also blend hard and soft labels during joint training to stop the recognizer growing overconfident on blurry input. Reading signs in the wild, from plates to shopfronts, is the use case. Small hints, softly held, is the design moral.

Catalog record

CREATORDongwoo Park, SUK PIL KO
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-00410

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NCAP: Scene Text Image Super-Resolution with Non-CAtegorical Prior Dongwoo Park, SUK PIL KO Abstract Scene text image super-resolution (STISR) enhances the resolution and quality of low-resolution images. Unlike previous studies that treated scene text images as natural images, recent methods using a text prior (TP), extracted from a pre-trained text recognizer, have shown strong performance. However, two major issues emerge: (1) Explicit categorical priors, like TP, can negatively impact STISR if incorrect. We reveal that these explicit priors are unstable and propose replacing them with Non-CAtegorical Prior (NCAP) using penultimate layer representations. (2) Pre-trained recognizers used to generate TP struggle with low-resolution images. To address this, most studies jointly train the recognizer with the STISR network to bridge the domain gap between low- and high-resolution images, but this can cause an overconfidence phenomenon in the prior modality. We highlight this issue and propose a method to mitigate it by mixing hard and soft labels. Experiments on the TextZoom dataset demonstrate an improvement by 3.5%, while our method significantly enhances generalization performance by 14.8% across four text recognition datasets. Our method generalizes to all TP-guided STISR networks. Figure 1 : Examples illustrating the negative impact of prior knowledge on an STISR task. (a), (b), and (c) of w/ and w/o Ours refer to the results of using CRNN [ 27 ] as a prior generator (pre-trained text recognizer) in TATT [ 22 ] , PARSeq [ 2 ] as a prior generator in TATT [ 22 ] , and LEMMA [ 13 ] , respectively. Even with various STISR networks and a prior generator, the wrong guidance of the explicit prior still appears. Blue indicates the characters that can be influenced by prior knowledge in the STISR results. Red indicates wrong recognition results. Without our method, prior knowledge negatively influences the STISR results; however, with our proposed method, this negative influence can be effectively eliminated. Prior refers to the argmaxed text prior and SR refers to the recognition result of the SR image. 1 Introduction Recognizing and interpreting the text contained in images is a crucial task. Despite numerous advancements in areas such as optical character recognition (OCR), scene text detection (STD), and scene text recognition (STR), the acquisition of low-resolution images is inevitable due to various factors, including the quality of the lens, motion blur, low light, occlusion, noise, etc. Thus, recognizing text in such images remains a challenging task. To reliably recover missing structural details from such images, scene text image super-resolution (STISR) is employed as a pre-processing step to address low-resolution issues. Many studies on scene text images achieve high performance by treating the text prior as an additional modality to leverage text semantics rather than interpreting it as natural scene images. However, due to the explicit nature of the text prior, misclassified categories within the prior knowledge have the potential to harm the STISR task. Thus, some studies address the challenges associated with the text prior. For instance, TPGSR [ 21 ] is a pioneering work that suggests incorporating feedback from a text recognizer, referred to as the text prior, as an additional modality. They also acknowledge accuracy issues with the text prior. To address this, they attempted to complement it by employing the teacher-student structure to distill the text logits from the teacher recognizer. DPMN [ 41 ] introduced a method to complement a pre-trained STISR network using two explicit image-level priors, namely the text mask and graphical recognition results. C3-STISR [ 39 ] improves the prior knowledge using the text prior and its transformed visual cue and linguistic information. LEMMA [ 13 ] enhances recognizer accuracy by minimizing errors in ground-truth text labels through fine-tuning loss. However, the aforementioned methods focus solely on improving the performance of the text prior without addressing the fundamental instability inherent in the text prior. As a pioneering work, we propose Non-CAtegorical Prior (NCAP)—penultimate layer representations processed by additional adapters—instead of relying on the text prior from a pre-trained text recognizer. Figure 1 illustrates the negative impact of incorrect prior knowledge on the STISR result. Our proposed method effectively addresses the aforementioned problem by transforming explicit prior knowledge into an implicit form. Another issue arises during joint training of the STISR network with a recognizer to minimize the domain gap between low-resolution (LR) and high-resolution (HR) images using ground-truth labels, which results in the overconfidence issue. The performance of existing pre-trained text recognizers degrades when applied to LR images. When training the recognizer along with the STISR network to overcome the domain gap between LR and HR images, training the recognizer with soft labels can lead to inaccurate prior knowledge, while training with hard labels leads to an overconfidence issue. Hence, we compare the differences between the hard labels and soft labels used in the existing loss function. We provide both theoretical and empirical analyses of the loss function. In conclusion, we propose a loss function that mitigates the overconfidence issue by employing a linear combination of the hard label and the soft label in a simple yet effective way. The contributions of our proposed method are as follows: (1) We present NCAP, a more information-rich and stable knowledge, achieved by replacing the existing unstable text logits with penultimate layer representations. Our proposed NCAP, including its adapters, requires only 0.3\% additional parameters. Compared with existing methods, NCAP fundamentally resolves explicit information issues. (2) Our work identifies the overconfidence issue that arises from joint training with ground-truth hard labels. We address this issue through a simple but effective approach employing a linear combination of softened Kullback-Leibler (KL) divergence and cross-entropy losses. Figure 2 : Overall architecture. We enhance the previous TP-guided STISR network by introducing a loss function that incorporates linear combinations of hard labels and soft labels, along with NCAP, which utilizes penultimate layer representations as prior knowledge. 2 Related Works 2.1 Scene Text Recognition (STR) Scene text recognition aims to recognize character sequences from scene text images. Convolutional Recurrent Neural Network (CRNN) [ 27 ] predicts character sequences using Connectionist Temporal Classification (CTC) [ 12 ] loss through a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). ASTER [ 28 ] is a pioneering study that introduces Thin-Plate Spline (TPS) transformation for handling spatially transformed text. MORAN [ 20 ] proposed a solution for irregular text by combining multi-object rectification and attention-based sequence prediction. ABINet [ 11 ] corrects predictions through external language modeling. PARSeq [ 2 ] does not use language models separately, and performs permuted sequence modeling while removing the dependence of sequence order through iteratively permuted internal language modeling. Using text as prior knowledge through STR is beneficial for STISR, but it remains a double-edged sword. The problem lies in the negative impact of the explicit categorical information from the recognizer, referred to as the text prior (TP), which can lead to incorrect STISR results when influenced by an inaccurate TP. To address this issue, we propose NCAP, which uses implicit prior knowledge. 2.2 Scene Text Image Super-Resolution (STISR) While scene image super-resolution (SISR) primarily emphasizes enhancing the visual quality of images, STISR has recently shifted its focus to not only improving visual quality but also prioritizing the ultimate goal of enhancing text

Read the complete text