Hear the Scene: Audio-Enhanced Text Spotting

Jing Li, Bo Wang · 2024

Read Hear the Scene: Audio-Enhanced Text Spotting by Jing Li, Bo Wang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19504.txt.

Introduction

Jing Li and Bo Wang spot text in scenes with help from sound. Reading signs and labels in photos is a core vision skill. Training such spotters normally needs costly box labels for every word. Their method trains on transcripts alone, with no boxes at all.

Queries for each word trade signals with image features to learn where the word sits. An attention map then sharpens the spot during reading. A circular teaching plan steadies the weak training signal. A coarse to fine attention step pins the place down. The frame also accepts spoken labels, cutting tagging time and opening the work to annotators with disabilities. Scores rival boxed training.

Catalog record

CREATORJing Li, Bo Wang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-19504

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hear the Scene: Audio-Enhanced Text Spotting Jing Li, Bo Wang Abstract Recent advancements in scene text spotting have focused on end-to-end methodologies that heavily rely on precise location annotations, which are often costly and labor-intensive to procure. In this study, we introduce an innovative approach that leverages only transcription annotations for training text spotting models, substantially reducing the dependency on elaborate annotation processes. Our methodology employs a query-based paradigm that facilitates the learning of implicit location features through the interaction between text queries and image embeddings. These features are later refined during the text recognition phase using an attention activation map. Addressing the challenges associated with training a weakly-supervised model from scratch, we implement a circular curriculum learning strategy to enhance model convergence. Additionally, we introduce a coarse-to-fine cross-attention localization mechanism for more accurate text instance localization. Notably, our framework supports audio-based annotation, which significantly diminishes annotation time and provides an inclusive alternative for individuals with disabilities. Our approach achieves competitive performance against existing benchmarks, demonstrating that high accuracy in text spotting can be attained without extensive location annotations. Figure 1: Different annotation styles for text spotting. The blue point and lines are the position annotation for the text. The transcription annotation “Hills” is displayed in the top left corner of each image. 1 Introduction Scene text spotting Tang et al. 2022a ; Liu et al. 2020a ; Xing et al. 2019 ; Liu et al. 2023 ; Liu et al. 2020b ; Tang et al. 2022c ; Tang et al. 2022b remains a vibrant area of research due to its extensive applications across various domains such as image-based translation, multimedia content analysis, and accessibility features in visual media. The field Liao et al. 2020 ; Zhao et al. 2024b ; feng2023docpedia ; Feng et al. 2023 has evolved from focusing primarily on horizontal or multi-oriented text to addressing more complex, arbitrarily-shaped text. This evolution has prompted shifts from traditional annotation formats like horizontal rectangles and quadrilaterals to more sophisticated polygonal annotations, as depicted in Figure 1. Traditionally, scene text spotting involved separate processes for text detection and recognition, where both tasks required annotations for training. Text recognition used transcription annotations, and text detection depended on geometric annotations like polygons or bounding boxes. The reliance on detailed geometric annotations for training not only escalates the cost but also increases the manpower required, making the process less efficient and scalable. Recent advancements have introduced methods Li et al. 2024b ; Li et al. 2024a ; Zhao et al. 2024c ; Zhao et al. 2024a ; Tang et al. 2024b ; Tang et al. 2023 ; Tang et al. 2024a that directly identify and classify characters or words within texts, which, while reducing some dependencies, still rely heavily on character-level annotations and thus remain costly. Furthermore, the granularity of these annotations makes the task time-consuming and limits the feasibility of large-scale applications on real-world data. For instance, the annotation of text in videos or cluttered urban scenes can be prohibitively expensive and labor-intensive due to the complexity and volume of data. Given these challenges, there is a growing interest in exploring methods that reduce reliance on expensive annotations. Preliminary efforts Wang et al. 2024 ; Wang et al. 2025b ; Sun et al. 2025 , such as the single-point annotation strategy proposed by SPTS Liu et al. 2023 , have marked a significant step forward but still leave room for improvement. This paper introduces a novel approach that rethinks scene text spotting from an image captioning perspective, where the text’s location is inferred implicitly through visual attention mechanisms, without direct location supervision. We propose a fully end-to-end architecture, termed EchoSpot, which leverages a query-based interaction between text queries and image embeddings to approximate text locations while ensuring robust text recognition. Our contributions are geared towards simplifying the annotation process, reducing costs, and making text spotting more accessible and scalable. By demonstrating that accurate text spotting can be achieved without explicit location annotations and by integrating novel strategies like circular curriculum learning and audio annotations, this work paves the way for future research and practical applications in the field of optical character recognition and beyond. Figure 2: Overview of the proposed EchoSpot model architecture. The visual and contextual embeddings are first extracted by a backbone network. The embeddings are then decoded to focus on the relevant text regions using a query-based cross-attention module. Next, the refine-stage text query, the mask generated by the attention activation map, and image embeddings are decoded together to obtain a refined text position. 2 Related Work The field of scene text spotting has witnessed a gradual shift from reliance on heavily annotated data towards methods that require fewer annotations. This evolution reflects the increasing need for cost-effective and scalable solutions. This section reviews existing literature, categorizing the studies into fully-supervised and weakly-supervised methods. 2.1 Fully-Supervised Text Spotting Traditionally, text spotting techniques have drawn heavily from generic object detection frameworks, employing two-stage models that first detect text and then recognize it. Early approaches, like the one proposed by Li et al. Shi et al. 2016 , utilized CNN-based detectors coupled with RNN-based recognition branches, applying RoI pooling to crop text regions for subsequent recognition. These models, however, were limited to handling horizontally oriented text. Extensions of these methods introduced adaptations to manage text with arbitrary orientations and shapes. For example, He et al. Xing et al. 2019 expanded the RoIAlign method to extract features of text in arbitrary orientations, and Sun et al. Liu et al. 2018 introduced a perspective RoI transform layer to align quadrangle proposals into feature maps. More recent innovations, such as the Mask TextSpotter series Liao et al. 2020 , have advanced capabilities to handle arbitrarily-shaped text through character-level semantic segmentation, which, while effective, significantly increases the annotation burden due to the necessity for precise character localization. 2.2 Weakly-Supervised Text Spotting The high cost and labor intensity of annotations required by fully-supervised methods Wang et al. 2025a ; Feng et al. 2024 ; Shan et al. 2024 ; Sun et al. 2024 have spurred interest in weakly-supervised approaches. These methods aim to reduce or entirely eliminate the need for detailed geometric annotations. Notable among the emerging solutions are systems that use minimalistic annotations, such as single-point annotations proposed by SPTS Liu et al. 2023 , which represent text positions with single points, reducing annotation complexity. Other exploratory works found in preprints like arXiv suggest innovative directions such as using image captions to train models that can identify text regions without explicit location data. These methods leverage advancements in deep learning, such as contrastive learning and transformer architectures, to intuit text locations from high-level descriptions or minimal guidance. Despite these advancements, there remains a discernible gap between the fully-supervised and weakly-supervised approaches in terms of accuracy and reliability, particularly in complex real-world scenarios. Our work builds on these foundations, aiming to bridge this gap by proposing a transcription-only supervised method that simplifies the annotation process to just text transcriptions, eliminating the need for any geometric data. This approach not only aligns with the trend toward reducing annotation dependencies but also enhances the scalability and accessibility of text spotting technologies. 3 Methodology The methodology of our EchoSpot

Read the complete text