Do You Need Text Rectification? Soft Attention Mask Embedding for Rectification-Free Scene Text Spotting
Antonio Colombo, Giovanni Bianchi · 2026
Read Do You Need Text Rectification? Soft Attention Mask Embedding for Rectification-Free Scene Text Spotting by Antonio Colombo, Giovanni Bianchi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18173.txt.
Introduction
Antonio Colombo's 2026 paper offers a rectification-free route to scene text spotting with a soft attention mask embedding module. The module uses transformer encoders to weigh high-level features and blend them with predicted masks, yielding sharper text boundaries that suppress background noise. Because the attention step is fully differentiable, recognition loss tunes detection as well, so the two goals train jointly. The resulting SAME-Net scores 84.02 percent end-to-end H-mean on Total-Text, beating the prior best GLASS by 1.02 points with no extra training data.
Catalog record
| CREATOR | Antonio Colombo, Giovanni Bianchi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33693 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18173
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Do You Need Text Rectification? Soft Attention Mask Embedding for Rectification-Free Scene Text Spotting Antonio Colombo, Giovanni Bianchi Abstract End-to-end scene text spotting, which unifies text detection and recognition within a single framework, has witnessed remarkable progress driven by deep learning advances. However, most existing approaches still suffer from incomplete mask proposals caused by multi-scale variation, arbitrary text shapes, and complex background interference, thereby degrading recognition accuracy. In this paper, we propose a novel Soft Attention Mask Embedding module (SAME) that leverages the global receptive field of Transformer encoders to encode high-level features and compute soft attention weights, which are then hierarchically embedded with predicted masks to generate refined text-boundary-aware masks that effectively suppress background noise. Building upon this module, we present SAME-Net, a robust end-to-end text spotting framework that requires neither character-level annotations nor auxiliary text rectification modules. Since the soft attention mechanism is fully differentiable, recognition loss gradients can be back-propagated through the SAME module to the detection branch, enabling joint optimization of detection and recognition objectives. Extensive experiments on challenging benchmarks demonstrate the effectiveness of our approach: SAME-Net achieves 84.02% end-to-end H-mean on the arbitrarily-shaped Total-Text dataset, surpassing the previous state-of-the-art GLASS by 1.02% in full-lexicon accuracy without additional training data, and obtains competitive 83.4% strong-lexicon results on the multi-oriented ICDAR 2015 dataset. 1 Introduction Natural scene text spotting, which involves detecting and recognizing text instances within unconstrained real-world imagery, has become a central research topic in computer vision owing to its broad practical applications in traffic sign recognition, content-based image retrieval, autonomous driving, and document analysis 15 ; 32 ; 55 ; 52 ; 2 . The ability to accurately extract textual information from natural scenes is essential for bridging visual perception and language understanding, and recent years have witnessed substantial advances fueled by deep learning methodologies 1 ; 19 ; 48 ; 38 . In particular, the advent of Transformer architectures 60 and their adaptation to visual tasks 4 have opened new avenues for modeling long-range dependencies in scene text understanding. Existing approaches to scene text spotting can be broadly categorized into two paradigms. The first paradigm adopts a cascaded pipeline that treats text detection and recognition as two independent tasks: a detector first localizes text regions, which are subsequently cropped and fed into a separate recognizer 65 ; 3 ; 28 . While conceptually straightforward, this two-stage design suffers from error accumulation between detection and recognition, and the isolated optimization of each module often fails to yield globally coherent results. The second paradigm embraces end-to-end architectures that integrate detection and recognition within a unified framework, enabling joint optimization and mutual enhancement between the two tasks 26 ; 34 ; 45 ; 12 ; 11 . Recent Transformer-based approaches have further advanced end-to-end text spotting by leveraging attention mechanisms for implicit feature alignment 21 ; 70 ; 68 . Despite significant progress, end-to-end scene text spotting methods continue to face several fundamental challenges. First, the detection branch typically relies on low-level features for precise localization while neglecting high-level global semantic information, which limits the model’s ability to distinguish adjacent or multi-scale text instances 46 ; 50 . The constrained local receptive field of convolutional feature extractors fundamentally restricts their capacity to capture long-range contextual dependencies critical for disambiguating overlapping text regions. Second, the representation of arbitrarily-shaped text using axis-aligned bounding boxes inevitably introduces substantial background noise, and even mask-based region-of-interest (RoI) extraction approaches may fail to capture complete text instances when the predicted masks are incomplete 43 ; 27 ; 40 ; 59 ; 58 . Several recent works have attempted to address the optimization inconsistency between detection and recognition branches. ARTS 74 employs differentiable spatial transformer networks (STN) to back-propagate recognition gradients to the detector, while ABCNet v2 36 leverages adaptive Bezier curves for flexible feature extraction. However, these methods introduce auxiliary text rectification modules that incur significant computational overhead without fundamentally resolving the incomplete mask problem 9 ; 72 ; 73 . More recently, ESTextSpotter 22 introduced explicit synergy in a Transformer for scene text spotting, while DeepSolo 68 demonstrated that a single Transformer decoder with explicit point queries can achieve competitive performance, inspiring further exploration of attention-based architectures for this task. To address these limitations, we propose a novel Soft Attention Mask Embedding (SAME) module that exploits the global receptive field of Transformer encoders to generate refined, text-boundary-aware masks. Specifically, SAME encodes the multi-scale features from the Feature Pyramid Network (FPN) 30 through a Transformer encoder to obtain soft attention weights, which are hierarchically embedded with the predicted segmentation masks. This hierarchical embedding progressively refines the mask boundaries while supplementing text regions that the detection branch failed to segment due to limited local receptive fields. The resulting refined masks more closely adhere to actual text boundaries, effectively suppressing background noise and improving the quality of RoI features fed to the recognizer 56 ; 51 . Building upon the SAME module, we present SAME-Net, a complete end-to-end text spotting framework. Since the soft attention mechanism is fully differentiable, recognition loss gradients can flow back through the SAME module to the Transformer attention encoder, enabling the detection branch to be jointly optimized by both detection and recognition objectives. This joint optimization eliminates the need for character-level annotations and auxiliary text rectification modules, significantly simplifying the training pipeline while improving performance. Our approach draws inspiration from recent advances in vision-language modeling 37 ; 57 ; 41 and demonstrates that leveraging global semantic features through attention-based mask refinement provides a more principled solution to the mask incompleteness problem 62 ; 64 ; 14 . The main contributions of this paper are summarized as follows: • We design a novel Soft Attention Mask Embedding (SAME) module that hierarchically embeds Transformer-encoded soft attention features with predicted masks, progressively refining mask boundaries and supplementing undetected text regions to effectively suppress background noise. • We propose SAME-Net, a robust end-to-end text spotting framework that achieves joint detection-recognition optimization through differentiable soft attention, eliminating the need for character-level annotations and auxiliary text rectification modules. • Extensive experiments on multiple challenging benchmarks demonstrate that SAME-Net achieves state-of-the-art performance, obtaining 84.02% end-to-end H-mean on Total-Text and competitive 83.4% strong-lexicon results on ICDAR 2015, validating the effectiveness of the proposed approach. 2 Related Work 2.1 Cascaded Text Detection and Recognition The traditional approach to scene text spotting decomposes the problem into two sequential stages: text detection followed by text recognition. In the detection stage, text regions are localized using various detection architectures, and the detected regions are cropped, rescaled, and optionally rotated to a canonical horizontal orientation before being fed into a standalone recognizer 65 ; 28 ; 3 . Early approaches to text recognition relied on sequence modeling with recurrent neural networks. Shi et al. 52 proposed an end-to-end trainable convolutional recurrent neural network (CRNN) that combined CNN feature extraction with bidirectional LSTM sequence modeling, establishing a foundational architecture for text recognition that has influenced numerous subsequent works. For text detection, Baek et al. 2 introduced CRAFT, a character-level text detector that leverages character region awareness and affinity scores to handle diverse text appearances. Wang et al. 65 pioneered sliding-window-based character detection followed by per-character classification. Bissacco et al. 3 combined deep neural