TiCLS : Tightly Coupled Language Text Spotter Leeje Jang, Yijun Lin, Yao-Yi Chiang, Jerod Weinman Abstract Scene text spotting aims to detect and recognize text in real-world images, where instances are often short, fragmented, or visually ambiguous. Existing methods primarily rely on visual cues and implicitly capture local character dependencies, but they overlook the benefits of external linguistic knowledge. Prior attempts to integrate language models either adapt language modeling objectives without external knowledge or apply pretrained models that are misaligned with the word-level granularity of scene text. We propose TiCLS , an end-to-end text spotter that explicitly incorporates external linguistic knowledge from a character-level pretrained language model. TiCLS contains a pretrained linguistic decoder that fuses visual and linguistic features, enabling robust recognition of ambiguous or fragmented text. Experiments on ICDAR 2015, Total-Text, and CTW1500 demonstrate that TiCLS achieves state-of-the-art performance, validating the effectiveness of PLM-guided linguistic integration for scene text spotting. The code is available at https://github.com/knowledge-computing/TiCLS . 1 Introduction Copyright ©2026 IEEE. Reprinted from B. Clipp, N. Damer, S. X. Huang and V. Morariu, editors, Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , March 2026. End-to-end scene text spotting detects and recognizes text from scene images in a unified framework. The visual appearance of scene text varies widely, often featuring unusual fonts, perspective distortion, unconventional typography, and even occlusions [ 35 ] . Nevertheless, linguistic patterns remain present in the underlying text. Existing text spotters typically rely on convolution- or transformer-based architectures to extract visual features and use these features to predict text. However, text in scene images often appears degraded (blurred, noisy, or occluded), making it insufficient to rely solely on visual cues. To address this limitation, some prior work incorporates linguistic context into text spotting. For instance, LMTextSpotter [ 39 ] incorporates a parallel recognition branch explicitly trained for autoregressive language modeling. The language modeling contextualizes each character representation using the representations of surrounding characters within the same word during training. However, this contextualization relies solely on the text available in scene text spotting datasets, since the model lacks the ability to pretrain on large-scale text corpora. As a result, the model fails to exploit abundant unlabeled text data and learns contextual patterns only from the limited training text corpora. Pretrained language models (PLMs) [ 5 , 19 , 30 , 13 , 31 ] , trained on large-scale corpora such as news articles and books, provide external linguistic knowledge by capturing semantic dependencies across sub-word tokens in well-structured sentences. TextBlockV2 [ 24 ] incorporates such external linguistic knowledge by detecting coarse, multi-word text blocks and then applying a PLM-based recognizer to transcribe the content within each text block instead of individual words. This approach initializes the recognizer with an existing PLM, which autoregressively decodes text sequences for each block, thereby producing multi-phrase outputs when a block contains multiple lines or phrases. This method highlights the potential of leveraging PLMs in text spotting, especially for images containing large chunks of text, but challenges remain due to the short and fragmented nature of scene text. Scene text differs significantly from well-structured, sentence-level language: instances are short, spatially fragmented, and often lack coherent grammatical structure. Simply initializing a recognition decoder with existing PLMs is therefore insufficient, as this initialization does not address the mismatch between sentence-level linguistic modeling and the word-level semantics required for scene text spotting. The key challenge is how to effectively adapt PLMs’ linguistic knowledge to scene text spotting so that linguistic cues serve as complementary information to support ambiguous or fragmented visual information. To bridge this gap, we introduce TiCLS ( Ti ghtly C oupled L anguage S potter), a novel end-to-end text spotting model that integrates external linguistic knowledge from a pretrained language model (PLM) specifically developed for scene text. TiCLS consists of three main components: a visual feature extractor, a visual decoder, and a linguistic decoder. Following prior transformer-based text spotters [ 9 , 41 , 44 ] , the visual feature extractor and decoder learn visual representations of scene text. To provide strong external linguistic knowledge tailored for scene text spotting, we propose a new transformer-based encoder-decoder PLM that we pretrain on a large text corpus with character-level prediction granularity instead of sub-word tokens. We then initialize TiCLS ’s linguistic decoder with this PLM, allowing the model to fuse our external linguistic priors with visual features in a tightly coupled manner to produce more accurate and robust text predictions. We summarize our contributions as follows: • We propose TiCLS , a novel end-to-end text spotter that explicitly incorporates linguistically rich prior knowledge for scene text spotting. • We develop a character-level pretrained language model tailored for scene text, providing linguistic knowledge for short and fragmented text instances. • We introduce a linguistic decoder initialized with our PLM, enabling tight fusion of linguistic and visual information and improving recognition of ambiguous or visually degraded text. Figure 1 : Overall architecture of TiCLS . TiCLS builds on a DETR-style visual encoder (yellow) and visual decoder (orange). A linguistic decoder (blue) receives visual representations from the visual-to-language projection head and performs visual-linguistic fusion for text recognition (green). The linguistic decoder is initialized with our proposed PLM (Section 3.2 ). 2 Related Work 2.1 End-to-end Scene Text Spotting End-to-end scene text spotting [ 18 , 44 , 20 , 41 , 9 , 15 , 4 , 7 , 39 ] localizes and recognizes text from natural scene images within a single network. These type of approaches enable shared feature representations between text detection and recognition, and jointly optimizes both tasks, leading to improved performance. The increasing success of transformer models has expanded their application to computer vision tasks. DETR-based text spotting methods [ 41 , 7 , 9 , 44 , 39 , 17 ] have achieved state-of-the-art performance. TESTR [ 44 ] consists of a single encoder and dual decoders, one for predicting control points localizing text regions and the other for character recognition. DeepSolo [ 41 ] performs transformer-based text spotting with a single decoder, localizing text regions with Bézier curves. ESTextSpotter [ 9 ] enables explicit communication between the detection and recognition tasks. SwinTextSpotter v2 [ 8 ] proposes Recognition Conversion and Recognition Alignment modules to strengthen the synergy between detection and recognition. While these end-to-end text spotters demonstrate strong performance, the explicit use of prior linguistic knowledge has remained largely unexplored until recently. 2.2 Language-Aware Scene Text Spotting Recent work in scene text spotting has explored incorporating linguistic clues to enhance performance. LMTextSpotter [ 39 ] introduces a separate language modeling recognition branch trained in an autoregressive manner to capture character-level dependencies within a text sequence. This autoregressive design aligns with modeling strategies used in natural language processing (NLP), where decoder-only transformer models such as GPT [ 30 ] generate text by predicting the next token based on previously generated tokens. Because LMTextSpotter trains only on text from scene images, it cannot fully benefit from the linguistic priors offered by broadly pretrained language models. Existing pretrained language models (PLMs) [ 5 , 19 , 30 , 13 , 31 ] are generally trained on large-scale text corpora such as news articles and books and are designed to learn the linguistic context of tokens within well-structured sentences. TextBlockV2 [ 24 ] leverages the pretrained GPT-2 [ 30 ] to initialize the text spotting decoder. This approach extracts visual embeddings for each multi-word text block using an encoder, and then decodes these embeddings to produce block-level text recognition outputs. This design attempts to leverage the ability of PLMs to capture linguistic information across sentence-level contexts. However, because PLMs typically learn to capture complete sentences with coherent semantic meaning, they do not naturally align with scene text spotting, where text often appears as short, spatially localized words or phrases that rarely combine into a complete sentence. As a result, such block-level approaches struggle to achieve consistently strong performance on standard scene text spotting benchmarks. 2.3 Text Recognition with Language Modeling Several works have incorporated language modeling into the scene text recognition (STR) task [ 37 , 1 , 14 ] , which differs from text spotting in that it skips detection and assumes cropped text regions are directly provided as input. Two general approaches exist that differ from TiCLS in mechanism or decoding granularity. First, some approaches adopt language modeling objectives from NLP tasks while maintaining a character-level decoding space, including masked language modeling [ 37 ] . For example, PARSeq [ 1 ] uses permutation language modeling [ 40 ] for causal self-attention and cross-attends to visual encoder outputs with decoder-side linguistic embeddings. This approach helps STR to learn linguistic regularities implicitly. TrOCR [ 14 ] directly integrates a pretrained language model (PLM) into the text recognition task, decoding over sub-word tokens. As described in Section 2.2 , integrating existing PLMs for text recognition tends to perform well on long text sequences, such as lines, phrases, or sentences (e.g., handwritten text recognition), rather than on word-level text from STR. Although recent STR work incorporates linguistic cues with notable success, integrating linguistic information into end-to-end scene text spotting remains underexplored, which forms a core contribution of this work. 3 TiCLS for Text Spotting TiCLS consists of three components (Figure 1 ): a visual feature extractor, a visual decoder, and a linguistic decoder. For the visual components, we adapt DeepSolo [ 41 ] , a DETR-based end-to-end text spotter. We remove DeepSolo’s character classification head and introduce a new visual-to-language projection module to map the visual features into the input space required by the linguistic decoder. The linguistic decoder fuses the transformed visual features with prior knowledge by initializing the decoder with a pretrained language model specifically designed for scene text. Section 3.1 reviews the visual feature extractor and decoder; Section 3.2 introduces the overall architecture and the pretraining of our proposed language model; and Section 3.3 describes the linguistic decoder in TiCLS , which fine-tunes the integrated PLM decoder. 3.1 Visual Feature Extractor and Decoder TiCLS builds on DeepSolo’s [ 41 ] visual feature extractor and decoder. The visual feature extractor leverages a ViTAEv2-based backbone to obtain multi-scale visual features, followed by a visual encoder. The visual encoder produces text region proposals together with their text-existence scores, and TiCLS selects the top K proposals based on these scores for further processing. For each proposal, we sample N points along the Bézier curves derived from the encoder output. We apply a sinusoidal positional encoding function to these points and feed the encoded representations into an MLP to obtain the position queries for the visual decoder. In the visual decoder, we add these position queries to the learnable content queries, and the resulting representations are iteratively refined through intra- and inter-sequence self-attention, which captures relationships among character-level visual features, and through cross-attention with the visual encoder outputs. The decoder produces H^{\mathrm{vis}}\!\in\!\mathbb{R}^{K\times N\times D_{\mathrm{V}}} as the input to three heads: i) a binary text-presence classifier, ii) a center points head regressing centerline Bézier offsets, and iii) a boundary points head regressing top/bottom Bézier offsets. To bridge the visual and linguistic decoders, we introduce a visual-to-language projection head that maps embeddings from the visual decoder’s {D_{\mathrm{V}}} -dimensional space into the linguistic decoder’s {D_{\mathrm{L}}} -dimensional space. The projected embeddings serve as the input to the linguistic decoder (Section 3.3 ), which performs text recognition with tightly coupled visual and linguistic representations. 3.2 Pretrained Language Model for Scene Text Figure 2 shows the architecture of our proposed language model, which consists of a transformer with a bidirectional encoder and a left-to-right decoder, inspired by BART [ 13 ] . The encoder learns bidirectional context from corrupted text and the decoder predicts each token autoregressively based on its preceding tokens and the encoder output. The core intuition is that the decoder, trained to reconstruct words from corrupted text during the PLM pretraining, enables TiCLS to better handle visually corrupted embeddings. Figure 2 : Proposed PLM architecture. The PLM encoder (yellow block) generates contextualized embeddings from the corrupted text, and the PLM decoder (blue block) autoregressively generates text (green) based on the encoder outputs. TiCLS initializes its linguistic decoder with the PLM decoder (blue block). Our approach differs from standard BART pretraining in three respects: i) Token granularity. Unlike standard BART, which uses a sub-word token vocabulary typical of PLMs, we adopt character-level tokens to better match short, fragmented scene text. ii) Input sequences. We pretrain the PLM on text sequences of at most three words instead of full sentences, which mirrors scene text characteristics. iii) Decoder cross attention source. During pretraining, the PLM decoder cross-attends to the PLM encoder states. However during the training and inference for TiCLS , the decoder instead cross-attends to the projected visual features from TiCLS ’s visual decoder. We follow BART’s denoising sequence-to-sequence pretraining paradigm. Among the multiple corruption functions explored in BART (i.e., token masking, deletion, text infilling, sentence permutation, and document rotation), we adopt text infilling, masking 20-40% of contiguous alphabetic spans while preserving non-alphabetic symbols such as digits and separators. Let \mathbf{y}=(y_{1},\ldots,y_{S}) denote a text sequence of length S , and let T be the maximum sequence length such that S\leq T . We generate a corrupted version of the sequence, \tilde{\mathbf{y}}=(\tilde{y}_{1},\ldots,\tilde{y}_{T}) , using a text infilling corruption function with padding tokens from positions S+1 to T . The bidirectional transformer encoder, composed of stacked self-attention blocks, processes \tilde{\mathbf{y}} to produce contextualized hidden representations H^{\text{enc}}\in\mathbb{R}^{T\times D_{\mathrm{L}}} , where D_{\mathrm{L}} denotes language model hidden dimension size. The left-to-right transformer decoder reconstructs the original sequence \mathbf{y} . Each decoder layer applies masked self-attention and cross-attention to the encoder states, followed by a position-wise feed-forward network and a linear prediction head. During training, we use a teacher forcing mechanism [ 38 ] to guide autoregressive output. The decoder receives a right-shifted target sequence \mathbf{y}^{\rightarrow}=(\texttt{},y_{1},\ldots,y_{T-1}) and predicts the next token y_{t} at each step based on the previous tokens \mathbf{y}_{p\end{cases}\quad(p,q=1,\ldots,T). (1) We train the model by minimizing the token-level negative log-likelihood, BART’s training objective, \mathcal{L}_{\text{PLM}}=-\sum_{t=1}^{T}\log p\!\left(y_{t}\mid\mathbf{y}_{