Hyper-Local Deformable Transformers for Text Spotting on Historical Maps

Yijun Lin, Yao-Yi Chiang · 2025

Read Hyper-Local Deformable Transformers for Text Spotting on Historical Maps by Yijun Lin, Yao-Yi Chiang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15010.txt.

Introduction

Yijun Lin and Yao-Yi Chiang spot text on old maps in a 2025 paper. Map text holds place lore: past borders, names, and land tales set by the maker hand. But old map text is long, spun at wild angles, and set on busy, stained ground, which breaks stock text spotters.

Their spotter bends its read to each text shape, point by point and char by char. That local bend fits map text where one stiff box would fail. With few past tools and thin training sets for this ground, a flex form is the draw. Archive teams gain a reader for a trove that sits locked in script.

Catalog record

CREATORYijun Lin, Yao-Yi Chiang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15010

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hyper-Local Deformable Transformers for Text Spotting on Historical Maps Yijun Lin, Yao-Yi Chiang Abstract . Text on historical maps contains valuable information providing georeferenced historical, political, and cultural contexts. However, text extraction from historical maps has been challenging due to the lack of (1) effective methods and (2) training data. Previous approaches use ad-hoc steps tailored to only specific map styles. Recent machine learning-based text spotters (e.g., for scene images) have the potential to solve these challenges because of their flexibility in supporting various types of text instances. However, these methods remain challenges in extracting precise image features for predicting every sub-component (boundary points and characters) in a text instance. This is critical because map text can be lengthy and highly rotated with complex backgrounds, posing difficulties in detecting relevant image features from a rough text region. This paper proposes PaLeTTe , an end-to-end text spotter for scanned historical maps of a wide variety. PaLeTTe introduces a novel hyper-local sampling module to explicitly learn localized image features around the target boundary points and characters of a text instance for detection and recognition. PaLeTTe also enables hyper-local positional embeddings to learn spatial interactions between boundary points and characters within and across text instances. In addition, this paper presents a novel approach to automatically generate synthetic map images, SynthMap+ , for training text spotters for historical maps. The experiment shows that PaLeTTe with SynthMap+ outperforms SOTA text spotters on two new benchmark datasets of historical maps, particularly for long and angled text. We have deployed PaLeTTe with SynthMap+ to process over 60,000 maps in the David Rumsey Historical Map collection and generated over 100 million text labels to support map searching. The project is released at https://github.com/kartta-foundation/mapkurator-palette-doc 1. Introduction Detecting and recognizing text from scanned documents is essential for effectively managing them and facilitating analysis and accessibility to their content. This is particularly important for historical and cultural documents, such as historical manuscripts, books, building & infrastructure plans, engineering drawings, handwritten records, and maps. Historical maps, for example, offer valuable information about how territories, landscapes, and civilizations have evolved over time, which are extremely crucial resources required in various scientific domains, including humanity, geography, urban planning, cartography, and environmental science ( Chiang et al. 2014 ; Chiang et al. 2020 ; Chiang et al. 2023 ) . This paper focuses on text spotting on scanned historical maps, an important data source having wide varieties (e.g., in cartographic designs and scanned quality) but lacking (1) effective approaches , (2) text labels for training , and (3) benchmark datasets . Text on historical maps, including names of places, labels for physical features (e.g., building types), and descriptions of areas or landmarks, provides unique insights on georeferenced historical, political, and cultural contexts ( Chiang et al. 2020 ; Chiang et al. 2014 ) . For example, text on historical geological maps indicates locations of rock formations, faults, folds, and mineral deposits, for which digital versions of this information are not available at the scale and coverage of the historical maps. 1 1 1 See the recent AI for Critical Mineral Assessment Competition from DARPA and USGS: https://criticalminerals.darpa.mil/ . Numerous types of scanned historical maps are accessible online, such as the US Geological Survey topographic maps ( Davis 1893 ) , the Sanborn maps ( Lamb 1961 ) , the Ordnance Survey maps ( Hewitt 2011 ) , and a vast range of maps in the David Rumsey Historical Map Collection ( Rumsey 2008 ) . Accessing desired scanned historical maps typically requires searching through map metadata, which are often limited or unavailable. Generating comprehensive metadata for individual map scans requires expert knowledge and extensive manual work. Therefore, accurately and automatically extracting text from scanned historical maps is crucial for enhancing their searchability and facilitating the generation of map metadata ( Chiang et al. 2023 ) . Previous attempts to extract map text involve ad-hoc steps to tailor and finetune existing text detection and recognition approaches for a specific map style ( Arundel et al. 2022 ; Chiang and Knoblock 2011 ; Li et al. 2018 ) or rely on external place name databases ( Li et al. 2020 ; Weinman et al. 2019 ) . Recent text spotting approaches have shown promising results on benchmark datasets (mostly scene images), indicating their potential for historical maps. One category of these methods requires a careful design of the region of interest (RoI) operation to extract image features within the detected regions for recognition ( He et al. 2017 ; Liu et al. 2018 ; Liu et al. 2020 ; Liu et al. 2021 ) . Another category needs additional post-processing to group the results, such as segmentation-based approaches ( Huang et al. 2022 ; Liao et al. 2020 ; Wang et al. 2021 ) . These methods are either poor in recognition due to imprecise detected regions or difficult to generalize for text instances with wide varieties of curvatures, sizes, spacings, orientations, and placement, e.g., historical map text. The third category mitigates these issues by detecting and recognizing text in parallel, for example, utilizing a deformable DETR ( Zhu et al. 2020 ) with a separate character decoder on top of the boundary point decoder ( Zhang et al. 2022 ; Ye et al. 2023 ) . However, there are two remaining challenges in handling lengthy, curved, and highly rotated text on historical maps. First , deformable DETR and its variant spotting models learn image features for detection and recognition around a coarse reference point that roughly indicates the position of a text instance (e.g., the instance center). This works well for object detection, for which the networks predict one object category for a detected object. However, text spotting requires further prediction on multiple sub-components (boundary points and characters) of a detected object (text instance). The reference point(s) and, consequentially, the sampled image features can be misaligned with the target boundary points or characters, especially for text instances that are long, winding, or with extra-large character spacing. For example, Figure 1(a) shows that the deformable DETR-based text spotter, TESTR, samples features from a fixed location (the text center as {\color[rgb]{1,0,0}\bullet} ) that can be distant from the target characters. Second , all boundary points and characters of a text instance share the same position information (e.g., the text center coordinate), which cannot well capture the inter- and intra- spatial relations between these sub-components within and across text instances. (a) TESTR’s sampling locations ( {\color[rgb]{0,0.2305,0.9336}\bullet} ) are around the text center ( {\color[rgb]{1,0,0}\bullet} ) for predicting the first character (left) and the last character (right) (b) PaLeTTe uses character locations ( {\color[rgb]{1,0,0}\bullet} ) to sample hyper-local features ( {\color[rgb]{0,0.2305,0.9336}\bullet} ) for predicting the first character (left) and the last character (right) Figure 1. Comparison between reference points ( {\color[rgb]{1,0,0}\bullet} ) and sampling locations ( {\color[rgb]{0,0.2305,0.9336}\bullet} ) in TESTR and PaLeTTe when querying the first character (left figure) and the last character (right figure) . Darker blue dots have higher attention scores. Furthermore, training these machine learning models requires a large number of text labels, while creating text annotations requires extensive manual work. Although there are existing synthetic datasets ( Wang et al. 2012 ; Jaderberg et al. 2014

Read the complete text