Zero-Shot Chinese Character Recognition via Global-Local Dual-Branch Alignment and Hierarchical Inference
Wei Cao, Hao Xu, Xiaolei Diao · 2026
Read Zero-Shot Chinese Character Recognition via Global-Local Dual-Branch Alignment and Hierarchical Inference by Wei Cao, Hao Xu, Xiaolei Diao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08814.txt.
Introduction
Wei Cao, Hao Xu, and Xiaolei Diao recognize Chinese characters the system has never seen in training, a constant need given the huge character set. Retrieval methods that match a whole character picture against its structural description in one vector often miss fine component differences, while detailed patch matching is noisy and slow over all candidates. Their network learns global and local views of both picture and description together.
A filtering mask mutes structural symbols with no visual form, and a two stage search recalls candidates globally, then reranks only the top few locally before fusing scores. Results stay competitive across splits, shine with little training data, and cut large scale search cost sharply.
Catalog record
| CREATOR | Wei Cao, Hao Xu, Xiaolei Diao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08814
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Zero-Shot Chinese Character Recognition via Global-Local Dual-Branch Alignment and Hierarchical Inference Wei Cao, Hao Xu, Xiaolei Diao Abstract Chinese character categories are extremely large, and unseen characters frequently arise in open-world scenarios, making zero-shot Chinese character recognition an important yet challenging problem. Existing IDS-based retrieval methods usually encode a character image and its ideographic description sequence into a single global vector for matching. Although efficient, such holistic alignment often under-models local component differences. Moreover, directly introducing patch-token level fine-grained interaction suffers from both the noise of structural operators in IDS and the high cost of full-candidate retrieval.To address these issues, we propose a Global-Local Hierarchical Perception Network (GL-HPN), which jointly learns global and local representations of character images and IDS sequences within a unified cross-modal alignment framework. The global branch supports efficient coarse recall, while the local branch improves component-level discrimination through patch-token interaction. We further introduce a structure filtering mask to suppress structurally meaningful but visually non-entity IDS operators in local similarity aggregation. On top of this, we design a coarse-to-fine hierarchical inference strategy that performs global retrieval over the full candidate set and local reranking only on Top- K K candidates, followed by parameter-free multiplicative fusion of normalized posterior scores. Experimental results show that GL-HPN achieves competitive performance across multiple zero-shot splits, performs especially well under low-resource settings, and substantially reduces the inference cost of large-scale candidate retrieval. 1 Introduction Chinese character recognition is of great importance in applications such as ancient document digitization, variant character normalization, and open-world document understanding. Unlike the closed-set setting, real-world scenarios often contain a large number of rare characters, variant characters, and newly coined characters that never appear during training, making zero-shot Chinese character recognition a challenging research problem. Although the number of Chinese character categories is extremely large, the set of radicals and spatial composition rules on which they are built is relatively limited 5 . Many unseen characters can therefore be regarded as recombinations of known radicals under different structural relations. As a result, if a model can learn transferable component representations and composition patterns, it has the potential to generalize effectively to unseen characters 4 . Existing zero-shot Chinese character recognition methods can be broadly divided into two categories. Predictive methods formulate recognition as the explicit prediction of radicals, structures, or stroke sequences, followed by rule-based parsing or lexicon matching to recover the final character. These methods are structurally interpretable, but usually suffer from error accumulation and long inference chains. In contrast, retrieval-based methods cast recognition as a cross-modal matching problem between character images and structural descriptions. Inspired by CLIP-style contrastive learning frameworks, recent methods such as CCR-CLIP 18 , FT-CLIP 9 align character images with ideographic description sequence (IDS) representations and have achieved substantial progress in zero-shot Chinese character recognition. Despite their effectiveness, most existing retrieval-based methods adopt a holistic alignment paradigm, where character images and IDS sequences are encoded into single global vectors and matched via global similarity. While efficient, this paradigm tends to weaken the modeling of local component differences and fine-grained compositional variations 1 , leading to unstable discrimination in scenarios dominated by local substitutions, subtle structural differences, or visually similar components.As highlighted in recent visual classification studies, obtaining robust and highly discriminative vision feature representations is crucial for overall model performance when dealing with such subtle visual patterns 13 . Furthermore, directly introducing patch-token level fine-grained interaction over the full candidate space leads to two additional challenges. First, structural operators in IDS do not correspond to concrete visual entities and may therefore introduce invalid matching noise during local alignment. Second, the cost of fine-grained interaction grows rapidly with the number of candidate categories, limiting its practicality in large-scale retrieval settings. To address these issues, we propose a Global-Local Hierarchical Perception Network (GL-HPN) for zero-shot Chinese character recognition. Within a unified vision-language alignment framework, GL-HPN jointly models global and local representations of character images and IDS sequences. The global branch learns stable holistic structural alignment for efficient recall over large candidate sets, while the local branch explicitly models the correspondence between image regions and radical tokens through patch-token interaction, thereby enhancing component-level discrimination. Considering that structural operators in IDS lack direct visual correspondence, we further introduce a structure filtering mask to suppress non-entity tokens during local similarity aggregation and reduce invalid responses. During inference, we design a coarse-to-fine hierarchical inference strategy that first performs global ranking over the full candidate set and then conducts local reranking only on Top- K candidates. Finally, global and local evidence are fused through multiplicative posterior fusion, improving recognition performance without significantly increasing inference cost. Our main contributions are summarized as follows: 1. We propose a global-local dual-branch cross-modal alignment framework for zero-shot Chinese character recognition, balancing holistic structural modeling and local component-level discrimination. 2. We introduce a structure filtering mask to explicitly suppress the noise caused by IDS structural operators in local fine-grained matching, improving the stability of local alignment. 3. We design a coarse-to-fine hierarchical inference strategy that restricts expensive local interaction to a small set of high-confidence candidates, achieving a favorable trade-off between recognition accuracy and inference efficiency. 2 Related Work 2.1 Zero-Shot Chinese Character Recognition Zero-shot Chinese character recognition for unseen characters has attracted increasing attention in recent years. Existing methods can be broadly categorized into predictive methods and retrieval-based methods. Predictive methods formulate recognition as the explicit prediction of discrete symbol sequences, such as radicals, structures, or strokes, and then recover the final character through rule-based parsing, lexicon matching, or reasoning modules. Representative works include DenseRAN 15 , STAR 19 , RSST 17 , RZCR 5 . These methods can effectively exploit the compositional regularities of Chinese characters and are often highly interpretable, but they usually suffer from cumulative sequence prediction errors, complex hierarchical decomposition pipelines, and the overhead of post-processing or reasoning modules. In contrast, retrieval-based methods do not explicitly generate discrete sequences. Instead, they formulate zero-shot recognition as similarity matching in a continuous embedding space. Such methods typically treat structural descriptions of characters as textual priors and learn cross-modal consistency between character images and structural descriptions, enabling retrieval and classification over a candidate set. Inspired by CLIP-style frameworks, CCR-CLIP 18 , FT-CLIP 9 , and GRSTR 7 have achieved notable progress in zero-shot Chinese character recognition. CCR-CLIP directly aligns character images with ideographic description sequence (IDS) representations. FT-CLIP introduces formation trees and a dedicated tree encoder to strengthen hierarchical structural modeling. GRSTR further exploits graph structures and direction-aware positional encoding to improve the modeling of spatial relations. Overall, retrieval-based methods enjoy simple inference pipelines, good scalability, and natural suitability for large-scale candidate retrieval. However, most existing methods still adopt a holistic alignment paradigm and therefore struggle to adequately model local component differences and fine-grained compositional changes. 2.2 Contrastive Learning and Fine-Grained Cross-Modal Alignment Contrastive learning aims to learn discriminative representations by pulling positive pairs closer and pushing negative pairs apart in the embedding space, and has become a fundamental paradigm for cross-modal representation learning. CLIP 12 adopts a dual-encoder architecture to project images and text into a shared embedding space and uses a symmetric image-text contrastive objective to achieve global semantic alignment. Owing to its strong scalability and zero-shot transfer ability, CLIP has