SemBridge: Language Transfer in Sparse Encoders via Multilingual Semantic Bridges

Seongtae Hong, Youngjoon Jang, Jia-Huei Ju, Hyeonseok Moon, Heuiseok Lim · 2026

Read SemBridge: Language Transfer in Sparse Encoders via Multilingual Semantic Bridges by Seongtae Hong, Youngjoon Jang, Jia-Huei Ju, Hyeonseok Moon, Heuiseok Lim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26002.txt.

Introduction

Hong and four colleagues adapt sparse search encoders to new languages without rebuilding them. Sparse encoders rank texts by word weight in a word space, but their English-centred build blocks transfer. SemBridge uses dense multilingual vectors as a bridge between word lists. For each new-language word, it picks a few tied source words and sets the new word as a tight mix of core synonyms. Cutting stray meaning this way speeds tuning and lifts search in varied languages.

Catalog record

CREATORSeongtae Hong, Youngjoon Jang, Jia-Huei Ju, Hyeonseok Moon, Heuiseok Lim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46725
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26002

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SemBridge: Language Transfer in Sparse Encoders via Multilingual Semantic Bridges Seongtae Hong, Youngjoon Jang, Jia-Huei Ju, Hyeonseok Moon, Heuiseok Lim Abstract Sparse encoders offer high-precision retrieval by representing term importance within a vocabulary space, yet their English-centric structures pose a critical impediment to language transfer for non-English languages. To overcome this structural limitation, we propose SemBridge, a novel embedding initialization method designed for cross-lingual adaptation in sparse encoders by leveraging multilingual bridge models. SemBridge establishes semantic alignments between source and target vocabularies using multilingual dense embeddings as a bridge. Rather than directly relying on all source tokens, SemBridge selects a small set of semantically related source-language tokens and uses them to initialize each target-language token, effectively filtering out semantic noise and reconstructing target tokens as precise linear combinations of core synonyms. This accelerates convergence during fine-tuning and improves training efficiency. Extensive experiments across five languages and four sparse architectures demonstrate that SemBridge achieves superior zero-shot retrieval performance and consistently improves retrieval performance after fine-tuning compared to existing baselines. These results validate SemBridge as a practical solution for deploying high-performance sparse retrieval systems in diverse linguistic environments. 1 Introduction Information Retrieval has evolved towards deep learning-based dense retrieval to address the lexical mismatch problem 46 ; 45 ; 43 ; 37 ; 21 ; 16 . To overcome dense retrieval’s low interpretability and lack of explicit term-matching capabilities 19 , sparse encoder models have emerged as an alternative 6 ; 50 ; 15 . By representing the contextual importance of terms as sparse vectors within the vocabulary space, they achieve both semantic understanding and keyword precision. Furthermore, their direct compatibility with existing Inverted Index infrastructure significantly enhances the efficiency of large-scale systems 3 ; 33 ; 32 ; 25 . These vocabulary-level representations also provide human-readable term weights, offering interpretable evidence for why a document is retrieved. As the demand for globalized information access grows, efforts to build language-specific retrieval models have gained increasing attention. While most of these efforts have centered on dense retrieval, extending sparse encoders to new linguistic environments is not straightforward. Unlike dense retrievers, whose representations are formed in continuous latent spaces, sparse encoders rely on their vocabulary space as the explicit output space for retrieval. Constrained by this inherent structure, merely fine-tuning an existing English-centric sparse encoder for a target language does not easily yield performance gains. Figure 1: Token distribution across sparse encoders. Other and ETC represent additional language and non-linguistic tokens, respectively. The underlying cause is evident from Figure 1 , which provides an empirical analysis of vocabulary distributions across sparse encoder models. Our analysis demonstrates that for most models, the proportion of non-English tokens is negligible. Notably, the granite-30m-sparse contains only two Korean tokens, and splade-v3 similarly exhibits a significant bias toward English. Given that the vocabulary in a sparse encoder serves as the explicit output space for representing semantics, the absence of target-language tokens creates a structural lack of dimensions through which the model can assign importance to target-language terms. Consequently, it is intrinsically difficult to capture the nuanced semantics of non-English languages within such an English-centric structure, posing a critical bottleneck in reproducing the source model’s retrieval capabilities in a target language. Therefore, even with target-language fine-tuning, the scarcity of target-language tokens remains a key limiting factor in achieving optimal performance. In this paper, to overcome these structural limitations and effectively deploy sparse encoders in target language environments, we propose SemBridge, a novel embedding initialization method that preserves the existing capabilities of a source sparse encoder while transferring its source-language knowledge to a target language. We leverage multilingual dense embeddings as a bridge to perform token-level semantic alignment between the source and target language vocabularies. By reconstructing sophisticated semantic correspondences between tokens with differing surface forms within the parameter space, our method initializes target token embeddings that serve as an optimal starting point. Rather than assigning target tokens randomly or relying only on surface overlap, SemBridge initializes each target token by selecting semantically related source-language tokens and transferring their embedding information through sparse semantic weighting. This ensures that the source model’s inherent retrieval capabilities are fully preserved and remain immediately effective in the target language environment. To demonstrate the generalizability and utility of the proposed method, we conduct extensive experiments across four sparse models and five languages: Arabic, Chinese, Hindi, Korean, and Russian. Experimental results confirm that SemBridge effectively transfers the source model’s retrieval capabilities in a zero-shot setting; furthermore, it achieves superior performance and faster convergence compared to baselines through fine-tuning. Through qualitative analysis, we further reveal that our method precisely aligns target language tokens with core synonyms in the source vocabulary while effectively filtering out unnecessary semantic noise. These results substantiate that SemBridge transcends lexical barriers to fully transplant the source model’s semantic discernment into target language environments. Ultimately, SemBridge serves as a practical and efficient solution for adapting and building high-performance sparse retrieval models in target-language environments, even in non-English settings facing data scarcity. 2 Related Work 2.1 Sparse Encoder Sparse encoders are first-stage retrieval models that represent text as high-dimensional sparse vectors by predicting token importance within the vocabulary space. Various approaches have been proposed to advance this paradigm, including learning the semantic importance distribution for all terms 3 , re-estimating the weights of existing terms 6 , expanding indices by predicting latent terms 38 , and maximizing token-level interactions 17 ; 50 . These approaches have garnered significant attention due to their practicality and interpretability. Because the encoded output aligns with the vocabulary, it can directly utilize existing Inverted Index infrastructure, enabling efficient retrieval without high-cost Approximate Nearest Neighbor (ANN) search indices 27 ; 23 . Furthermore, they explicitly reveal the tokens contributing to the retrieval score 15 and allow flexible control over the balance between memory usage and performance 14 ; 13 . However, many recent sparse encoders are predominantly trained on English 2 ; 7 . In sparse encoders, where the vocabulary space itself serves as the representation space, a small proportion of target language tokens leads to a structural lack of “dimensions” to represent that language, making simple fine-tuning ineffective. While sparse encoders trained specifically for certain languages exist 31 ; 44 , training such models requires a strong MLM model trained from scratch on the corresponding language, large-scale retrieval training data, and significant computational resources. 2.2 Language Transfer Language transfer primarily refers to an approach that adapts models pre-trained in resource-rich languages, such as English, to a target language environment to efficiently achieve performance even with limited data and computational resources. Generally, transfer is attempted through continued pretraining and finetuning 4 ; 11 ; 30 . Another line of work performs vocabulary expansion, which introduces new target language tokens into the existing vocabulary and initializes embeddings only for those added tokens 22 ; 36 . Beyond vocabulary expansion, tokenizer replacement directly addresses vocabulary mismatch by replacing the source tokenizer with one constructed for the target language. In this setting, the central challenge is how to initialize the target tokenizer embeddings while preserving the representation space of the source model. Basic strategies, such as random or source-statistics-based initialization, preserve only generic or distributional properties and fail to align source and target token semantics

Read the complete text