Language on Demand, Knowledge at Core : Composing LLMs with Encoder-Decoder Translation Models for Extensible Multilinguality
Mengyu Bu, Yang Feng · 2026
Read Language on Demand, Knowledge at Core : Composing LLMs with Encoder-Decoder Translation Models for Extensible Multilinguality by Mengyu Bu, Yang Feng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17512.txt.
Introduction
Bu and Feng propose XBridge, a way to give language models wider language reach. Big models hold rich knowledge but serve low-resource languages poorly. XBridge bolts a pre-trained translation encoder-decoder around an English-centric model core. Light mapping layers and an optimal transport loss align the two representation spaces. Tests on four models covered understanding, reasoning, and summarization. Gains ran largest on low-resource and unseen languages, they report.
Catalog record
| CREATOR | Mengyu Bu, Yang Feng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64093 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17512
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Language on Demand, Knowledge at Core : Composing LLMs with Encoder-Decoder Translation Models for Extensible Multilinguality Mengyu Bu, Yang Feng Abstract Large language models (LLMs) exhibit strong general intelligence, yet their multilingual performance remains highly imbalanced. Although LLMs encode substantial cross-lingual knowledge in a unified semantic space, they often struggle to reliably interface this knowledge with low-resource or unseen languages. Fortunately, pretrained encoder-decoder translation models already possess balanced multilingual capability, suggesting a natural complement to LLMs. In this work, we propose XBridge, a compositional encoder-LLM-decoder architecture that offloads multilingual understanding and generation to external pretrained translation models, while preserving the LLM as an English-centric core for general knowledge processing. To address the resulting representation misalignment across models, we introduce lightweight cross-model mapping layers and an optimal transport-based alignment objective, enabling fine-grained semantic consistency for multilingual generation. Experiments on four LLMs across multilingual understanding, reasoning, summarization, and generation indicate that XBridge outperforms strong baselines, especially on low-resource and previously unseen languages, without retraining the LLM. 1 1 1 https://github.com/ictnlp/XBridge 1 Introduction Figure 1: Overview of XBridge. Pretrained multilingual NMT models provide broad language coverage but limited general reasoning capability, while English-centric LLMs excel at general reasoning yet struggle with low-resource or unseen languages. XBridge harmonizes these strengths through model composition, offloading multilingual processing to the pretrained multilingual model while leveraging the LLM as a knowledge core. Large language models (LLMs) have demonstrated remarkable general intelligence and reasoning abilities 25 ; 26 ; 19 , which are largely grounded in a unified semantic knowledge space. However, despite possessing substantial cross-lingual knowledge, LLMs exhibit imbalanced multilingual performance: while performing reliably in English and a few high-resource languages, they often fail to robustly understand or generate text in low-resource or unseen languages 39 ; 3 . This suggests that the core limitation of LLMs lies not in the absence of knowledge, but in the difficulty of interfacing this knowledge with diverse linguistic representation spaces. Fortunately, a wealth of encoder-decoder based neural machine translation (NMT) models 29 ; 24 specialize in multilingual understanding and generation, and thus provide complementary capabilities to LLMs. These models support semantic transfer across hundreds of languages, including many low-resource ones, by learning a shared semantic representation space across languages. In such models, the encoder maps input text from different languages into the shared semantic space, while the decoder subsequently projects these shared representations into target-language outputs. This closed semantic loop between understanding and generation, along with the modular design of encoder and decoder, naturally complements LLMs. Realizing such a composition would provide LLMs with extensible multilingual capability, particularly for low-resource or unseen languages that are well modeled by NMT systems but remain challenging for LLMs. However, existing approaches only partially address this goal, which integrate multilingual encoders to improve multilingual understanding by injecting encoder representations into LLM inputs 31 ; 11 ; 21 . While effective for input understanding, these approaches leave generation largely English-centric. A natural extension is to further incorporate the multilingual decoder, but doing so introduces a fundamental structural challenge. In NMT, the encoder and decoder are jointly trained within a unified representation space, whereas inserting a frozen LLM in between introduces a transformation from the LLM input space to a different output space shaped by its internal knowledge processing. Consequently, the LLM outputs no longer match the decoder’s expected cross-attention representations, resulting in semantic misalignment that cannot be resolved by simple projection. To address this challenge, we propose XBridge, which composes LLMs with pretrained multilingual NMT models for extensible multilinguality. XBridge adopts an encoder-LLM-decoder architecture, where a multilingual encoder provides robust semantic representations for multilingual inputs, a frozen LLM serves as an English-centric core for knowledge processing, and a multilingual decoder generates outputs in the target language. From a representation perspective, XBridge constructs a semantic bridge that transforms representations from the multilingual semantic space to the LLM input space, through the LLM output space after knowledge transformation, and finally into the decoder’s generation space. By explicitly aligning heterogeneous representation spaces across these modules, XBridge resolves the semantic mismatch introduced by inserting a frozen LLM, achieving extensible and generalizable multilingual understanding and generation. We evaluate XBridge on four LLMs across multilingual understanding, reasoning, summarization, and generation tasks. XBridge outperforms strong baselines, with significant gains on low-resource and unseen languages while preserving LLM’s core capability. With minimal additional parameters, limited training data, and parameter-efficient training, XBridge brings low-resource and unseen language performance close to that of external NMT models, substantially narrowing the gap across languages without retraining the LLM. 2 Related Work 2.1 Data-Level Multilingual Enhancement for LLMs A line of work augments the multilingual capabilities of LLMs at the data level by constructing multilingual training corpora using pretrained multilingual or machine translation models 13 ; 34 ; 35 ; 37 . Typical approaches translate English instruction or task data into multiple languages 4 , or employ translation-based prompting schemes that map non-English inputs into English before task execution 18 ; 2 . These methods are widely adopted for multilingual instruction tuning and cross-lingual transfer. Such approaches generally require continual multilingual training of LLMs, which may introduce translation noise and interfere with existing language capabilities. In practice, balancing performance across high- and low-resource languages remains challenging, as gains on low-resource languages often come at degradation on high-resource ones 7 . In contrast, XBridge achieves multilingual generalization through model composition without multilingual retraining of the LLM. 2.2 Encoder-Augmented Multilingual LLMs Another line of work augments LLMs with pretrained multilingual encoders, injecting encoder representations into the LLM to improve multilingual understanding. 31 leverage multilingual encoders to support cross-lingual understanding, while 11 reintroduce multilingual inputs to better exploit the complementary strengths of language understanding and reasoning in LLMs. 21 further explore layer-wise fusion strategies to enhance the utilization of encoder semantics. These approaches primarily focus on improving multilingual understanding at the input side, while generation remains governed by the LLM’s native language distribution, typically English. Moreover, due to differences in training objectives and tokenization schemes, representation gaps persist between multilingual encoders and LLMs, which limit the effective exploitation of encoder semantics. XBridge differs from prior encoder-augmented methods by additionally incorporating a multilingual decoder to support multilingual generation and by explicitly aligning representations across models, enabling more effective end-to-end multilingual behavior. 3 Method Figure 2: Left: XBridge composes a pretrained multilingual encoder-decoder with an LLM via lightweight mapping layers for multilingual understanding and generation, keeping the LLM frozen as a knowledge core. Right: A three-stage training strategy progressively aligns heterogeneous representations and adapts the encoder and decoder. Figure 2 presents the framework of our XBridge, a compositional multilingual framework that integrates a pretrained encoder-decoder NMT model with an LLM. XBridge efficiently offloads multilingual burden to the external NMT model while preserving the LLM as an English-centric core for general knowledge processing. XBridge adopts an encoder-LLM-decoder architecture, connected by lightweight cross-model mapping layers (Section 3.1 ). To facilitate fine-grained semantic transfer for multilingual generation, we introduce an optimal transport-based token alignment objective at the LLM-decoder interface (Section 3.2 ). For stable optimization, XBridge employs a three-stage training strategy that decouples coarse-grained cross-model alignment from task-specific adaptation (Section 3.3 ). 3.1 Architecture XBridge adopts an encoder-LLM-decoder architecture to compose a pretrained encoder-decoder NMT model