Optimizing Language Models for Crosslingual Knowledge Consistency
Tianyu Liu, Jirui Qi, Mrinmaya Sachan, Ryan Cotterell, Raquel Fernández, Arianna Bisazza · 2026
Read Optimizing Language Models for Crosslingual Knowledge Consistency by Tianyu Liu, Jirui Qi, Mrinmaya Sachan, Ryan Cotterell, Raquel Fernández, Arianna Bisazza (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04678.txt.
Introduction
Liu and colleagues improve cross-language match in language models via post-training. Models often answer rendered prompts across languages. The paper defines cross-language match as a gap between a response spread and its round-trip version. PCO couples this gap with a KL penalty to a fixed base model. Since direct optimization needs costly on-policy roll-outs, direct optimization offers a stand-in. Across models, DCO improves cross-language match across 26 languages, beats rivals, enabling aimed tuning for low-resource languages.
Catalog record
| CREATOR | Tianyu Liu, Jirui Qi, Mrinmaya Sachan, Ryan Cotterell, Raquel Fernández, Arianna Bisazza |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96624 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.04678
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimizing Language Models for Crosslingual Knowledge Consistency Tianyu Liu, Jirui Qi, Mrinmaya Sachan, Ryan Cotterell, Raquel Fernández, Arianna Bisazza Abstract Large language models are known to often exhibit inconsistent knowledge. This is particularly problematic in multilingual scenarios, where models are likely to be asked similar questions in different languages, and inconsistent responses can undermine their reliability. In this work, we show that this issue can be mitigated using reinforcement learning with a structured reward function, which leads to an optimal policy with consistent crosslingual responses. We introduce D irect C onsistency O ptimization (DCO), a DPO-inspired method that requires no explicit reward model and is derived directly from the LLM itself. Comprehensive experiments show that DCO significantly improves crosslingual consistency across diverse LLMs and outperforms existing methods when training with samples of multiple languages, while complementing DPO when gold labels are available. Extra experiments demonstrate the effectiveness of DCO in bilingual settings, significant out-of-domain generalizability, and controllable alignment via direction hyperparameters. Taken together, these results establish DCO as a robust and efficient solution for improving knowledge consistency across languages in multilingual LLMs. All code, training scripts, and evaluation benchmarks are released at https://github.com/Betswish/ConsistencyRL . † † affiliationnotice: * Co-first authors Figure 1 : Illustration of DCO, which promotes crosslingual consistency by aligning the completion likelihoods across parallel prompts. Without alignment, the distributions over candidate answers lead to inconsistent preferences between answer pairs. After alignment, the distributions yield two consistent lists that preserve the ranking of answers in both languages. 1 Introduction As multilingual capabilities become a standard feature of modern large language models (LLMs) ( 27 ; 20 ; 5 ) , ensuring crosslingual consistency (CLC) has become increasingly critical. Ideally, an LLM should provide consistent answers question regardless of the language in which it is asked. However, this is far from guaranteed: prior work ( 13 ; 22 ; 30 ) has demonstrated that LLMs often produce conflicting responses across languages, as illustrated in Figure 1 (top). Such inconsistencies can confuse users with diverse language backgrounds and undermine trust in multilingual systems. To address this challenge, we aim to improve CLC by Reinforcement Learning (RL), inspired by principles of alignment with human preferences. Existing post-training algorithms for aligning with human preferences, such as proximal policy optimization ( 25 , PPO;) and direct preference optimization ( 24 , DPO;) , rely on reward functions defined over pairs of responses, often modeled using the Bradley–Terry framework. However, CLC involves connecting multiple languages and requires a different approach to reward design and optimization. To this end, we propose a new reward function that promotes CLC by leveraging the likelihoods assigned by a model to the same answer expressed in different languages. Specifically, to align two languages, {\color[rgb]{0,0,0}L}_{1} and {\color[rgb]{0,0,0}L}_{2} , we define the reward for {\color[rgb]{0,0,0}L}_{1} based on the log-likelihood of answers generated when prompted in {\color[rgb]{0,0,0}L}_{2} , and vice versa. This leads to a policy expressed as a product of experts ( 12 ) : {\color[rgb]{0,0,0}\pi^{\star}}({\color[rgb]{0,0,0}{\color[rgb]{0,0,0}\bm{y}}^{i}}\mid{\color[rgb]{0,0,0}\bm{x}^{i}})=\frac{1}{Z}\prod_{j}\Big({\color[rgb]{0,0,0}\pi_{\textsc{ref}}}\big({\color[rgb]{0,0,0}\tau}^{j}({\color[rgb]{0,0,0}{\color[rgb]{0,0,0}\bm{y}}^{i}})\mid{\color[rgb]{0,0,0}\tau}^{j}({\color[rgb]{0,0,0}\bm{x}^{i}})\big)\Big)^{w_{ij}}, where {\color[rgb]{0,0,0}\pi_{\textsc{ref}}} is the base multilingual LLM, {\color[rgb]{0,0,0}\tau}^{i} translates a prompt or response to {\color[rgb]{0,0,0}L}_{i} , w_{ij} are controllable parameters, and Z is a normalization term. By controlling w_{ij} , the optimal policy {\color[rgb]{0,0,0}\pi^{\star}} can be theoretically guaranteed to be consistent while preserving the model’s overall performance. Building on this foundation, we propose direct consistency optimization (DCO), an efficient algorithm that adapts a DPO-like procedure to our proposed objective without explicit reward models. We theoretically prove that DCO can bypass the online sampling step in RL, yet still arrive at the same optimal policy as the original RL formulation. As illustrated in Figure 1 (middle), training involves parallel prompts and completions across languages, and the optimization encourages consistent distributions over completions in both languages, shown in Figure 1 (bottom). We evaluate the effectiveness of the proposed reward and optimization algorithm with nine LLMs across three datasets, covering 26 languages. Experimental results demonstrate that DCO significantly improves crosslingual consistency while maintaining, and often improving, answer accuracy in the post-trained languages. In summary, our contributions are as follows: • We propose a reward function tailored for CLC and introduce DCO, an algorithm that solves the RL objective, with theoretical guarantees of improved CLC and preserved task performance. • We empirically validate DCO on advanced LLMs across diverse benchmarks. • We provide extensive analyses, including comparisons with other alignment techniques, cross-domain generalization, and control over language preference. 2 Related Work Crosslingual knowledge consistency is a crucial property for multilingual LLMs. Measuring crosslingual consistency. Several studies have explored methods to assess the consistency of knowledge in multilingual LLMs. 35 and 1 evaluate consistency by measuring the agreement of top-1 generated answers to the same question posed in different languages, whereas 13 compute the average overlapping ratio of correct predictions across languages. To assess CLC more comprehensively and to disentangle it from accuracy, 22 introduce the RankC metric, based on a weighted average of the overlapping ratio of top-1 to top- N ranked candidates. These studies reveal significant crosslingual knowledge inconsistencies in a wide range of LLMs. Improving crosslingual consistency. A number of recent studies attempt to improve CLC by applying vector interventions on the hidden representations of LLMs ( 18 ; 29 ; 17 ) . While promising and insightful, these interpretability-based methods are typically tested on small datasets and specific models, making them challenging to scale to broader applications. Closer to our work, 30 proposes CALM, which improves CLC using RL. Their approach first selects a target answer by majority voting based on the model’s completions across multiple languages. Then, they use DPO to increase the likelihood of the target majority answer across languages. However, CALM requires more than two languages, restricting its usage in practical bilingual scenarios. Moreover, it does not necessarily benefit from adding more languages as majority voting can become unreliable when multiple low-resource languages are included. 3 Preliminaries Reinforcement Learning from Human Feedback. Before applying RL, supervised fine-tuning (SFT) is often applied to a pre-trained language model {\color[rgb]{0,0,0}\pi_{{\color[rgb]{0,0,0}\bm{\theta}}}} on a dataset {\color[rgb]{0,0,0}{\color[rgb]{0,0,0}\mathcal{D}}_{\textsc{sft}}} containing annotated examples for downstream tasks to minimize the loss {\color[rgb]{0,0,0}{\color[rgb]{0,0,0}\mathcal{L}}^{\textsc{ml}}}({\color[rgb]{0,0,0}\bm{\theta}})=-\E_{({\color[rgb]{0,0,0}\bm{x}},{\color[rgb]{0,0,0}\bm{y}})\sim{\color[rgb]{0,0,0}{\color[rgb]{0,0,0}\mathcal{D}}_{\textsc{sft}}}}\Bigl[\log{\color[rgb]{0,0,0}\pi_{{\color[rgb]{0,0,0}\bm{\theta}}}}({\color[rgb]{0,0,0}\bm{y}}\mid{\color[rgb]{0,0,0}\bm{x}})\Bigr] . The resulting fine-tuned model is denoted by {\color[rgb]{0,0,0}\pi_{\textsc{sft}}} . However, it may not always reflect human preferences. This misalignment arises because the maximum likelihood estimation objective does not differentiate between major and minor errors in the model’s responses. To address this, a reward optimization step is introduced. Assuming the availability of a reward model {\color[rgb]{0,0,0}r}({\color[rgb]{0,0,0}\bm{x}},{\color[rgb]{0,0,0}\bm{y}}) , which is trained on a dataset with human feedback {\color[rgb]{0,0,0}{\color[rgb]{0,0,0}\mathcal{D}}_{\textsc{hf}}} , the RLHF objective aims to maximize the expected reward of the model’s outputs. A KL divergence regularization term is added to the objective ( 26 ) to prevent reward hacking ( 2 ) and ensure the model does not deviate excessively from {\color[rgb]{0,0,0}\pi_{\textsc{sft}}} . The ultimate target is to obtain a {\color[rgb]{0,0,0}\pi_{{\color[rgb]{0,0,0}\bm{\theta}}}} : \displaystyle\max_{{\color[rgb]{0,0,0}\pi_{{\color[rgb]{0,0,0}\bm{\theta}}}}} \displaystyle\E_{{\color[rgb]{0,0,0}\bm{x}}\sim{\color[rgb]{0,0,0}\mathcal{D}},{\color[rgb]{0,0,0}\bm{y}}\sim{\color[rgb]{0,0,0}\pi_{{\color[rgb]{0,0,0}\bm{\theta}}}}(\cdot|{\color[rgb]{0,0,0}\bm{x}})}\Big[{\color[rgb]{0,0,0}r}({\color[rgb]{0,0,0}\bm{x}},{\color[rgb]{0,0,0}\bm{y}})\Big] (1) \displaystyle-\beta\cdot{\color[rgb]{0,0,0}\textsc{kl}}\big[{\color[rgb]{0,0,0}\pi_{{\color[rgb]{0,0,0}\bm{\theta}}}}({\color[rgb]{0,0,0}\bm{y}}\mid{\color[rgb]{0,0,0}\bm{x}})\,\|\,{\color[rgb]{0,0,0}\pi_{\textsc{sft}}}({\color[rgb]{0,0,0}\bm{y}}\mid{\color[rgb]{0,0,0}\bm{x}})\big], where \beta is a hyperparameter controlling the adherence to {\color[rgb]{0,0,0}\pi_{\textsc{sft}}} . This objective is typically optimized using algorithms such as PPO or other actor–critic methods ( 19 ; 8 ) . As proposed by 24 , the optimal {\color[rgb]{0,0,0}\pi^{\star}} can be expressed in the closed form: \displaystyle{\color[rgb]{0,0,0}\pi^{\star}}({\color[rgb]{0,0,0}\bm{y}}\mid{\color[rgb]{0,0,0}\bm{x}})=\frac{1}{Z({\color[rgb]{0,0,0}\bm{x}})}\,{\color[rgb]{0,0,0}\pi_{\textsc{sft}}}({\color[rgb]{0,0,0}\bm{y}}\mid{\color[rgb]{0,0,0}\bm{x}})\exp\Big(\frac{1}{\beta}{\color[rgb]{0,0,0}r}({\color[rgb]{0,0,0}\bm{x}},{\color[rgb]{0,0,0}\bm{y}})\Big). (2) where Z({\color[rgb]{0,0,0}\bm{x}}) is the partition function ensuring normalization. Direct Preference Optimization. 24 observe that by rearranging the terms