Gained In Translation : Privileged Pairwise Judges Enhance Multilingual Reasoning
Lintang Sutawika, Gokul Swamy, Zhiwei Steven Wu, Graham Neubig · 2026
Read Gained In Translation : Privileged Pairwise Judges Enhance Multilingual Reasoning by Lintang Sutawika, Gokul Swamy, Zhiwei Steven Wu, Graham Neubig (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18722.txt.
Introduction
Lintang Sutawika and three colleagues find a way to lift a model's thought in tongues it was barely trained on. When a hard ask comes in a rare tongue, scores drop far past the score for the same ask in English. That gap locks out much of the world from the best of these tools. The team use no fresh data in the rare tongue at all.
Their plan has the model play it-self in pairs. A judge twin, fed the English ask as a hid edge, grades pairs of tries in the rare tongue. The grades teach the model to pick its best line of thought, tongue by tongue. A lift with no new data is a prize for low-resource tongues. It could spread strong thought tools to lands they now skip.
Catalog record
| CREATOR | Lintang Sutawika, Gokul Swamy, Zhiwei Steven Wu, Graham Neubig |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-18722
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Gained In Translation : Privileged Pairwise Judges Enhance Multilingual Reasoning Lintang Sutawika, Gokul Swamy, Zhiwei Steven Wu, Graham Neubig Abstract When asked a question in a language less seen in its training data, current reasoning large language models (RLMs) often exhibit dramatically lower performance than when asked the same question in English. In response, we introduce SP3F (Self-Play with Privileged Pairwise Feedback), a two-stage framework for enhancing multilingual reasoning without any data in the target language(s). First, we supervise fine-tune (SFT) on translated versions of English question-answer pairs to raise base model correctness. Second, we perform RL with feedback from a pairwise judge in a self-play fashion 52 , with the judge receiving the English reference response as privileged information . Thus, even when none of the model’s responses are completely correct, the privileged pairwise judge can still tell which response is better. End-to-end, SP3F greatly improves base model performance, even outperforming fully post-trained models on multiple math and non-math tasks with less than 1 / 8 1/8 of the training data across the single-language, multilingual, and generalization to unseen language settings. Figure 1: We propose SP3F : Self-Play with Privileged Pairwise Feedback : a method for training multilingual reasoning models without any data in the target language(s). SP3F-7B out-performs Qwen2.5-7B-Instruct across 4 tasks with roughly 1/8 of the training data ( 125,000 for SP3F-7B vs. 1,000,000 for Qwen2.5-7B-Instruct), both in terms of accuracy and language fidelity (did the model answer in the target language?). Figure 2: The second stage of the SP3F pipeline is to perform RL (GRPO, 43 ) with feedback from verifiable rewards 28 and a pairwise judge. To aid in its judgments, the judge LLM is given access to privileged information in the form of an English reference response. Concretely, we sample N responses from the model (left), ask the privileged judge to pick a winner from each pair (center), and then use the average win-rate of each response against the other N-1 samples as the reward for RL (right, 52 ). 1 Introduction Current reasoning large language models (RLMs) are trained on data (e.g., chains of thought, CoTs) that is primarily in English 15 . This means that when an RLM is asked the same question in a non-English language, it often exhibits dramatically lower performance than if it were asked the question in English ( 58 ; 33 ; 45 ; 53 ) . Improving reasoning performance in lower resource languages (e.g., Indonesian, Swahili, Bengali) is challenging as we lack large amounts of data in the target language for supervised fine-tuning (SFT), and the base model’s probability of generating the correct answer might be so low that getting positive signal for reinforcement learning (RL) to succeed is computationally challenging. Furthermore, for reasoning tasks, outcome-level verifiable rewards ( 28 ) that consider just the final answer provide only indirect supervision on the CoT, making exploration challenging due to the sparsity of feedback ( 23 ) . Put together, we face a cold start problem we can’t easily offline fine-tune our way out of. In response, we propose SP3F (Self-Play with Privileged Pairwise Feedback): a two-stage framework for increasing reasoning performance in non-English target language(s) that doesn’t require any data in the target language(s). First, we apply SFT on translated versions of English reference responses to raise our RLM’s probability of generating correct answers. Second, we perform RL with a combination of verifiable rewards (e.g., answer correctness, language fidelity) and preference feedback from an LLM judge ( 61 ) . The LLM judge directly supervises the CoT of the RLM, giving it a direction of improvement even when it can’t produce a correct final answer, which helps mitigate the cold start issue of lower-resource language reasoning. While conceptually promising, the noisiness of the feedback provided by LLM judges makes incorporating them into RL training challenging. First, if the LLM judge itself is unfamiliar with a lower-resource language, it may be unable to provide accurate feedback. In response, we provide the English reference response as privileged information 54 to the judge, asking it to merely pick which of the two RLM responses more closely aligns with the English reference response. This is an easier translation task than judgment in the abstract. We find that the use of privileged information improves judgment quality. Second, due to their pretraining on vast swathes of internet text, LLMs often exhibit intransitive (i.e., cyclic) preferences where they might rank A\succ B , B\succ C , and C\succ A ( 55 ) . Such intransitivity means no scalar reward function can faithfully represent the judge’s preferences, making standard reward modeling fundamentally misspecified. Rather than fitting an inconsistent reward model, we adopt a self-play style approach that optimizes pairwise preferences directly: after sampling a batch of candidate responses, we use the judge to compare all pairs and assign each response a score equal to its empirical win rate. This aggregation converts pairwise judgments into a learning objective that reliably improves the model despite intransitivity ( 52 ) . Put together, we propose to use privileged pairwise judges to provide denser feedback to the RLM during RL. Our key insight is that we can use English reference responses during both SFT and RL by framing both learning problems in terms of translation. We use reference responses as data for translation during SFT and as privileged information for the pairwise judge during downstream RL. Our contribution is three-fold: 1. We introduce SP3F : a multi-step framework for increasing reasoning performance in a target language without data in said language. We find that RLMs trained via SP3F out-perform fully post-trained models on both in-domain math and out-of-domain non-math tasks in a target language. 2. We apply SP3F on data from 18 languages, producing a model that out-performs fully post-trained models using \frac{1}{8} as much training data. We outperform Qwen2.5-7B-Instruct across math and on-math reasoning tasks. We find particularly large improvements on lower-resource languages and see better generalization to unseen languages. 3. We perform an in-depth exploration of the benefits provided by privileged information. We find that privileged information is particularly helpful with reducing the intransitivity of the judge model, as well as in improving detection of correct reasoning chains, even if the final answer is incorrect. Model Overall MGSM MT Math100 Belebele Global MMLU Lite Acc Lang Acc Lang Acc Lang Acc Lang Acc Lang Qwen2.5-7B 14.79 78.78 22.15 90.67 21.16 58.22 7.52 80.39 8.34 85.85 + SFT 21.70 82.11 33.66 91.37 26.72 58.26 12.94 89.18 13.48 89.62 + RLVR 57.79 96.09 65.34 99.75 44.50 86.10 68.18 98.73 53.15 99.78 SP3F-7B 61.91 95.35 72.50 99.38 56.84 82.93 67.54 99.65 50.76 99.45 Qwen2.5-7B-Instruct 55.87 89.21 66.36 98.38 52.12 65.66 56.79 96.59 48.20 96.21 + Translate Test 57.01 85.98 66.15 95.81 60.08 59.34 48.09 92.27 53.73 96.49 Table 1: Across in-domain math tasks (MGSM and MT Math100) and out-of-domain tasks non-math tasks (Belebele and Global MMLU Lite) over a subset of 18 languages ( Table 6 ) that were used to train SP3F-7B , we see SP3F-7B consistently outperforms the Qwen2.5-7B-Instruct. We measure performance in percentage by Accuracy (Acc) and Language Fidelity (Lang). Highest score presented in bold and second highest underlined . Notably, SP3F-7B required only \frac{1}{8} as