Rethinking the Multilingual Reasoning Gap with Layer Swap
Maxence Lasbordes, Amélie Chatelain, Djamé Seddah · 2026
Read Rethinking the Multilingual Reasoning Gap with Layer Swap by Maxence Lasbordes, Amélie Chatelain, Djamé Seddah (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26735.txt.
Introduction
Lasbordes, Chatelain, and Seddah revisit the gap between reasoning in a model's native language and reasoning in English first, training specialists in both regimes on the same base model across six languages. The native gap shrinks to a small margin, and weight analysis reveals a largely language-agnostic reasoning core in the middle layers surrounded by language-specific outer layers. Swapping the English specialist's middle layers into native specialists closes most of the remaining gap.
Catalog record
| CREATOR | Maxence Lasbordes, Amélie Chatelain, Djamé Seddah |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53483 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26735
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rethinking the Multilingual Reasoning Gap with Layer Swap Maxence Lasbordes, Amélie Chatelain, Djamé Seddah Abstract Recent reasoning Large Language Models produce a chain-of-thought (CoT) predominantly in English, even when prompted in non-English languages. Prior work suggests that forcing the CoT to remain in the input language ( native reasoning ) substantially degrades performance relative to allowing the model to reason in English before answering in the input language ( English-pivoted reasoning ). However, most studies of this native reasoning gap rely on inference-time interventions or limited native-language training data. We revisit this comparison at a larger scale and under comparable supervision. We construct long multilingual reasoning datasets across six languages (English, French, German, Spanish, Chinese and Swahili); fine-tune specialists in both native and English-pivoted regimes on top of Qwen/Qwen3-8B-Base , and evaluate across mathematics, science, general knowledge, and code. In this setting, the average native reasoning gap shrinks to 1.9--3.5% across the five non-English languages, considerably smaller than previously reported. Weight-space analysis of the native specialists reveals aligned fine-tuning updates in the middle layers and divergence in the outer layers. This points to a largely language-agnostic reasoning core surrounded by language-specific layers. Exploiting this structure, we introduce a Layer Swap: transferring the English specialist’s stronger reasoning mid-layers into each native specialist, closing most of the native reasoning gap across the five non-English languages while preserving CoT in the target language. We release all models and datasets. 1 1 1 https://huggingface.co/collections/lightonai/multilingual-reasoning Figure 1: ( left ) Layer Swap : transferring a mid-stack window of the English specialist into the native specialist keeps the CoT in the input language while reducing the remaining native reasoning gap. ( right ) The two baselines compared in this work: Native Reasoning (CoT in the input language, here French) and English-pivoted Reasoning (CoT in English regardless of input), the default of open multilingual reasoning models. 1 Introduction Reasoning models such as OpenAI o1 ( 14 ) , DeepSeek-R1 ( 9 ) , and Qwen3 ( 33 ) rely on long chain-of-thought (CoT) to tackle complex tasks in mathematics, code, and science. In current open reasoning models, the CoT is overwhelmingly produced in English, including on non-English inputs: the model switches back to the input language only for the final response ( 23 ; 18 ) , a regime we refer to as English-pivoted reasoning (Figure 1 ). This default has practical costs. English-only reasoning limits interpretability for non-English users, reduces the linguistic and cultural nuance captured by native reasoning traces, and accumulates translation-style errors that compound with task complexity ( 23 ) . Constraining the CoT to remain in the input language, native reasoning (Figure 1 ), is therefore desirable, but most attempts substantially degrade accuracy on key benchmarks ( 3 ; 34 ) . We call models trained for native reasoning in a specific non-English language native specialists . However, these measurements are typically obtained either through prompting alone ( 23 ; 15 ) or through fine-tuning on small amounts of native reasoning data ( 3 ; 20 ) , using post-training budgets far below those required for meaningful native reasoning supervision. Whether the reported gap persists once native post-training is brought to a scale comparable to English-pivoted post-training remains, to our knowledge, understudied. In this work, we revisit the comparison under strictly comparable training conditions. We construct a large multilingual reasoning dataset spanning six languages (French, German, Spanish, Swahili, Chinese, and English), with approximately 500k samples per language up to 32k tokens, and perform supervised fine-tuning (SFT) on Qwen/Qwen3-8B-Base with roughly 10B tokens per language in both regimes (native and English-pivoted). Across general knowledge, mathematics, code and science benchmarks, the native reasoning gap shrinks to 1.9–3.5% on average across the five non-English languages (Figure 2 ), substantially smaller than previous reports suggest ( 3 ; 20 ) ; the gap concentrates on complex reasoning benchmarks, with much smaller gaps on the other benchmarks. Weight-space analysis of the per-language SFT updates further shows that cross-language updates align tightly in the middle layers but diverge at the edges. We exploit this structure with a Layer Swap ( 1 ) (Figure 1 ), a training-free method that transfers the English specialist’s middle layers into each native specialist. To our knowledge, this technique has not previously been applied to long-CoT reasoning models, nor to pairs of experts that share the same reasoning skills but differ in their training language, the setting we study here. Layer Swap closes 83 – 89 % of the gap on French and German, 60 % on Swahili, 27 % on Chinese, and matches the English-pivoted ceiling on Spanish, all while keeping the CoT in the target language (Figure 2 ). Our contributions are: (i) a publicly released long-CoT reasoning corpus across six languages at 32k context, with CoTs in the target language, covering both European and non-European languages, including Swahili and Chinese; (ii) a large-scale, strictly controlled measurement of the native-vs-English-pivoted reasoning gap under controlled SFT token budgets; (iii) a weight-space analysis revealing a largely language-agnostic reasoning core in the middle layers, and a training-free Layer Swap that exploits this structure to close most of the remaining gap while preserving the target-language CoT; and (iv) an input-language ablation showing that input understanding remains a primary bottleneck under matched native SFT. Figure 2: Average accuracy across MGSM-Rev2, Global-MMLU-Lite, GPQA-Diamond, AIME 24/25, and HumanEvalPlus in the target language, for xx \in { fr , de , es , zh , sw }. Three settings are compared per language: Native Reasoning ( Qwen3-8B-xx , CoT in xx ), Layer Swap ( Qwen3-8B-xx-Swap , a mid-stack window of the English specialist transferred into the native specialist, CoT in xx ; see § 5 ), and English-Pivoted Reasoning ( Qwen3-8B-xx-Pivot-EN , CoT in English). 2 Related Work Native multilingual reasoning training Recent work has approached the native reasoning gap through both data and post-training objectives with various degrees of success. Publicly available native long CoT corpora remain scarce ( 7 ) : existing releases provide only a few hundred to a few thousand samples per language ( 3 ; 20 ) , while broader multilingual corpora either retain English CoT or do not target long-form reasoning. Within this limited regime, 3 show that translated long-CoT supervision can effectively train non-English reasoners, motivating our translated-data pipeline at substantially larger scale across six languages. Post-training approaches similarly reveal a substantial native reasoning gap. Under pure SFT on Qwen/Qwen3-8B-Base , 3 find that target-language CoT underperforms English-pivoted reasoning, with AIME 24/25 gaps averaging \sim 19% across nine languages ( \sim 17% for French). 29 report a similar gap in Korean and mitigate it by inserting English anchor segments into the reasoning trace. Reinforcement learning-based (RL) methods offer mixed evidence: 11 show that RL on non-English data can improve cross-lingual transfer, whereas Cross-lingual Collapse ( 18 ) identifies a recurrent failure mode in which CoT drifts back to English as accuracy improves under GRPO ( 26 ) . Hybrid approaches combine both stages: Think Natively ( 34 ) applies SFT followed by GRPO with language-consistency and cross-lingual alignment rewards, while concurrent independent work, ReasonXL ( 10 ) , combines SFT and RLVR on SmolLM3-3B across five European languages at 16k context. Like