Rethinking the Multilingual Reasoning Gap with Layer Swap

Maxence Lasbordes, Amélie Chatelain, Djamé Seddah · 2026

Read Rethinking the Multilingual Reasoning Gap with Layer Swap by Maxence Lasbordes, Amélie Chatelain, Djamé Seddah (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26735.txt.

Introduction

Lasbordes, Chatelain, and Seddah revisit the gap between reasoning in a model's native language and reasoning in English first, training specialists in both regimes on the same base model across six languages. The native gap shrinks to a small margin, and weight analysis reveals a largely language-agnostic reasoning core in the middle layers surrounded by language-specific outer layers. Swapping the English specialist's middle layers into native specialists closes most of the remaining gap.

Catalog record

CREATORMaxence Lasbordes, Amélie Chatelain, Djamé Seddah
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53483
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26735

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking the Multilingual Reasoning Gap with Layer Swap Maxence Lasbordes, Amélie Chatelain, Djamé Seddah Abstract Recent reasoning Large Language Models produce a chain-of-thought (CoT) predominantly in English, even when prompted in non-English languages. Prior work suggests that forcing the CoT to remain in the input language ( native reasoning ) substantially degrades performance relative to allowing the model to reason in English before answering in the input language ( English-pivoted reasoning ). However, most studies of this native reasoning gap rely on inference-time interventions or limited native-language training data. We revisit this comparison at a larger scale and under comparable supervision. We construct long multilingual reasoning datasets across six languages (English, French, German, Spanish, Chinese and Swahili); fine-tune specialists in both native and English-pivoted regimes on top of Qwen/Qwen3-8B-Base , and evaluate across mathematics, science, general knowledge, and code. In this setting, the average native reasoning gap shrinks to 1.9--3.5% across the five non-English languages, considerably smaller than previously reported. Weight-space analysis of the native specialists reveals aligned fine-tuning updates in the middle layers and divergence in the outer layers. This points to a largely language-agnostic reasoning core surrounded by language-specific layers. Exploiting this structure, we introduce a Layer Swap: transferring the English specialist’s stronger reasoning mid-layers into each native specialist, closing most of the native reasoning gap across the five non-English languages while preserving CoT in the target language. We release all models and datasets. 1 1 1 https://huggingface.co/collections/lightonai/multilingual-reasoning Figure 1: ( left ) Layer Swap : transferring a mid-stack window of the English specialist into the native specialist keeps the CoT in the input language while reducing the remaining native reasoning gap. ( right ) The two baselines compared in this work: Native Reasoning (CoT in the input language, here French) and English-pivoted Reasoning (CoT in English regardless of input), the default of open multilingual reasoning models. 1 Introduction Reasoning models such as OpenAI o1 ( 14 ) , DeepSeek-R1 ( 9 ) , and Qwen3 ( 33 ) rely on long chain-of-thought (CoT) to tackle complex tasks in mathematics, code, and science. In current open reasoning models, the CoT is overwhelmingly produced in English, including on non-English inputs: the model switches back to the input language only for the final response ( 23 ; 18 ) , a regime we refer to as English-pivoted reasoning (Figure 1 ). This default has practical costs. English-only reasoning limits interpretability for non-English users, reduces the linguistic and cultural nuance captured by native reasoning traces, and accumulates translation-style errors that compound with task complexity ( 23 ) . Constraining the CoT to remain in the input language, native reasoning (Figure 1 ), is therefore desirable, but most attempts substantially degrade accuracy on key benchmarks ( 3 ; 34 ) . We call models trained for native reasoning in a specific non-English language native specialists . However, these measurements are typically obtained either through prompting alone ( 23 ; 15 ) or through fine-tuning on small amounts of native reasoning data ( 3 ; 20 ) , using post-training budgets far below those required for meaningful native reasoning supervision. Whether the reported gap persists once native post-training is brought to a scale comparable to English-pivoted post-training remains, to our knowledge, understudied. In this work, we revisit the comparison under strictly comparable training conditions. We construct a large multilingual reasoning dataset spanning six languages (French, German, Spanish, Swahili, Chinese, and English), with approximately 500k samples per language up to 32k tokens, and perform supervised fine-tuning (SFT) on Qwen/Qwen3-8B-Base with roughly 10B tokens per language in both regimes (native and English-pivoted). Across general knowledge, mathematics, code and science benchmarks, the native reasoning gap shrinks to 1.9–3.5% on average across the five non-English languages (Figure 2 ), substantially smaller than previous reports suggest ( 3 ; 20 ) ; the gap concentrates on complex reasoning benchmarks, with much smaller gaps on the other benchmarks. Weight-space analysis of the per-language SFT updates further shows that cross-language updates align tightly in the middle layers but diverge at the edges. We exploit this structure with a Layer Swap ( 1 ) (Figure 1 ), a training-free method that transfers the English specialist’s middle layers into each native specialist. To our knowledge, this technique has not previously been applied to long-CoT reasoning models, nor to pairs of experts that share the same reasoning skills but differ in their training language, the setting we study here. Layer Swap closes 83 – 89 % of the gap on French and German, 60 % on Swahili, 27 % on Chinese, and matches the English-pivoted ceiling on Spanish, all while keeping the CoT in the target language (Figure 2 ). Our contributions are: (i) a publicly released long-CoT reasoning corpus across six languages at 32k context, with CoTs in the target language, covering both European and non-European languages, including Swahili and Chinese; (ii) a large-scale, strictly controlled measurement of the native-vs-English-pivoted reasoning gap under controlled SFT token budgets; (iii) a weight-space analysis revealing a largely language-agnostic reasoning core in the middle layers, and a training-free Layer Swap that exploits this structure to close most of the remaining gap while preserving the target-language CoT; and (iv) an input-language ablation showing that input understanding remains a primary bottleneck under matched native SFT. Figure 2: Average accuracy across MGSM-Rev2, Global-MMLU-Lite, GPQA-Diamond, AIME 24/25, and HumanEvalPlus in the target language, for xx \in { fr , de , es , zh , sw }. Three settings are compared per language: Native Reasoning ( Qwen3-8B-xx , CoT in xx ), Layer Swap ( Qwen3-8B-xx-Swap , a mid-stack window of the English specialist transferred into the native specialist, CoT in xx ; see § 5 ), and English-Pivoted Reasoning ( Qwen3-8B-xx-Pivot-EN , CoT in English). 2 Related Work Native multilingual reasoning training Recent work has approached the native reasoning gap through both data and post-training objectives with various degrees of success. Publicly available native long CoT corpora remain scarce ( 7 ) : existing releases provide only a few hundred to a few thousand samples per language ( 3 ; 20 ) , while broader multilingual corpora either retain English CoT or do not target long-form reasoning. Within this limited regime, 3 show that translated long-CoT supervision can effectively train non-English reasoners, motivating our translated-data pipeline at substantially larger scale across six languages. Post-training approaches similarly reveal a substantial native reasoning gap. Under pure SFT on Qwen/Qwen3-8B-Base , 3 find that target-language CoT underperforms English-pivoted reasoning, with AIME 24/25 gaps averaging \sim 19% across nine languages ( \sim 17% for French). 29 report a similar gap in Korean and mitigate it by inserting English anchor segments into the reasoning trace. Reinforcement learning-based (RL) methods offer mixed evidence: 11 show that RL on non-English data can improve cross-lingual transfer, whereas Cross-lingual Collapse ( 18 ) identifies a recurrent failure mode in which CoT drifts back to English as accuracy improves under GRPO ( 26 ) . Hybrid approaches combine both stages: Think Natively ( 34 ) applies SFT followed by GRPO with language-consistency and cross-lingual alignment rewards, while concurrent independent work, ReasonXL ( 10 ) , combines SFT and RLVR on SmolLM3-3B across five European languages at 16k context. Like

Read the complete text