Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs

Shulin Huang, Yiran Ding, Junshu Pan, Yue Zhang · 2025

Read Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs by Shulin Huang, Yiran Ding, Junshu Pan, Yue Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23657.txt.

Introduction

Reasoning skill learned in one language does not always travel to another. Shulin Huang and 3 co-authors run the first systematic test of how reinforcement learning, versus supervised fine-tuning, affects cross-lingual reasoning. Their base model is Qwen2.5-3B, and the tasks cover math, commonsense, and science reasoning across languages. Two findings stand out.

Reinforcement learning not only scored higher but also carried its gains across languages far better than fine-tuning. Training it on non-English data beat training on English, a pattern fine-tuning did not show. Mechanistic analyses probe why the learned strategies stay robust, and the authors draw lessons for fairer multilingual reasoning.

Catalog record

CREATORShulin Huang, Yiran Ding, Junshu Pan, Yue Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-23657

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs Shulin Huang, Yiran Ding, Junshu Pan, Yue Zhang Abstract Enhancing the complex reasoning capabilities of Large Language Models (LLMs) attracts widespread attention. While reinforcement learning (RL) has shown superior performance for improving complex reasoning, its impact on cross-lingual generalization compared to Supervised Fine-Tuning (SFT) remains unexplored. We present the first systematic investigation into cross-lingual reasoning generalization of RL and SFT. Using Qwen2.5-3B-Base as our foundation model, we conduct experiments on diverse multilingual reasoning benchmarks, including math reasoning, commonsense reasoning, and scientific reasoning. Our investigation yields two significant findings: (1) Tuning with RL not only achieves higher accuracy but also demonstrates substantially stronger cross-lingual generalization capabilities compared to SFT. (2) RL training on non-English data yields better overall performance and generalization than training on English data, which is not observed with SFT. Furthermore, through comprehensive mechanistic analyses, we explore the underlying factors of RL’s superiority and generalization across languages. Our results provide compelling evidence that RL enables the model with more robust reasoning strategies, offering crucial guidance for more equitable and effective multilingual reasoning. 1 Introduction Figure 1: SFT and RL performance improvement using English, Chinese and German training data on the same base model, Qwen2.5-3B-Base. Performance improvements are measured relative to the base model. We report the performance improvement in six language settings. Multilingualism plays a significant role in human society and occupies a critical position in the development of large language models (LLMs). With over 7,000 languages worldwide, each encapsulates unique cultural contexts and expressive modalities ( Campbell & Grondona 2008 ) . LLMs not only break down language barriers and facilitate cross-cultural communication, but also enable equitable global Artificial Intelligence benefits ( Howard & Ruder 2018 ; Sharma et al. 2025 ) . With advances in LLMs reasoning ( Hao et al. 2023 ; Yue et al. 2025 ) , cross-lingual reasoning has received increasing attention ( Wang et al. 2024a ; Chai et al. 2025 ; Payoungkhamdee et al. 2025 ) . Multilingual reasoning requires models to not only comprehend the semantic content of different languages, but also to possess the ability to perform logical inference and problem-solving across diverse linguistic environments ( Alam et al. 2024 ) . Current research demonstrates that while large-scale pre-trained language models have achieved remarkable progress in English comprehension and generation, significant performance gaps persist for other languages ( Qiu et al. 2024 ) . Consequently, enhancing multilingual reasoning capabilities and achieving effective cross-lingual generalization has emerged as a significant challenge ( Pham et al. 2024 ; Hu et al. 2025 ) . Reinforcement Learning (RL) is considered a pivotal tool for enhancing reasoning capabilities ( Wang et al. 2024b ; Guo et al. 2025 ) . Recent studies show that RL training exhibits superior performance on complex tasks such as mathematical and logical reasoning ( Xie et al. 2025 ) . Compared to Supervised Fine-tuning (SFT), Reinforcement Learning, through reward-guided mechanisms, enables models with more robust and generalizable reasoning strategies ( Huan et al. 2025 ) . Notably, researches reveal that RL not only significantly improves model performance, but also enables stronger cross-task generalization capabilities ( Shao et al. 2024 ) . In this work, we conduct the first investigation into whether RL exhibits strong generalization capabilities across languages in reasoning. Experimentally, we compare the performance of RL and SFT on diverse languages, exploring their performance across various languages to examine their cross-lingual generalization abilities. To fully reflect reasoning capability, we evaluate performance on diverse multilingual reasoning benchmarks, including math reasoning, commonsense reasoning, and scientific reasoning ( Shi et al. ; She et al. 2024 ; Son et al. 2025 ; Xuan et al. 2025 ; Qi et al. 2025 ) . The empirical investigation yields two notable findings: (1) As illustrated in Figure 1 , RL demonstrates superior performance improvements compared to SFT, with enhanced cross-lingual generalization capabilities. Our results indicate that models trained with RL can more effectively transfer reasoning abilities learned in one language to another. This finding of cross-lingual reasoning is consistent with existing findings for cross-task transfer ( Korkmaz 2024 ; Huan et al. 2025 ; Cheng et al. 2025 ; Chu et al. ) . (2) Given that the pre-training corpora of most existing LLMs are predominantly English-centric ( Morishita et al. 2024 ; Rytting & Wingate 2021 ; Singh et al. 2024 ) , the conventional expectation is that RL training with English data would maximally leverage the model’s potential ( Yoon et al. 2024 ; She et al. 2024 ) . However, as shown in Figure 1, our findings surprisingly reveal a counter-intuitive phenomenon: RL training using non-English data (such as Chinese and German) yields better cross-lingual reasoning performance and superior generalization than using English data. This contrasts with SFT, where no such phenomenon is observed, as performance remains comparable and, in some datasets, even shows an opposite trend. To investigate underlying reasons for the findings, we conduct preliminary analyses: (1) First, we analyze whether the language used in reasoning is consistent with the language of input question. Our investigation reveals that language inconsistency serves as a potential factor of RL’s cross-lingual generalization, and the superiority of the non-English data in RL. (2) Second, we examine the role of sampling mechanisms in RL’s superior performance. We find that the sampling mechanism in RL explores sufficient and diverse solution paths, allowing models to learn more robust and generalizable strategies. (3) Third, we explore the semantic shift of the model after training. We find that the stability of the semantic space contributes to RL’s superior cross-lingual generalization. Our preliminary explorations provide insights for future research in multilingual reasoning. The main contributions of this work are as follows: (1) We present the first systematic analysis of the differences between RL and SFT in cross-lingual reasoning generalization, filling an important gap in this research area. (2) We reveal two significant findings: 1) RL excels over SFT in cross-lingual generalization, and 2) Counterintuitively, non-English data is superior to English data for RL training. To our knowledge, we are the first to demonstrate that using non-English data for RL more effectively enhances performance and cross-lingual generalization, although most models are pre-trained mainly on English. (3) Through comprehensive analyses, we explore three potential factors underlying RL’s superiority: linguistic inconsistency in reasoning, sampling-driven policy optimization, and the semantic shift after training, which provides a crucial foundation for multilingual reasoning. 2 Related Work Multilingual Reasoning. Multilingual reasoning is a challenging and representative task for evaluating the intelligence of large language models ( Ahn et al. 2024 ; She et al. 2024 ; Yoon et al. 2024 ) . Shi et al. establish the foundation for this field by translating English mathematical problems from GSM8K ( Cobbe et al. 2021 ) into multiple languages, creating the multilingual benchmark MGSM ( Shi et al. ) . To enhance multilingual reasoning capabilities, existing work primarily employs prompting strategies. Qin et al. 2023 and Huang et al. 2023 propose a translate-then-solve approach that first translates non-English questions into English before problem-solving, achieving promising results on closed-source models like ChatGPT ( Ouyang et al. 2022 ) . However, less attention has been paid to how

Read the complete text