rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset

Yifei Liu †, Li Lyna Zhang †⋄, Yi Zhu †, Bingcheng Dong † §, Xudong Zhou †‡, Ning Shang †, Fan Yang †, Mao Yang † · 2025

Read rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset by Yifei Liu †, Li Lyna Zhang †⋄, Yi Zhu †, Bingcheng Dong † §, Xudong Zhou †‡, Ning Shang †, Fan Yang †, Mao Yang † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21297.txt.

Introduction

Teaching language models to solve hard coding problems takes large sets of problems with checked answers, and such sets are scarce. Yifei Liu and seven co-authors built one in 2025 under the name rStar-Coder. It holds 418 thousand contest level problems and 580 thousand long worked solutions, each backed by test cases.

The team made new problems from contest sets with known solutions, then built test inputs and outputs through a three step process with cross checks. Models trained on the data gained sharply: one mid sized model jumped from about seventeen percent to fifty-seven percent on a live coding bench. A smaller model even beat a much larger rival on a computing olympiad bench.

Catalog record

CREATORYifei Liu †, Li Lyna Zhang †⋄, Yi Zhu †, Bingcheng Dong † §, Xudong Zhou †‡, Ning Shang †, Fan Yang †, Mao Yang †
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21297

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset Yifei Liu †, Li Lyna Zhang †⋄, Yi Zhu †, Bingcheng Dong † §, Xudong Zhou †‡, Ning Shang †, Fan Yang †, Mao Yang † Abstract Advancing code reasoning in large language models (LLMs) is fundamentally limited by the scarcity of high-difficulty datasets, especially those with verifiable input-output test cases necessary for rigorous solution validation at scale. We introduce rStar-Coder , which significantly improves LLM code reasoning capabilities by constructing a large-scale, verified dataset of 418K competition-level code problems, 580K long-reasoning solutions along with rich test cases of varying difficulty. This is achieved through three core contributions: (1) we curate competitive programming code problems and oracle solutions to synthesize new, solvable problems; (2) we introduce a reliable input-output test case synthesis pipeline that decouples the generation into a three-step input generation method and a mutual verification mechanism for effective output labeling ; (3) we augment problems with high-quality, test-case-verified long-reasoning solutions. Extensive experiments on Qwen models (1.5B-14B) across various code reasoning benchmarks demonstrate the superiority of rStar-Coder dataset, achieving leading performance comparable to frontier reasoning LLMs with much smaller model sizes. On LiveCodeBench, rStar-Coder improves Qwen2.5-7B from 17.4% to an impressive 57.3%, and Qwen2.5-14B from 23.3% to 62.5%, surpassing o3-mini (low) by3.1%. On the more challenging USA Computing Olympiad, our 7B model achieves an average pass@1 accuracy of 16.15%, outperforming the frontier-level QWQ-32B. Code and the dataset will be released at https://github.com/microsoft/rStar . Figure 1: Pass@1 accuracy on code reasoning benchmarks. rStar-Coder consistently delivers competitive performance compared to significantly larger state-of-the-art reasoning LLMs. On the highly challenging USACO 2025, rStar-Coder outperforms QWQ-32B despite being at the 7B scale. $\diamond$ $\diamond$ footnotetext: Project leader; correspondence to [email protected] $\lx@sectionsign\ddagger$ $\lx@sectionsign\ddagger$ footnotetext: Bingcheng Dong and Xudong Zhou did this work during the internship at MSRA. 1 Introduction Recent large language models (LLMs) have made significant strides in reasoning, with models like OpenAI o1/o3 ( 17 ) and DeepSeek-R1 ( 9 ) showing strong performance on complex code problems. However, to further improve advanced code reasoning, a persistent challenge is the scarcity of large-scale, high-quality datasets that contain verifiable, high-difficulty programming problems requiring both algorithmic thinking and efficient code implementations. Unlike datasets on math word problems, where solution correctness can be verified through simple rule-based matching against reference answers ( 9 , 8 ) , code reasoning requires executing solutions against diverse test cases to detect logical and implementation errors. These input-output test cases must differ in content, scale, and complexity while meeting problem-specific constraints. Critically, each test output must correctly label the expected result for its corresponding test input. Obtaining such reliable and verifiable input-output test cases at scale remains highly challenging. Human-curated datasets, such as CodeContests ( 22 ) and TACO ( 20 ) , provide high-quality problems but typically lack comprehensive test coverage, often including only a few simple test cases that fail to capture diverse input scales of edge conditions. Synthetic datasets such as WizardCoder ( 24 ) , Magicoder ( 30 ) , and KodCode ( 31 ) primarily target function-level code generation, where correctness can be verified with minimal or even no test cases. These datasets are therefore insufficient for advanced competitive-level code reasoning. While frontier LLMs may offer a scalable way for test case synthesis, two substantial challenges arise: generating semantically valid, constraint-aware inputs varying in scale and difficulty; and assigning correct outputs without ground truth solutions, particularly since synthetic problems inherently lack reference implementations. In this paper, we introduce rStar-Coder, a novel approach that reliably constructs a large-scale, high-difficulty dataset for training advanced code reasoning LLMs. This dataset includes 418K unique competitive programming problems and 580K long-reasoning solutions, each verified through diverse synthetic test cases across difficulty levels. rStar-Coder incorporates three key components. First, we curate and clean 37.7K expert-written problems with oracle solutions from competitive programming platforms (e.g., IOI, Codeforces) and use them as seeds to synthesize new, solvable problems. Unlike prior works ( 31 , 24 ) that emphasize diversity, we prioritize solvability and correctness, since the seed set already spans a wide range of topics and difficulty levels. However, these problems are often too difficult for even frontier models like GPT-4o, and directly prompting the LLM with only the problem statement often results in invalid or unsolvable outputs. To address this, we design a structured prompt that incorporates both the problem statement and its oracle solution, guiding the model to understand core algorithmic concepts and generate novel yet solvable problems. Second, we address the challenge of generating reliable and diverse input-output test cases for solution validation by decoupling the process into two stages. (1) First, we propose a three-step approach to generate valid test inputs of varying scale and complexity. We prompt GPT-4o to generate two utility functions for each problem: one for synthesizing semantically valid inputs with exposed scale-controlling parameters, and another for validating whether the inputs satisfy problem-specific constraints. We then sample input scale values (10 0 - 10 5 ) for these scale-controlling parameters to cover a diverse range of complexities. Executing the two utility functions with these sampled scales yields diverse, constraint-satisfying test inputs. (2) Second, to reliably label outputs, we introduce a mutual verification mechanism. We sample multiple long-reasoning solutions from a strong reasoning model (QWQ-32B), and accept both the test outputs and solutions if a majority produce consistent results across all test inputs . This mechanism is effective because incorrect solutions tend to diverge in errors, while correct solutions converge. Thus, consistent agreement among multiple solutions on identical outputs across diverse test inputs, particularly those of varying complexity, serves as strong evidence for the correctness of both the solution logic and the resulting outputs. Our ablation study demonstrate the effectiveness of this approach, achieving 96.8% accuracy in output labeling. Finally, we augment curated expert-written problems with verified long-reasoning solutions. While these problems are of high-quality, their original solutions often lack detailed reasoning steps. Generating them directly with frontier LLMs is unreliable, as few simple original tests are insufficient for validation. To address this, we use our test generation method to produce diverse, constraint-aware inputs. Since curated problems include oracle solutions, we run them to get ground-truth outputs. We then prompt QWQ-32B for long-reasoning solutions, keeping only those that pass all generated tests. Extensive experiments across different-sized LLMs (1.5B-14B) and diverse code reasoning benchmarks demonstrate the effectiveness of rStar-Coder, consistently improving the code reasoning capabilities of all base models to state-of-the-art levels, even with significantly smaller sizes. On LiveCodeBench, rStar-Coder improves the 14B model from 23.3% to 62.5%, surpassing R1-distill-70B and o3-mini (low) by +5.0% and 3.1%. And even the 1.5B model reaches 40.1%, outperforming both R1-distill-7B and GPT-4o. On the highly competitive USA Computing Olympiad, rStar-Coder-7B and rStar-Coder-14B outperform the frontier reasoning model QWQ-32B. Furthermore, rStar-Coder also generalizes remarkably well on standard code generation tasks like HumanEval and MBPP. 2 Related Works Instruction Code Data Synthesis . Prior to the rise of reasoning-focused LLMs, code data synthesis methods ( 24 , 30 , 32 ) primarily targeted scaling instruction data to improve LLMs capabilities to generate code aligned with use intent. Efforts like Code Alpaca ( 4 ) , WizardCoder ( 24 ) , Magicoder (

Read the complete text