Reasoning Gym : Reasoning Environments for Reinforcement Learning with Verifiable Rewards

Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, Andreas Köpf · 2025

Read Reasoning Gym : Reasoning Environments for Reinforcement Learning with Verifiable Rewards by Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, Andreas Köpf (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24760.txt.

Introduction

Zafir Stojanovski and six co-authors present Reasoning Gym, a kit of tasks to train and test reasoning in learning models. Their work appeared in twenty twenty-five. Most past task sets are fixed, so a model can run out of fresh cases or face just one level of hard tasks. This kit makes new cases on the fly, with the hard level set by the user.

The kit holds more than one hundred case makers and checkers. Fields span algebra, sums, code thought, games, shapes, graphs, and logic. Each case can be checked for a right or wrong answer, which suits reward-based training. Tests show the kit works both to score models and to train them.

Catalog record

CREATORZafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, Andreas Köpf
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24760

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reasoning Gym : Reasoning Environments for Reinforcement Learning with Verifiable Rewards Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, Andreas Köpf Abstract We introduce Reasoning Gym ( RG), a library of reasoning environments for reinforcement learning with verifiable rewards. It provides over 100 data generators and verifiers spanning multiple domains including algebra, arithmetic, computation, cognition, geometry, graph theory, logic, and various common games. Its key innovation is the ability to generate virtually infinite training data with adjustable complexity, unlike most previous reasoning datasets, which are typically fixed. This procedural generation approach allows for continuous evaluation across varying difficulty levels. Our experimental results demonstrate the efficacy of RG in both evaluating and reinforcement learning of reasoning models. GitHub Figure 1 : Example RG tasks from three categories. 1 Introduction The reasoning abilities of large language models (LLMs) have recently leapt forward, with models like OpenAI-o1 [ 49 ] , DeepSeek-R1 [ 19 ] , and QwQ-32B [ 72 ] setting new benchmarks. At the heart of this progress is Reinforcement Learning with Verifiable Rewards (RLVR) [ 36 , 19 ] , which leverages outcome-based feedback to unlock open-ended reasoning processes with diverse solution paths. However, the success of RLVR hinges critically on the availability of high-quality training data. Current approaches face a fundamental scalability bottleneck [ 40 ] : they depend either on expensive human-curated question-answer pairs [ 11 ] or on internet-scraped content [ 46 , 1 ] that is neither sustainable nor reliable in the long term [ 74 , 29 ] . As reasoning models continue to advance, this data scarcity threatens to become an increasingly severe constraint on further progress. We address this challenge with Reasoning Gym ( RG), a comprehensive library of procedurally generated [ 10 ] reasoning environments designed specifically for RLVR training. Unlike traditional reasoning benchmarks that provide fixed datasets, RG offers over 100 algorithmically verifiable tasks that can generate unlimited training instances with controllable difficulty and structural variation. These environments span diverse reasoning domains: symbolic algebra, discrete algorithms, spatial geometry, formal logic, pattern recognition, and constraint-based puzzles. Each task is equipped with verification mechanisms and parameters that enable fine-grained control over problem complexity. The procedural nature of RG addresses several critical limitations of existing approaches. First, it eliminates memorization concerns by ensuring that no two generated instances are identical. Second, it enables dynamic curriculum learning, where task difficulty can be adjusted based on model performance. Third, it provides unlimited training data, removing the bottleneck imposed by fixed dataset sizes. Finally, it offers precise experimental control, allowing researchers to isolate specific reasoning capabilities and study their development systematically. Our experimental investigation reveals several key insights: • Zero-shot performance of frontier LLMs is low for many RG tasks , specifically the ones that represent visual concepts in text format like ARC, cognition, and games categories. • Increasing task difficulty creates sharp performance cliffs . When transitioning from easy to hard configurations, performance drops are most severe in algorithmic reasoning ( 28\% ), code generation ( 62\% ), and graph problems ( 30\% ). • Larger non-reasoning models often underperform smaller reasoning models. Performance drops are highest when transitioning from reasoning to non-reasoning models, underlining the value of reasoning data. • Curriculum RLVR results in improved final models , with adaptive difficulty progression outperforming fixed-difficulty training across reasoning tasks. • RLVR generalizes across tasks from the same domain , from mathematics to games. We observe this intra-domain improvement in both tasks the LLM is already competent in, as well as tasks the pre-RLVRed model fails to solve. • Signs of cross-domain transfer emerge from RLVR training. A model trained on algorithmic tasks exhibits substantial improvements in math domains such as algebra and geometry. • Skills transfer to external benchmarks . RLVR training on RG tasks improves performance on benchmarks such as MATH [ 21 ] , GSM8K [ 11 ] , Big-Bench Hard [ 70 ] , and MMLU-Pro [ 79 ] . We release the complete library, including all task generators, training infrastructure, and experimental configurations, at https://github.com/open-thought/reasoning-gym/ . 2 Reasoning Gym ( RG) Despite rapid progress in language model reasoning, empirical work is bottlenecked by benchmarks that are either fixed in size, quickly memorized, or too noisy. Therefore, Reasoning Gym ( RG) is motivated by the need for an open-ended playground where models can be pushed past the “dataset ceiling”, exposed to ever-harder instances, and evaluated with fully automatic, unambiguous rewards so that genuine reasoning improvements, and not dataset familiarity, drive the next wave of advances. Figure 2 : Frontier models struggle with challenging RG configurations. Reasoning models like o3-mini [ 48 ] and DeepSeek-R1 [ 19 ] tend to outperform non-reasoning models, but the tasks configured with challenging parameters are still far from being saturated. Following are the core design principles that underpin RG: (P1) Algorithmic Verifiability. Every task admits automatic verification and requires no human judgment. This enables reliable RLVR training while eliminating subjective evaluation. (P2) Large Solution Spaces. Tasks are designed with expansive solution spaces, rewarding generalizable strategies above overfitting and mitigating reward hacking. (P3) Parametric Difficulty Control. Configurable parameters systematically control problem characteristics, enabling dynamic curricula via precise difficulty adjustment. To probe reasoning competence across a broad spectrum of skills, we partition RG’s generators into several high-level categories that mirror the abstractions humans rely on when solving problems: Mathematical domains : algebra, arithmetic, geometry Algorithmic thinking : search, optimization, procedures Logical reasoning : formal proofs, inference rules Pattern recognition : sequences, visual analogies Constraint satisfaction : games, puzzles, planning Figure 1 shows representative examples demonstrating the diversity of reasoning challenges, and Table 6 outlines the full set of categories alongside the data generators for each. We believe this taxonomy lets practitioners target specific abilities during training or evaluation while still drawing from a rich, procedurally generated mix of challenges. Concretely, within each category we instantiate tasks not as fixed question-answer pairs, but as generative algorithms whose parameters continuously modulate problem characteristics: • Difficulty Parameters directly control complexity (node counts for graphs, polynomial degrees for algebra, word lengths for language tasks). • Structural Parameters determine fundamental problem properties (dimensionality, constraint types, proof depth). • Stylistic Parameters vary presentation without affecting difficulty (variable names, number formats, problem framing). (a) Zero-shot performance of frontier LLMs. (b) Difficulty Cliff: easy vs. hard tasks. Figure 3 : Model and task difficulty comparison. Left: Zero-shot ability across model types on the hard configs. Right: Impact of dataset difficulty on per-category accuracy. Section details the easy and hard parameter configurations for each dataset. 3 Zero-shot performance of LLMs We conduct a comprehensive evaluation of state-of-the-art language models on RG tasks, revealing challenges that persist even for frontier models. Our analysis encompasses both zero-shot capabilities and the effects of task difficulty scaling. Figures and report the precise scores for every task–model combination under the easy and hard settings, respectively. 3.1 Model Capabilities Across Reasoning Domains Figures 2 and 3 present our core findings on model performance across RG tasks. The results reveal a clear hierarchy among different model classes, with reasoning-optimized systems demonstrating substantial advantages over general-purpose alternatives. Reasoning vs. Non-reasoning Models. The performance gap between reasoning-optimized and general-purpose models is striking and consistent.

Read the complete text