How Reasoning Evolves from Post-Training Data: An Empirical Study Using Chess

Lucas Dionisopoulos, Nicklas Majamaki, Prithviraj Ammanabrolu · 2026

Read How Reasoning Evolves from Post-Training Data: An Empirical Study Using Chess by Lucas Dionisopoulos, Nicklas Majamaki, Prithviraj Ammanabrolu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05134.txt.

Introduction

Dionisopoulos, Majamaki, and Ammanabrolu study how thinking evolves from post-training data, using chess as a testbed. They trace a language model from guided fine-tuning to reinforcement learning, asking how theory-inspired sets shape chess performance. Fine-tuning a model to predict the best move leads to reinforcement learning and downstream results. Yet reinforcement learning elicits false thinking, thinking at odds with the chosen move. Training on multi-move paths instead yields similar downstream results with faithful thinking and more stable learning.

Catalog record

CREATORLucas Dionisopoulos, Nicklas Majamaki, Prithviraj Ammanabrolu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61744
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.05134

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Reasoning Evolves from Post-Training Data: An Empirical Study Using Chess Lucas Dionisopoulos, Nicklas Majamaki, Prithviraj Ammanabrolu Abstract We study how reasoning evolves in a language model – from supervised fine-tuning (SFT) to reinforcement learning (RL) – by analyzing how a set of theoretically-inspired datasets influences language model performance in chess. We find that fine-tuning a model to directly predict the best move leads to effective RL and the strongest downstream performance – however, the RL stage elicits unfaithful reasoning (reasoning inconsistent with the chosen move). Alternatively, training on multi-move trajectories yields comparable downstream performance with faithful reasoning and more stable RL. We analyze multiple qualitative and quantitative measures and highlight how these evolve from SFT through RL; we find several SFT-checkpoint metrics – spanning evaluation performance, hallucination rates, and reasoning quality – to be predictive of post-RL model performance. Finally, we ground our results with an experiment measuring chess information density in our custom datasets. We release models as well as training data, evaluations, and code that allowed us to surpass leading open-source reasoning models in chess with a 7B-parameter model. Code, models, and data are accessible at lang-chess . 1 Introduction What is required to train a language model to reason through RL? Several ingredients appear critical – a strong base model and a compatible domain are sensible starting points. But what is a strong base model? And once you have a domain, how do you train it to reason effectively? We seek to address these motivating questions by training a language model to reason in a verifiable, sequential decision process. Specifically, we choose chess as our focus because of several convenient elements: intrinsic difficulty for LLMs, established theory, favorable structure (episodic MDP), large datasets, and an efficient oracle (chess engines) for verifiable rewards and high-quality synthetic data generation. As a result, we can measure, in a controlled setting, how different training datasets influence our language model through SFT and how RL further evolves reasoning. Language-based reasoning ( 24 ) has emerged as a technique to advance language model capabilities, although research remains largely confined to domains such as math and coding. Reasoning, often characterized as extending a model’s "chain-of-thought" behavior ( 38 ) using methods such as RL, benefits from these domains being clearly verifiable: the math is correct or the code passes all tests. This verifiable nature – combined with downstream applicability and skill transfer – has stimulated much research in these domains. As a result, reasoning models have achieved profound results across related benchmarks ( 25 ; 2 ; 10 ) , long-duration tasks ( 15 ; 22 ) , and earned an International Math Olympiad gold medal ( 6 ; 27 ) . Figure 1 : Following initial data inclusion experiments, we scaled SFT on our two best-performing datasets. Both resulted in comparably strong final evaluation performance, but training on optimal move trajectories ( Best Line ) led to more stable RL and faithful reasoning compared to training on single best moves ( Best Move ). Although recent work explores reasoning in more subjective domains ( 39 ) through techniques such as "LLM-as-a-judge" ( 48 ) , we focus here on those that are verifiable. The core verifiable domains – math and coding – benefit from years of continued research that has established a corpus of high-quality training data which produces strong out-of-the-box performance. While we benefit from stronger general base models, working on these explored domains would pose a challenge in isolating the influence of training interventions. Chess thus emerges as attractive for studying reasoning. Language models have historically struggled with chess ( 1 ; 8 ) and even state-of-the-art reasoning models still falter – often responding with illegal moves or simple blunders as seen in the Kaggle AI Chess Exhibition ( 12 ) . Models underperform partly because chess data, while abundant, is rarely emphasized in pretraining; further, the game’s combinatorial structure makes generalization challenging. While this poses a difficulty, access to superhuman verification (in chess engines) provides an efficient method for verifiable rewards and synthetic data generation. All these aspects make chess a compelling testbed for reasoning. Motivated by this setting, we train a 7B-parameter model on custom datasets using both SFT and RL to achieve performance surpassing gpt-oss-120b ( 26 ) on several benchmarks. We focus our study on the following: • Q1 : How do different datasets ( e.g., programmatically generated, various synthetic generations ) impact downstream performance after SFT and RL? • Q2 : How does RL influence a model’s qualitative behaviors ( e.g., move quality distribution, reasoning strategies used, rate of hallucination )? • Q3 : Which SFT-checkpoint metrics are predictive of final RL performance? Figure 2 : Performance of our best reasoning model trained from Qwen2.5 7B-Instruct across our evaluations . Trivial performance (i.e., random guessing) is 0.2 for the Best Move and Worst Move tasks. See Appendix B for example evaluation questions and Appendix D for full results. We show that focused SFT on predicting a single best move ( Best Move ) leads to strong performance but unfaithful reasoning through RL; by contrast, training on multi-step move trajectories ( Best Line ) has more stable RL and faithful reasoning. We find that RL leads to fewer hallucinations and a substantial positive shift in move quality, and we see that several SFT-checkpoint metrics (both qualitative and quantitative ) are predictive of final RL performance. Finally, we ground our results with an experiment measuring chess information density for each custom dataset. 2 Related work 2.1 Reasoning in language models Reasoning in causal language models can be interpreted as self-guided search that makes a task more tractable. Consider a numerical math problem: effective reasoning should increase the probability of producing the correct number more than if the model had immediately predicted the final answer. Note that this reasoning need not be wholly interpretable – for example, it can exist in continuous space ( 11 ) or shift between languages ( 7 ) – what ultimately matters is that the intermediate steps are beneficial to the model. For this work, we focus on language-based reasoning. The era of reasoning models , notably initiated with OpenAI’s release of o1 ( 24 ) , builds upon much prior work in self-guided in-context adaptation. Models, when told to work "step by step" and write down intermediate results on a "scratchpad" ( 23 ) , saw performance improvement on multi-step computations – this result was reinforced at scale and termed "chain-of-thought" in later work ( 38 ) . Further, these reasoning traces can be used for iterated improvement through fine-tuning on successful generations as evidenced by STaR ( 45 ) . Quiet-STaR ( 44 ) extended this from fine-tuning by using a reinforcement learning policy-gradient update in REINFORCE ( 40 ) over tokens influenced by intermediate reasoning steps. This iterated bootstrapping using RL for policy-gradient updates has been the primary underlying method fueling the latest developments in reasoning models. Following OpenAI’s release of o1, many leading systems began incorporating similar reasoning techniques to improve performance. Notably, DeepSeek-R1 ( 7 ) and Kimi k1.5 ( 13 ) were among the first reasoning models to effectively approach state-of-the-art ability and publicize the training methods.

Read the complete text