Data Recipes for Reasoning Models

Etash Guha*, Ryan Marten*, Sedrick Keh*, Negin Raoof*, Georgios Smyrnis*, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang · 2025

Read Data Recipes for Reasoning Models by Etash Guha*, Ryan Marten*, Sedrick Keh*, Negin Raoof*, Georgios Smyrnis*, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04178.txt.

Introduction

Etash Guha and a large team of co-authors describe open data sets for training reasoning models, in a 2025 paper. Top reasoning models often train on private data, leaving open questions about the best recipes. The OpenThoughts project aims to make such data public. An earlier set from the project trained a model that matched a leading rival on standard tests.

The team then ran more than a thousand controlled tests on each step of their data pipeline. The improved set holds well over a million examples. A model trained on it set new marks for its size on math, code, and science tests. Gains over a key rival ranged from fifteen to twenty points. All data sets and models are public, the authors note, so others can build on them.

Catalog record

CREATOREtash Guha*, Ryan Marten*, Sedrick Keh*, Negin Raoof*, Georgios Smyrnis*, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE188 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-04178

Browse the full library · About the Tower of Babel