DataRater: Meta-Learned Dataset Curation

Dan A. Calian, Gregory Farquhar, Iurii Kemaev, Luisa M. Zintgraf, Jeremy Shar, Junhyuk Oh, András György, Tom Schaul, Jeffrey Dean, Hado van Hasselt, David Silver · 2025

Read DataRater: Meta-Learned Dataset Curation by Dan A. Calian, Gregory Farquhar, Iurii Kemaev, Luisa M. Zintgraf, Jeremy Shar, Junhyuk Oh, András György, Tom Schaul, Jeffrey Dean, Hado van Hasselt, David Silver (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17895.txt.

Description

Dan Calian and co-authors present DataRater, a system that learns which training examples are worth teaching to a language model, in a 2025 paper. Data quality drives model quality, yet curation today relies on hand-tuned mixtures and rough filters. DataRater instead scores each example by meta-learning: it estimates how much training on that example would help performance on held-out data.

The scoring is learned through meta-gradients, so the rater improves at judging value as it sees more training runs. Across model sizes and datasets, filtering training data with DataRater significantly improved compute efficiency, meaning models reached the same quality with less computation. The authors argue learned curation scales better than manual rules.

Catalog record

CREATORDan A. Calian, Gregory Farquhar, Iurii Kemaev, Luisa M. Zintgraf, Jeremy Shar, Junhyuk Oh, András György, Tom Schaul, Jeffrey Dean, Hado van Hasselt, David Silver
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17895

Browse the full library · About the Tower of Babel