Influence-driven Curriculum Learning for Pre-training on Limited Data
Loris Schoenegger, Lukas Thoma, Terra Blevins, Benjamin Roth · 2025
Read Influence-driven Curriculum Learning for Pre-training on Limited Data by Loris Schoenegger, Lukas Thoma, Terra Blevins, Benjamin Roth (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15475.txt.
Description
Loris Schoenegger and three co-authors revisit curriculum learning in twenty twenty-five. Curriculum learning feeds training data in order of difficulty, from simple to complex. For pretraining language models, it had shown only limited success. The authors suspect the fault lies in how difficulty was measured, by human-centered scores.
They sort examples instead by training data influence. That score estimates how much each example changes the model's output. Models trained on influence-ordered curricula beat random order by over ten percentage points on benchmarks. The gain held in the limited-data setting they studied. Readers learn that curriculum learning works for pretraining when difficulty is defined by the model, not by people.
Catalog record
| CREATOR | Loris Schoenegger, Lukas Thoma, Terra Blevins, Benjamin Roth |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-15475