Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning
Jaehun Jung, Seungju Han, Ximing Lu, Skyler Hallinan, David Acuna, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Yejin Choi · 2025
Read Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning by Jaehun Jung, Seungju Han, Ximing Lu, Skyler Hallinan, David Acuna, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Yejin Choi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20161.txt.
Description
Jaehun Jung and colleagues measure training data diversity by what it does to the model, not by how varied it looks. Diversity drives generalization, yet surface heuristics for it miss the mark. Writing in twenty twenty-five, the team ran hundreds of controlled training runs. Diversity proved a strong predictor of performance on unseen reasoning tests. Their new metric scores a data set by the entropy of the gradients it induces in a small proxy model. It correlated tightly with out-of-distribution success on inference and math tasks, beating alternative measures.
The companion framework generates synthetic data aimed at underrepresented regions of that gradient space. Models trained on the prismatic output kept improving as synthetic data scaled. Gains showed on familiar tests and on unseen ones. The paper reframes data curation as gradient coverage.
Catalog record
| CREATOR | Jaehun Jung, Seungju Han, Ximing Lu, Skyler Hallinan, David Acuna, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Yejin Choi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 89 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20161