Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning

Jaehun Jung, Seungju Han, Ximing Lu, Skyler Hallinan, David Acuna, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Yejin Choi · 2025

Read Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning by Jaehun Jung, Seungju Han, Ximing Lu, Skyler Hallinan, David Acuna, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Yejin Choi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20161.txt.

Description

Jaehun Jung and colleagues measure training data diversity by what it does to the model, not by how varied it looks. Diversity drives generalization, yet surface heuristics for it miss the mark. Writing in twenty twenty-five, the team ran hundreds of controlled training runs. Diversity proved a strong predictor of performance on unseen reasoning tests. Their new metric scores a data set by the entropy of the gradients it induces in a small proxy model. It correlated tightly with out-of-distribution success on inference and math tasks, beating alternative measures.

The companion framework generates synthetic data aimed at underrepresented regions of that gradient space. Models trained on the prismatic output kept improving as synthetic data scaled. Gains showed on familiar tests and on unseen ones. The paper reframes data curation as gradient coverage.

Catalog record

CREATORJaehun Jung, Seungju Han, Ximing Lu, Skyler Hallinan, David Acuna, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Yejin Choi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE89 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20161

Browse the full library · About the Tower of Babel