Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG
Seungju Han, Konwoo Kim, Chanwoo Park, Benjamin Newman, Suhas Kotha, Jaehun Jung, James Zou, Yejin Choi · 2026
Read Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG by Seungju Han, Konwoo Kim, Chanwoo Park, Benjamin Newman, Suhas Kotha, Jaehun Jung, James Zou, Yejin Choi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23562.txt.
Description
Han and seven co-authors teach models new facts with mixed synthetic data that can beat retrieval. Synthetic drills alone hit a ceiling below RAG as they scale. Their Synthetic Mixed Training blends made-up questions with made-up documents. The two streams give complementary signals, so gains grow log-linearly with data and generator strength. A Focal Rewriting trick ties each document to set questions for more variety. On QuaLITY, a tuned Llama 8B beats RAG by a clear margin. Across six settings, the recipe wins five.
Catalog record
| CREATOR | Seungju Han, Konwoo Kim, Chanwoo Park, Benjamin Newman, Suhas Kotha, Jaehun Jung, James Zou, Yejin Choi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43583 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.23562