Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG

Seungju Han, Konwoo Kim, Chanwoo Park, Benjamin Newman, Suhas Kotha, Jaehun Jung, James Zou, Yejin Choi · 2026

Read Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG by Seungju Han, Konwoo Kim, Chanwoo Park, Benjamin Newman, Suhas Kotha, Jaehun Jung, James Zou, Yejin Choi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23562.txt.

Description

Han and seven co-authors teach models new facts with mixed synthetic data that can beat retrieval. Synthetic drills alone hit a ceiling below RAG as they scale. Their Synthetic Mixed Training blends made-up questions with made-up documents. The two streams give complementary signals, so gains grow log-linearly with data and generator strength. A Focal Rewriting trick ties each document to set questions for more variety. On QuaLITY, a tuned Llama 8B beats RAG by a clear margin. Across six settings, the recipe wins five.

Catalog record

CREATORSeungju Han, Konwoo Kim, Chanwoo Park, Benjamin Newman, Suhas Kotha, Jaehun Jung, James Zou, Yejin Choi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43583
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.23562

Browse the full library · About the Tower of Babel