SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection

Kexian Tang, Jiani Wang, Shaowen Wang, Kaifeng Lyu · 2026

Read SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection by Kexian Tang, Jiani Wang, Shaowen Wang, Kaifeng Lyu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22213.txt.

Introduction

Tang and co-authors propose SPA, a plain baseline for injecting new knowledge into language models. Models still lack coverage in narrow, data-scarce fields. SPA uses a small set of hand-designed prompts to generate large synthetic training sets. In systematic comparisons, SPA beats several strong baselines. RL-based methods gain token efficiency at small scale but lose diversity as data grows. Multi-stage prompting wins only until simple prompts are tuned with care. Careful prompt design plus scale, they conclude, is hard to beat.

Catalog record

CREATORKexian Tang, Jiani Wang, Shaowen Wang, Kaifeng Lyu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75286
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.22213

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection Kexian Tang, Jiani Wang, Shaowen Wang, Kaifeng Lyu Abstract While large language models (LLMs) are pretrained on massive amounts of data, their knowledge coverage remains incomplete in specialized, data-scarce domains, motivating extensive efforts to study synthetic data generation for knowledge injection. We propose SPA ( S caling P rompt-engineered A ugmentation), a simple but tough-to-beat baseline that uses a small set of carefully designed prompts to generate large-scale synthetic data for knowledge injection. Through systematic comparisons, we find that SPA outperforms several strong baselines. Furthermore, we identify two key limitations of prior approaches: (1) while RL-based methods may improve the token efficiency of LLM-based data augmentation at small scale, they suffer from diversity collapse as data scales, leading to diminishing returns; and (2) while multi-stage prompting may outperform simple augmentation methods, their advantages can disappear after careful prompt tuning. Our results suggest that, for knowledge injection, careful prompt design combined with straightforward large-scale augmentation can be surprisingly effective, and we hope SPA can serve as a strong baseline for future studies in this area. Our code is available at https://github.com/Tangkexian/SPA . † † affiliationnotice: Equal contribution 1 Introduction Large language models (LLMs) can acquire broad world knowledge and general capabilities from massive web text, but their coverage of specialized domains is often incomplete. A common approach to address this issue is to inject domain knowledge into these models by further fine-tuning or continually pretraining them on domain-specific data that are not readily available on the public web ( 47 ; 52 ; 54 ) . However, domain-specific datasets are usually limited in both scale and diversity. In contrast to the high redundancy of web text, domain knowledge often lacks the repetitive variations required for effective learning. Compounding this challenge, LLMs are known to be data-inefficient, necessitating frequent and varied exposure to facts before they can be reliably internalized ( 24 ; 3 ; 1 ) . Consequently, directly fine-tuning LLMs on such sparse data often leads to overfitting on specific surface forms rather than robust knowledge acquisition, causing the model to fail when encountering unseen queries or variations ( 6 ; 46 ) . To mitigate this scarcity, an increasingly popular paradigm is to expand the domain-specific data into a much larger synthetic corpus using an LLM-based data generator, and then train the target model on this augmented dataset. Recent efforts to design such augmentation methods generally follow two distinct methodologies. RL-based approaches, such as SEAL ( 56 ) , employ reinforcement learning (RL) to train a dedicated generator, where the reward is defined as the performance of the target model trained on the augmented data. Another line of works, such as EntiGraph ( 53 ) and Active Reading ( 37 ) , relies on prompting pipelines consisting of multiple stages, where the data generator is prompted to rewrite the original corpus into a high-quality synthetic corpus through a sequence of intermediate transformations. Figure 1 : Overview of Scaling Prompt-engineered Augmentation (SPA). Our baseline method rewrites a small source corpus into a large synthetic corpus by repeatedly prompting a generator with a fixed set of seven human-curated prompt templates, which are designed based on three levels of learning strategies: Concept Learning, Critical Thinking, and Generative Learning. In this work, we aim to push the frontier of synthetic data generation for knowledge injection by proposing a simple but tough-to-beat baseline method: SPA ( S caling P rompt-engineered A ugmentation), which is a knowledge injection method that consistently outperforms existing methods at scale ( Figure 1 ). The key idea of SPA is very simple: 1. Prompt Engineering : We draw upon insights from cognitive science and educational psychology to design a set of 7 prompt templates based on effective human learning strategies, including concept learning , critical thinking , and generative learning . 2. Scaling : We repeatedly prompt an LLM to rewrite the source content using templates from our prompt set, progressively scaling the augmented corpus into a large-scale synthetic corpus. 3. Training : Finally, we train the target model on the synthetic corpus using the same experimental settings as in prior work. Perhaps surprisingly, we find that this simple baseline is already highly effective. We evaluate SPA on three representative benchmarks that span Wikipedia-based question answering (SQuAD, 42 ), long-document comprehension (QuALITY, 39 ), and multi-hop reasoning (MultiHop-RAG, 50 ). Across all these settings, we use almost the same experimental setup as in the previous works ( 56 ; 53 ) , and change only the data augmentation strategy. We observe that SPA consistently matches or outperforms a range of substantially more complex methods, including SEAL ( 56 ) , PaST ( 49 ) , EntiGraph ( 53 ) , SoG ( 30 ) , and Active Reading ( 37 ) , and the gains of SPA become more pronounced as we scale the synthetic corpus. Our contributions are: • We propose S caling P rompt-engineered A ugmentation (SPA), a simple knowledge injection baseline that uses a small set of prompt templates to generate large-scale synthetic corpora ( Section 3 ). • We conduct a series of strictly controlled, token-matched evaluations comparing SPA against many complex methods ( Section 4 ), and demonstrate that SPA consistently matches or outperforms these methods at scale ( Section 5 ). This suggests that SPA is a competitive baseline for knowledge injection despite its simplicity. • Our results provide insights into the limitations of existing methods: (1) RL-based methods suffer from diversity collapse as data scales, leading to diminishing returns; and (2) adding more stages in the prompting pipeline may not yield larger gains than improving the quality of the prompt set for one-stage methods. While SPA demonstrates strong effectiveness, we do not suggest that knowledge injection should just rely on SPA. Rather, we hope our results provide a solid foundation for future work to develop methods that genuinely improve upon competitive baselines. 2 Related Work Small Corpus Knowledge Injection. Knowledge injection for domain-specific data has a wide range of applications. For example, in domains such as medicine ( 40 ; 25 ; 29 ) , finance ( 4 ; 28 ; 27 ) , and law ( 11 ; 19 ) , it is often necessary to inject specialized knowledge into large language models to support professional tasks. However, domain-specific corpora are typically small and suffer from data scarcity. As a result, directly performing continued pretraining or fine-tuning on such corpora often exhibits limited effectiveness and poor scalability ( 18 ; 24 ; 3 ; 41 ; 48 ) . To address this issue, our method SPA aims to scale up small corpora by synthetic data generation based on the original data, following recent advances in this line of work ( 56 ; 38 ; 49 ; 32 ; 53 ; 37 ; 30 ) . Synthetic Data Generation. Prior work has explored various strategies for synthetic generation of pretraining data, including (1) prompting LLMs to rewrite documents in diverse styles such as children’s stories ( 14 ) , “Wikipedia-style” data, and “question–answer” formats ( 31 ) , and (2) directly prompting LLMs to generate synthetic corpora such as textbook-style data ( 20

Read the complete text