Knowledge-Instruct: Effective Continual Pre-training from Limited Data using Instructions
Oded Ovadia, Meni Brief, Rachel Lemberg, Eitam Sheetrit · 2025
Read Knowledge-Instruct: Effective Continual Pre-training from Limited Data using Instructions by Oded Ovadia, Meni Brief, Rachel Lemberg, Eitam Sheetrit (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05571.txt.
Introduction
Oded Ovadia, Meni Brief, Rachel Lemberg, and Eitam Sheetrit present a 2025 method for teaching language models new facts from scant text. Extra training on thin data often wrecks old skills, the feared forgetting. Knowledge-Instruct skips raw text: small models first turn the source into dense question-and-answer style lessons. The big model then trains on those lessons alone, as pure instruction work.
Tests show strong fact uptake with far less loss of general skill. Multi-step reasoning over the new facts also improved, and the lessons plug into lookup systems well. A fresh company data set, released with the paper, gauges injection skill. Thin sources, from manuals to niche files, become teachable at last.
Catalog record
| CREATOR | Oded Ovadia, Meni Brief, Rachel Lemberg, Eitam Sheetrit |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-05571
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Knowledge-Instruct: Effective Continual Pre-training from Limited Data using Instructions Oded Ovadia, Meni Brief, Rachel Lemberg, Eitam Sheetrit Abstract While Large Language Models (LLMs) acquire vast knowledge during pre-training, they often lack domain-specific, new, or niche information. Continual pre-training (CPT) attempts to address this gap but suffers from catastrophic forgetting and inefficiencies in low-data regimes. We introduce Knowledge-Instruct , a novel approach to efficiently inject knowledge from limited corpora through pure instruction-tuning. By generating information-dense synthetic instruction data, it effectively integrates new knowledge while preserving general reasoning and instruction-following abilities. Knowledge-Instruct demonstrates superior factual memorization, minimizes catastrophic forgetting, and remains scalable by leveraging synthetic data from relatively small language models. Additionally, it enhances contextual understanding, including complex multi-hop reasoning, facilitating integration with retrieval systems. We validate its effectiveness across diverse benchmarks, including Companies , a new dataset that we release to measure knowledge injection capabilities. Figure 1: Visualization of the Knowledge-Instruct framework. A small text corpus is transformed into a set of information-dense instructions following the steps outlined in Section 3.2 . 1 Introduction Large language models (LLMs) encapsulate extensive knowledge within their pre-trained weights ( Petroni et al., 2019 ; Cohen et al., 2023 ; Hu et al., 2023 ; Chang et al., 2023 ) . This capacity is evidenced by the exceptional performance of modern LLMs in knowledge-intensive tasks ( Achiam et al., 2023 ; Dubey et al., 2024 ; Anthropic, 2024 ; Team et al., 2023 ; Liu et al., 2024 ) . Such knowledge is predominantly acquired during the pre-training phase ( Zhou et al., 2024 ) , which relies on extensive datasets containing trillions of tokens sourced from a wide array of domains, with a significant proportion derived from the web ( Conneau, 2019 ; Gao et al., 2020 ; Weber et al., 2024 ; Touvron et al., 2023 ) . Many of these datasets, being representative of available textual content, include substantial amounts of duplicated material ( Lee et al., 2021 ; Penedo et al., 2024 ) . Consequently, during the pre-training process, LLMs are trained on an enormous volume of factual information shaped by the underlying distribution of the training data. The inherent repetition of knowledge within these datasets can significantly enhance LLM knowledge acquisition, as LLMs benefit from encountering different formulations of the same fact to thoroughly internalize and utilize it ( Allen-Zhu and Li, 2023 ) . One example of this phenomenon is the so-called reversal curse ( Berglund et al., 2023 ) , where LLMs trained on statements like "A is B" struggle to correctly learn the inverse relationship, "B is A." The abundance of related content within training datasets can help mitigate such issues, facilitating more robust knowledge acquisition. One limitation of this heavy reliance on general text corpora is that many less common pieces of factual information are significantly underrepresented in many training datasets, making them less likely to be effectively learned by the model ( Kandpal et al., 2023 ) . Examples include domain-specific knowledge, geographically localized content, or other niche information. Moreover, certain data may be entirely absent from the training corpus, such as events that occurred after the model’s knowledge cutoff or proprietary datasets unavailable during pre-training. So, teaching LLMs niche knowledge is challenging due to the limited variations in fact representations within data-constrained corpora. This highlights a fundamental limitation in many practical scenarios. For instance, a manual, handbook, or textbook might comprehensively cover a topic using only \approx 100 K tokens with minimal repetition. While this approach is highly effective for human learners, it poses significant challenges for LLMs, which rely on diverse and repeated formulations to effectively internalize information. Knowledge-Instruct: In this work, we introduce Knowledge-Instruct , a novel approach for teaching pre-trained LLMs new knowledge from small corpora using synthetic instruction data. This method enables efficient knowledge acquisition by directly addressing the challenges outlined earlier. Knowledge-Instruct offers several key advantages over existing alternatives: 1. Superior Factual Memorization: Effectively learns factual information, outperforming other CPT methods. 2. Compatibility with Instruct Models: Facilitates continual pre-training directly on instruction-tuned models, reducing reliance on unsupervised training that can disrupt chat templates and is often restricted for API-based models. 3. Minimizes Catastrophic Forgetting: Exhibits minimal degradation in other model capabilities while integrating new knowledge. 4. Cost-Efficient: Requires only a relatively small language model to generate synthetic training data. 5. Enhanced Context Understanding: Strengthens the model’s ability to interpret and reason over retrieved context more effectively, making retrieval-augmented systems more accurate and reliable, even in challenging multi-hop scenarios. To validate these advantages, we empirically evaluate Knowledge-Instruct across diverse datasets and models. Our empirical results confirm its effectiveness in efficiently integrating new knowledge while preserving model stability and conversational fluency. Notably, our method achieves strong performance even with limited data, making it particularly suited for long-tail knowledge and domain-specific applications. 2 Related Work Continual Pre-training Continual pre-training adapts pre-trained language models to specialized domains through additional training on domain-specific corpora. Prior work has demonstrated success in broad domains like medicine ( Chen et al., 2023 ; Wu et al., 2023 ) , code ( Rozière et al., 2024 ) , and mathematics ( Shao et al., 2024 ; Mitra et al., 2024 ) using massive datasets (10B–500B tokens). However, these methods fail to scale to small corpora (~1M tokens), where limited diversity and repetition hinder effective knowledge acquisition ( Kandpal et al., 2023 ) . This challenge stems from the inherent data inefficiency of language models, which require multiple contextual exposures to internalize facts ( Allen-Zhu and Li, 2023 ) . Recent approaches address this through synthetic data augmentation, generating paraphrased texts ( Maini et al., 2024 ; Ovadia et al., 2023 ) or structured entity-based expansions ( Yang et al., 2024 ) . While Yang et al. (2024) ’s method is highly effective for adding new knowledge, it demands a very large amount of tokens, and also requires an additional instruction-tuning phase to be useful in practice. Knowledge Injection Alternative methods for knowledge integration include knowledge editing , which updates specific model parameters to insert facts while preserving existing capabilities ( Mitchell et al., 2021 ; Meng et al., 2022 ) . While effective for atomic edits, these methods struggle to scale for broader knowledge. Retrieval-augmented generation (RAG) ( Lewis et al., 2020 ) bypasses parametric learning by dynamically accessing external documents, but depends on retrieval quality, and potentially demands many tokens per query. Synthetic data generation allows for creating diverse synthetic pre-training corpora via complex external prompting ( Li et al., 2023 ; Mukherjee et al., 2023 ) , combined with further training. Supervised fine-tuning (SFT), particularly instruction tuning ( Wang et al., 2022 ) , is highly effective at improving zero-shot and reasoning capabilities. However, SFT is primarily used to enhance task generalization rather than introducing new factual knowledge ( Mitra et al., 2023 ; Chia et al., 2023 ; Zhou et al., 2024 ) . 3 Knowledge-Instruct The core idea behind this methodology is to transform a corpus of raw textual data into an information-dense set of factual instructions, thereby facilitating a knowledge-focused SFT phase. 3.1 Rationale Our approach is guided by key observations and hypotheses that we see as fundamental to building knowledge injection framework