Select to Know: An Internal-External Knowledge Self-Selection Framework for Domain-Specific Question Answering

Bolei He, Xinran He, Run Shao, Shanfu Shu, Xianwei Xue, Mingquan Cheng, Haifeng Li, Zhen-Hua Ling · 2025

Read Select to Know: An Internal-External Knowledge Self-Selection Framework for Domain-Specific Question Answering by Bolei He, Xinran He, Run Shao, Shanfu Shu, Xianwei Xue, Mingquan Cheng, Haifeng Li, Zhen-Hua Ling (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15213.txt.

Introduction

Language models can be taught a specialty by letting them pick what they lack, in a 2025 paper by Bolei He and co-authors. General models stumble in fields like medicine, law, and finance, while fetching documents at question time adds noise and delay, and full retraining is costly.

Their framework, Select2Know, has the model judge its own gaps and pull only the knowledge worth internalizing, through selective fine-tuning. Learning is staged like human study: grasp the concepts first, then practice reasoning, with generated practice data and a reinforcement step to sharpen the second stage. On medical, legal, and financial tests, the method beat rivals and matched models pretrained on the whole domain, at far lower cost.

Catalog record

CREATORBolei He, Xinran He, Run Shao, Shanfu Shu, Xianwei Xue, Mingquan Cheng, Haifeng Li, Zhen-Hua Ling
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-15213

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Select to Know: An Internal-External Knowledge Self-Selection Framework for Domain-Specific Question Answering Bolei He, Xinran He, Run Shao, Shanfu Shu, Xianwei Xue, Mingquan Cheng, Haifeng Li, Zhen-Hua Ling Abstract Large Language Models (LLMs) perform well in general QA but often struggle in domain-specific scenarios. Retrieval-Augmented Generation (RAG) introduces external knowledge but suffers from hallucinations and latency due to noisy retrievals. Continued pretraining internalizes domain knowledge but is costly and lacks cross-domain flexibility. We attribute this challenge to the long-tail distribution of domain knowledge, which causes partially internalized yet useful knowledge to be underutilized. We further argue that knowledge acquisition should be progressive, mirroring human learning: first understanding concepts, then applying them to complex reasoning. To address this, we propose S elect 2 K now (S2K), a cost-effective framework that internalizes domain knowledge through an internal-external knowledge self-selection strategy and selective supervised fine-tuning. We also introduce a structured reasoning data generation pipeline and integrate GRPO to enhance reasoning ability. Experiments on medical, law, and financial QA benchmarks show that S2K consistently outperforms existing methods and matches domain-pretrained LLMs with significantly lower cost. 1 Introduction With the rapid advancement of large language models (LLMs), their effectiveness in general question answering has been widely validated Devlin et al. 2019 ; Brown et al. 2020 ; Lewis et al. 2020 ; Shailendra et al. 2024 . However, LLMs still exhibit noticeable performance gaps in domain-specific QA tasks Yang et al. 2023 ; Yue 2025 . To address these challenges, a variety of approaches have been explored to improve domain-specific QA (DSQA) performance. A common solution is the use of Retrieval-Augmented Generation (RAG) Lewis et al. 2020 ; Press et al. 2023 ; Asai et al. 2023 ; He et al. 2024 , where an retriever is used to access external knowledge from a domain corpus. While RAG helps incorporate up-to-date information, it introduces extra latency and computation due to redundant retrievals. Additionally, distribution mismatches may lead the retriever to return irrelevant or conflicting information, increasing the risk of hallucinations Rawte et al. 2023 ; Ji et al. 2023 ; Ye et al. 2023 ; Maynez et al. 2020 ; Xu et al. 2024 . Figure 1: Visualization of token-level prediction probabilities. Low-probability tokens indicate unmastered domain knowledge, while high-probability tokens reflect internal parametric knowledge. This highlights the need for integrating internal and external knowledge in domain adaptation. (Note: Schematic illustration; see Appendix A.1 for real examples.) Another line of research focuses on enhancing domain adaptation through continued pretraining Labrak et al. 2024 ; Qiu et al. 2024 ; Shu et al. 2024 ; Li et al. 2024 ; Chen et al. 2023 . These methods can achieve strong performance, but they are extremely resource-intensive and often lack transferability to other domains. (BioMistral Labrak et al. 2024 requires training on a corpus of three billion tokens.) We argue that the fundamental reason behind LLMs’ poor performance in DSQA lies in the long-tail distribution of domain knowledge in pretraining data. As illustrated in Figure 1 , LLMs have already internalized parts of domain knowledge during pretraining. While this knowledge is often incomplete, it can complement or even correct external domain inputs, making external-only methods suboptimal. Furthermore, we believe knowledge acquisition should follow a human-inspired staged progression—first achieving conceptual comprehension, then advancing to complex reasoning. Building on this insight, we propose a low-cost post-training framework, S elect 2 K now (S2K), for domain-specific question answering, which integrates both internal parametric knowledge and external domain knowledge. Specifically, we first introduce a token-level internal-external knowledge self-selection strategy to construct fusion training data. We then propose Selective Supervised Fine-Tuning (Selective SFT) to guide the model toward focusing on domain knowledge it has not yet mastered. In addition, we design a structured data generation pipeline to efficiently produce high-quality reasoning data, and incorporate Group Relative Policy Optimization (GRPO) Shao et al. 2024 to enhance the model’s ability to apply learned knowledge to real-world reasoning tasks. Our main contributions are as follows: • We propose a token-level knowledge self-selection strategy to fuse internal parametric knowledge and external domain knowledge. • We propose a low-cost post-training framework to boost LLM performance on DSQA. • Experiments across the medicine, law, and finance demonstrate that S2K matches pretrained LLMs with significantly lower training cost. 2 Problem Definition We aim to design a general pipeline that enables LLMs to efficiently generalize to domain-specific QA tasks with minimal cost. To closely reflect real-world scenarios, we make the following assumptions: (1) No existing QA training datasets are available in the target domain. (2) The only accessible resource is a collection of unstructured domain-specific corpus \mathcal{D}=\{d_{1},d_{2},...,d_{n}\} , such as news, textbooks, regulatory documents, etc. (3) A pre-trained general LLM \mathcal{M}_{0} (e.g., LLaMA Touvron et al. 2023 ; Grattafiori et al. 2024 , Qwen Yang et al. 2024a ; Yang et al. 2024b ) is used as the foundation. Our goal is to develop a pipeline \mathcal{P} such that the resulting domain-adapted model \mathcal{M}_{\mathcal{D}}=\mathcal{P}(\mathcal{M}_{0},\mathcal{D}) achieves strong performance on the domain QA task \mathcal{T}_{\mathcal{QA}} . Formally, we aim for \text{Perf}(\mathcal{M}_{\mathcal{D}},\mathcal{T}_{\mathcal{QA}})\gg\text{Perf}(\mathcal{M}_{0},\mathcal{T}_{\mathcal{QA}}) , where \text{Perf}(\cdot) denotes the evaluation performance on domain QA tasks. 3 Methods Figure 2: Overview of S2K, a low-cost post-training framework for domain-specific QA. The method comprises: data generation (1-X) and model training (2-X). In data generation, question-style meta knowledge is extracted from domain corpora, followed by token-level fusion of internal and external knowledge, and reasoning QA construction via relevance-based sampling and structured prompts. In model training, Selective SFT emphasizes unmastered knowledge using token-level uncertainty, while GRPO-based reinforcement learning enhances reasoning. We introduce S2K, a low-cost post-training framework for adapting general LLMs to domain-specific QA. As illustrated in Figure 2 , S2K first extracts question-style meta knowledge from raw domain corpora (Section 3.1.1 ). We then design a token-level self-selection mechanism to fuse internal and external knowledge (Section 3.1.2 ), complemented by Selective SFT, which guides the model to focus on unfamiliar domain knowledge (Section 3.2 ). We further introduce structured reasoning data generation pipeline (Section 3.1.3 ), and incorporate GRPO to enhance the model’s reasoning ability for complex real-world scenarios (Section 3.3 ). 3.1 Domain Knowledge Generation 3.1.1 Meta Knowledge As described in Section 2 , we construct domain QA data by first extracting question-style meta knowledge from raw domain corpora \mathcal{D} . Since such corpora are often redundant and unstructured, containing irrelevant details such as timestamps or publisher metadata, we first cleaning the data to remove non-informative content, then segment the corpus into token-balanced chunks using NLTK Bird 2006 to preserve semantic coherence. For each chunk d_{i}\in\mathcal{D} , we prompt a LLM (e.g., DeepSeek-v3 Liu et al. 2024 or GPT-4o Hurst et al. 2024 ) to generate a knowledge question. Formally, the question-style meta knowledge is defined as: {\mathcal{Q}_{i}=f_{\text{prompt}}(\mathcal{L},d_{i})} (1) where \mathcal{L} denotes the LLM used for prompting, f_{\text{prompt}} is the prompting process, and \mathcal{Q}_{i} is the meta question. Detailed prompts are provided in Appendix A.5 . 3.1.2 Internal-External Fusion Knowledge An intuitive approach to domain knowledge training is using answers generated from question-style meta knowledge and their corresponding text chunks. However, these answers rely only on external documents, which may introduce noise and ignore the model’s internal knowledge. To address this, we propose a token-level internal-external knowledge

Read the complete text