GenKI: Enhancing Open-Domain Question Answering with Knowledge Integration and Controllable Generation in Large Language Models DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; Appril 28–May 02, 2025; Sydney, SYD ISBN: 978-1-4503-XX

Tingjia Shen, Hao Wang, Chuan Qin, Ruijun Sun, Yang Song, Defu Lian, Hengshu Zhu, Enhong Chen · 2025

Read GenKI: Enhancing Open-Domain Question Answering with Knowledge Integration and Controllable Generation in Large Language Models DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; Appril 28–May 02, 2025; Sydney, SYD ISBN: 978-1-4503-XX by Tingjia Shen, Hao Wang, Chuan Qin, Ruijun Sun, Yang Song, Defu Lian, Hengshu Zhu, Enhong Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19660.txt.

Introduction

Tingjia Shen and co-authors present a framework that feeds outside knowledge into language models for open question answering, in this 2025 paper. Models must both find facts and shape answers to fit each task's format. Their system, GenKI, works on both needs at once.

A retriever first pulls matched passages from a knowledge base. A second model folds that knowledge into the tuning instructions. A final ensemble checks answers for coherence, fluency, and format fit. Tests on three public sets beat strong baselines. Deeper analysis tied how often facts were retrieved to how well the model recalled them.

Catalog record

CREATORTingjia Shen, Hao Wang, Chuan Qin, Ruijun Sun, Yang Song, Defu Lian, Hengshu Zhu, Enhong Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19660

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GenKI: Enhancing Open-Domain Question Answering with Knowledge Integration and Controllable Generation in Large Language Models DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; Appril 28–May 02, 2025; Sydney, SYD ISBN: 978-1-4503-XXXX-X/18/06 Tingjia Shen, Hao Wang, Chuan Qin, Ruijun Sun, Yang Song, Defu Lian, Hengshu Zhu, Enhong Chen Abstract . Open-domain question answering (OpenQA) represents a cornerstone in natural language processing (NLP), primarily focused on extracting answers from unstructured textual data. With the rapid advancements in Large Language Models (LLMs), LLM-based OpenQA methods have reaped the benefits of emergent understanding and answering capabilities enabled by massive parameters compared to traditional methods. However, most of these methods encounter two critical challenges: how to integrate knowledge into LLMs effectively and how to adaptively generate results with specific answer formats for various task situations. To address these challenges, we propose a novel framework named GenKI , which aims to improve the OpenQA performance by exploring K nowledge I ntegration and controllable Gen eration on LLMs simultaneously. Specifically, we first train a dense passage retrieval model to retrieve associated knowledge from a given knowledge base. Subsequently, we introduce a novel knowledge integration model that incorporates the retrieval knowledge into instructions during fine-tuning to intensify the model. Furthermore, to enable controllable generation in LLMs, we leverage a certain fine-tuned LLM and an ensemble based on text consistency incorporating all coherence, fluency, and answer format assurance. Finally, extensive experiments conducted on the TriviaQA, MSMARCO, and CMRC2018 datasets, featuring diverse answer formats, have demonstrated the effectiveness of GenKI with comparison of state-of-the-art baselines. Moreover, ablation studies have disclosed a linear relationship between the frequency of retrieved knowledge and the model’s ability to recall knowledge accurately against the ground truth. Tests focusing on the out-of-domain scenario and knowledge base independence scenario have further affirmed the robustness and controllable capability of GenKI. Our code of GenKI is available at https://github.com/USTC-StarTeam/GenKI . 1. Introduction Open-domain question answering (OpenQA) has garnered significant attention in the Natural Language Processing (NLP) community, as it offers enhanced user-friendliness recommendation ( wang2025generative ; guo2024scaling ; wu2024survey ; liu2023user ) and efficiency compared to conventional search engines. OpenQA seeks to respond to questions utilizing auxiliary knowledge sources ( zhu2021retrieving ) , necessitating models that possess both knowledge capability and comprehension abilities. Conventional methods usually entail a retriever-reader framework ( DBLP:conf/iclr/DasDZM19 ) . The evolution of retriever models has progressed from BM25 ( 10.1561/1500000019 ) and TF-IDF ( aizawa2003information ) to dense-vector-based ( wang2025universal ; wang2025mf ; wang2019mcne ; wang2021hypersorec ; wang2021decoupled ) approaches, such as DPR ( karpukhin-etal-2020-dense ) and SEAL ( DBLP:conf/nips/BevilacquaOLY0P22 ) . Concurrently, reader models have diversified, spanning from extractive readers like DPR-reader ( cui-emnlp2019-cmrc2018 ) to generative readers, such as FiD ( DBLP:journals/corr/abs-2007-01282 ) and RAG ( DBLP:conf/nips/LewisPPPKGKLYR020 ) , targeting span and free answering tasks respectively. Recently, the rapid development of Large Language Models (LLMs) ( gu2025rapid ; yin2024entropy ; shen2024exploring ; huang2024chemeval ) has motivated researchers to incorporate them into OpenQA, driven by the models’ advanced abilities in natural language reasoning. For instance, ICL ( li2023selfprompting ) contains to use certain prompts towards better integration of external knowledge with in-context learning, while Replug ( shi2023replug ) enhances an LLM with retrieved knowledge as prompts. However, when utilizing language models for OpenQA, two notable error phenomena persist, as depicted in Figure 1 . These errors include: (1) Knowledge Deficiency . When prompted to identify “ which nuclear plant’s detectors first alerted the world to the Chornobyl disaster ”, the LLM fabricated an incorrect response. This issue stems from the limited memorization capacity of LLMs. Specifically, LLMs not only face challenges in retaining less frequent knowledge ( kandpal2023large ) , but also tend to produce hallucinations ( DBLP:conf/emnlp/0001PCKW21 ) when the requisite knowledge is absent from their pretraining data. Existing methods primarily address this issue through pre-training ( DBLP:journals/corr/abs-2204-02311 ) or prompting ( ren2023investigating ) . Pre-training is too broad and lacks precision while prompting overlooks the model’s potent knowledge storage capabilities ( petroni-etal-2019-language ) . (2) Answer Format Alignment . Various OpenQA datasets demonstrate unique answer formats, exemplified by the three markedly different formats depicted in Figure 1 . However, even when tasked to produce answers in a specific format, the outputs of LLMs often diverge from these expected formats. This discrepancy is largely due to the distributional biases present in the pre-training corpora, as discussed in ( kandpal2023large ) . Presently, to tackle challenge (1), PaLM 540B ( DBLP:journals/corr/abs-2204-02311 ) enlarges the model scale to accommodate a greater wealth of knowledge, which, however, leads to unacceptable computational complexity. So the ideology of RAG ( gao2023retrieval ) integrating knowledge with prompts for efficiency has raised. After researchers recognized the significance of retrieving knowledge, they sought to enhance the quality of retrieved content. Typical examples are enabling large models to autonomously assess the quality of retrieved content for answering questions ( ren2023investigating ) , or iteratively conducting multi-step retrievals to progressively enhance the quality of retrieved content and align with the preferences of LLM ( asai2023self ) . Nevertheless, integrating knowledge through prompt-based approaches treats LLMs merely as instruction interpreters, disregarding their potent knowledge storage capabilities ( petroni-etal-2019-language ) . Consequently, this approach yields unstable and suboptimal results. Toward challenge (2), InstructGPT ( ouyang2022training ) and FLAN ( DBLP:conf/iclr/WeiBZGYLDDL22 ) adopt a pre-training and instruction tuning paradigm and attempt to encapsulate all instructions along with the language understanding capabilities equipped with knowledge at the same time. Nevertheless, this paradigm will require the storage of massive real-world data ( shen2024optimizing ; yin2024dataset ; zhang2025td3 ) , and the computational burden is quadratic with respect to the number of model parameters ( hoffmann2022training ) . To address this issue, CoF-CoT ( nguyen2023cof ) attempts to utilize a chain of instruction to control the output format. However, a distribution gap persists between the formatted output and knowledge. Consequently, the collaborative training procedure is prone to encountering contradictions ( DBLP:conf/acl/MallenAZDKH23 ) . Figure 1. An illustration of hallucination encountered when employing LLMs in OpenQA task, along with the variations in answer formats across different OpenQA datasets. In general, although the aforementioned generator enhancement methods have achieved remarkable progress in the OpenQA field, current pre-training or fine-tuning-based methods only focus on either full knowledge adaption or applying LLM on both reasoning and answer alignment missions, considering two different kinds of task in one stage of tuning. This leads to distribution misalignment and limitations in sufficient knowledge integration and controllable generation performance in different task situations. Towards these challenges, in this paper, we propose a novel framework named GenKI , to improve the OpenQA performance by exploring K nowledge I ntegration and controllable Gen eration on LLMs. We introduced a novel three-stage paradigm for retrieval, knowledge integration, and controllable generation instead of the two-stage paradigm in RAG, ensuring that the model focuses on either knowledge integration or controlled generation in each instance, to avoid the distributional differences brought by these two tasks. Specifically, in the knowledge integration model, rather than adopting on normal fine-tuning-based method, we propose an innovative method of LLM combining autoregressive training loss and supervised fine-tuning loss to involve the retrieval results. In comparison to the prior research

Read the complete text