RankCoT: Refining Knowledge for Retrieval-Augmented Generation through Ranking Chain-of-Thoughts
Mingyan Wu, Zhenghao Liu, Yukun Yan, Xinze Li, Shi Yu, Zheni Zeng, Yu Gu, Ge Yu · 2025
Read RankCoT: Refining Knowledge for Retrieval-Augmented Generation through Ranking Chain-of-Thoughts by Mingyan Wu, Zhenghao Liu, Yukun Yan, Xinze Li, Shi Yu, Zheni Zeng, Yu Gu, Ge Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17888.txt.
Introduction
Mingyan Wu and co-authors sharpen how search-fed language models use what they fetch in this 2025 paper. Such models answer with help from retrieved documents, but stray or noisy pages can mislead them. Their method, RankCoT, folds ranking signals into a written reasoning summary of the sources.
In training, the model drafts reasoning chains for each fetched page, then learns to rebuild the best chain while seeing all pages at once. A self-check step polishes those chains into cleaner training data. Tests show the method beats rival refinement schemes and yields shorter yet useful summaries. Answers downstream grow more accurate, and the code is public.
Catalog record
| CREATOR | Mingyan Wu, Zhenghao Liu, Yukun Yan, Xinze Li, Shi Yu, Zheni Zeng, Yu Gu, Ge Yu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-17888
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RankCoT: Refining Knowledge for Retrieval-Augmented Generation through Ranking Chain-of-Thoughts Mingyan Wu, Zhenghao Liu, Yukun Yan, Xinze Li, Shi Yu, Zheni Zeng, Yu Gu, Ge Yu Abstract Retrieval-Augmented Generation (RAG) enhances the performance of Large Language Models (LLMs) by incorporating external knowledge. However, LLMs still encounter challenges in effectively utilizing the knowledge from retrieved documents, often being misled by irrelevant or noisy information. To address this issue, we introduce RankCoT, a knowledge refinement method that incorporates reranking signals in generating CoT-based summarization for knowledge refinement based on given query and all retrieval documents. During training, RankCoT prompts the LLM to generate Chain-of-Thought (CoT) candidates based on the query and individual documents. It then fine-tunes the LLM to directly reproduce the best CoT from these candidate outputs based on all retrieved documents, which requires LLM to filter out irrelevant documents during generating CoT-style summarization. Additionally, RankCoT incorporates a self-reflection mechanism that further refines the CoT outputs, resulting in higher-quality training data. Our experiments demonstrate the effectiveness of RankCoT, showing its superior performance over other knowledge refinement models. Further analysis reveals that RankCoT can provide shorter but effective refinement results, enabling the generator to produce more accurate answers. All code and data are available at https://github.com/NEUIR/RankCoT . 1 Introduction Retrieval-Augmented Generation (RAG) Lewis et al. (2020b) ; Guu et al. (2020) ; Ram et al. (2023) ; Shi et al. (2024) empowers Large Language Models (LLMs) to access external knowledge, providing up-to-date information during the generation process. RAG models have demonstrated their effectiveness in mitigating the hallucination problem commonly encountered by LLMs Shuster et al. (2021) , enhancing the performance of LLMs, such as GPT-4 Achiam et al. (2023) and LLaMA Touvron et al. (2023) , in different NLP tasks. Figure 1: Illustration of RankCoT. We present how knowledge refinement can be achieved by incorporating reranking into CoT-based summarization. RAG models typically use dense retrieval methods Karpukhin et al. (2020) ; Xiong et al. (2021) to retrieve query-relevant documents from external knowledge bases. These documents, along with the query, are then fed as the input context into LLMs Ram et al. (2023) . Thriving on their in-context learning capabilities Brown et al. (2020) ; Dong et al. (2023) , LLMs are able to extract relevant semantics from retrieved documents and generate appropriate responses to address the given query. However, the potential knowledge conflict between external knowledge and parameterized memory still poses a challenge for LLMs in generating precise responses Chen et al. (2024b) ; Asai et al. (2024b) . Many RAG models focus on building modular RAG pipelines to enhance retrieval performance Gao et al. (2024) ; Asai et al. (2024a) . These models primarily aim to refine the retrieved knowledge by assessing the relevance of each document to the query and subsequently filtering out irrelevant ones Yan et al. (2024) ; Asai et al. (2024a) . However, the reranking model still requires feeding the remaining documents into LLMs, which means that query-unrelated content within a relevant document may still mislead the generators Xu et al. (2024a) . Some models address this problem by prompting LLMs to summarize query-relevant knowledge from the retrieved documents, thereby reducing the influence of irrelevant information Vig et al. (2022) ; Yu et al. (2024a) ; Xu et al. (2024a) . This summarization approach often incorporates information from unrelated documents as part of the summaries, resulting in the introduction of noise. Both the reranking and summarization modules have advantages for knowledge refinement. However, in existing RAG systems, these modules are typically modeled separately by prompting the same LLMs. This paper presents RankCoT, a knowledge refinement method that combines the strengths of both ranking and summarization to effectively enhance the process for retrieval result refinement. As shown in Figure 1 , we feed both the query and all retrieved documents into the RankCoT model, which incorporates reranking signals in generating CoT-based summarization as knowledge refinements, thereby aiding LLMs in generating more accurate responses for answering the given query. During training RankCoT, we independently feed the query and retrieved document to the LLM, asking it to generate several Chain-of-Thought (CoT) responses to answer the question, which can be considered as summarization results. We then design the self-refinement model to prompt LLMs to answer the question according to these sampled CoTs, helping to refine the CoT results for more effective training. If the refined CoT contains the ground truth answer, it is considered a positive refinement result, while those that do not contain the ground truth answer are considered negative refinements. Our experiments demonstrate that RankCoT outperforms all baseline models, achieving over a 2% improvement. Notably, RankCoT proves effective across LLMs of various scales. It generates shorter knowledge refinement results compared to both reranking and summarization methods, while enhancing the response accuracy of generator. Further analysis reveals that RankCoT successfully incorporates ground truth answers into the knowledge refinement results, while also including more query-relevant content. Additionally, RankCoT effectively extracts crucial semantics from the retrieved documents and alleviates the conflict between retrieved contents and internal knowledge. 2 Related Work Retrieval-Augmented Generation (RAG) aims to enhance Large Language Models (LLMs) by enabling them to access external knowledge bases, providing up-to-date information during the generation process Shi et al. (2024) ; Ram et al. (2023) . This approach has demonstrated promising results across various NLP tasks, including open-domain question answering Izacard et al. (2023) , code generation Zhou et al. (2023) , and dialogue Shuster et al. (2022) . In these RAG models, retrieved documents are typically used as context to assist LLMs in generating more accurate responses Ram et al. (2023) . However, the conflict between the external knowledge and the parametric memory of LLMs often undermines the effectiveness of current RAG systems Asai et al. (2024b) ; Xie et al. (2024) ; Chen et al. (2024b) . To mitigate the potentially negative impact of retrieved knowledge, existing models focus on refining the external knowledge through various modules designed to help LLMs generate more precise responses. Earlier works concentrate on reranking the retrieved documents Yu et al. (2023) ; Shi et al. (2024) ; Yu et al. (2024b) , while others employ query-focused summarization techniques Vig et al. (2022) ; Xu et al. (2023) to reduce noise. However, reranking models often overlook noise within individual passages, and summarization models may fail to account for query-document relevance, sometimes incorporating misleading content in the summarization results. Chain-of-Note Yu et al. (2024a) attempts to instruct LLMs to generate query-related notes when answering a given query. This model incorporates the knowledge refinement process into the reasoning stage Wei et al. (2022) and heavily relies on the capabilities of LLMs, which may limit its applicability in RAG systems Gao et al. (2024) . Modular RAG systems Gao et al. (2024) ; Xu et al. (2024c) focus on refining external knowledge through different modules implemented by LLMs, which have become a key trend in the RAG area. For instance, Self-RAG Asai et al. (2024a) uses different tags for adaptive retrieval Jiang et al. (2023) and self-reflection to refine knowledge. Some approaches also focus on reformulating queries to identify more useful documents for answering questions Yan et al. (2024) ; Trivedi et al. (2023) . Yan et al. (2024) introduce