SiReRAG : Indexing Similar and Related Information for Multihop Reasoning

Nan Zhang, Prafulla Kumar Choubey, Alexander Fabbri, Gabriel Bernadett-Shapiro, Rui Zhang Prasenjit Mitra Caiming Xiong Chien-Sheng Wu · 2024

Read SiReRAG : Indexing Similar and Related Information for Multihop Reasoning by Nan Zhang, Prafulla Kumar Choubey, Alexander Fabbri, Gabriel Bernadett-Shapiro, Rui Zhang Prasenjit Mitra Caiming Xiong Chien-Sheng Wu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06206.txt.

Introduction

Nan Zhang and co-authors present SiReRAG, an indexing method for retrieval systems that answer questions needing several hops across documents. Most indexes organize text by either similarity of meaning or by relatedness through shared facts, but not both. Using only one view, they argue, leaves knowledge poorly joined for hard questions.

Their system builds two trees: a similarity tree from recursive summaries, and a relatedness tree that groups statements by shared entities and summarizes them in turn. Both are flattened into one retrieval pool. On three multihop question sets, the method beat leading indexing approaches, and it also boosted reranking methods by a larger margin.

Catalog record

CREATORNan Zhang, Prafulla Kumar Choubey, Alexander Fabbri, Gabriel Bernadett-Shapiro, Rui Zhang Prasenjit Mitra Caiming Xiong Chien-Sheng Wu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06206

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SiReRAG : Indexing Similar and Related Information for Multihop Reasoning Nan Zhang, Prafulla Kumar Choubey, Alexander Fabbri, Gabriel Bernadett-Shapiro, Rui Zhang Prasenjit Mitra Caiming Xiong Chien-Sheng Wu Abstract Indexing is an important step towards strong performance in retrieval-augmented generation (RAG) systems. However, existing methods organize data based on either semantic similarity (similarity) or related information (relatedness), but do not cover both perspectives comprehensively. Our analysis reveals that modeling only one perspective results in insufficient knowledge synthesis, leading to suboptimal performance on complex tasks requiring multihop reasoning. In this paper, we propose SiReRAG , a novel RAG indexing approach that explicitly considers both similar and related information. On the similarity side, we follow existing work and explore some variances to construct a similarity tree based on recursive summarization. On the relatedness side, SiReRAG extracts propositions and entities from texts, groups propositions via shared entities, and generates recursive summaries to construct a relatedness tree. We index and flatten both similarity and relatedness trees into a unified retrieval pool. Our experiments demonstrate that SiReRAG consistently outperforms state-of-the-art indexing methods on three multihop datasets (MuSiQue, 2WikiMultiHopQA, and HotpotQA), with an average 1.9% improvement in F1 scores. As a reasonably efficient solution, SiReRAG enhances existing reranking methods significantly, with up to 7.8% improvement in average F1 scores. Our code is available at https://github.com/SalesforceAIResearch/SiReRAG . 1 Introduction Retrieval-augmented generation (RAG) has shown strong potential in augmenting large language models (LLMs) with highly specialized and constantly updated knowledge ( Lewis et al. 2020 ; Gao et al. 2024 ) . Getting rid of fine-tuning LLMs, it is an efficient method for handling users’ queries that require domain knowledge. A typical RAG pipeline may involve chunking, embedding, indexing, retrieval with queries, reranking, and LLM response generation ( Wang et al. 2024 ) . The indexing step is a prerequisite. It focuses on organizing a large amount of data and serves as an upstream step of retrieval. For example, RAPTOR ( Sarthi et al. 2024 ) shows a significant performance improvement by adding recursive summaries to text chunks of a dataset, which demonstrates the potential of adding synthesized information for retrieval. The added recursive summaries combine semantically similar information within a dataset. GraphRAG ( Edge et al. 2024 ) , on the other hand, indices an entire corpus via an entity-guided knowledge graph. It then constructs summaries from closely-related entities and their mentions, synthesizing the connections and relatedness among different pieces of information. However, none of the existing methods address the importance of indexing from both similarity and relatedness sides, which limits a holistic understanding of the provided dataset. We define similarity as the semantic distance of text pieces and relatedness as the degree of connection of texts based on signals such as entities and propositions. Indexing similar and related information facilitates more comprehensive knowledge integration than indexing individual kind of information. As shown in Figure 1 , a complex question that involves two hops of reasoning requires the retrieval and synthesis of relevant entity chunks. For example, synthesizing entity 1 and 2 chunks would encourage LLMs to generate “Francis Bacon”, which is a common mistake. Entity 1 and 2 chunks have close semantic distance. On the other hand, synthesizing entity 2 and 3 chunks would not maximize the probability of the correct answer even when entity 1 chunk is retrieved, since LLMs may struggle to reason about the painter of “Head I” in long-context environment ( Liu et al. 2024 ) . In this case, entity 2 and 3 chunks are related due to a shared topic. Therefore, it is important to synthesize both similar and related information in order to maximize the chance of retrieving relevant knowledge, in particular, for multihop reasoning questions. We verify the bottleneck of solely modeling similarity or relatedness through quantitative methods in Section 3 and demonstrate that neither perspective yields the optimal performance. In this paper, we propose SiReRAG , which stands for RAG indexing of si milarity and re latedness as shown in Figure 2 . On the similarity side, SiReRAG follows RAPTOR ( Sarthi et al. 2024 ) to build a recursive tree based on chunk similarity. We adopt a shallow tree with 4 levels in total. On the relatedness side, SiReRAG first extracts entities ( e.g. , “Sonnet 110” and “William Shakespeare”) and fine-grained propositions ( e.g. , “Sonnet 110 is one of 154 sonnets written by William Shakespeare.”) from each text chunk/document using LLMs. We group these propositions into aggregated ones via entities, simply concatenating them with the original order in chunk/document. These proposition aggregates contain related information, because they mention shared entities. Then, recursive summaries with soft clustering are built on top of those aggregated propositions. Finally, we index both trees by flattening nodes in each tree for retrieval. We show that SiReRAG is effective on a variety of multihop question answering (QA) datasets including MuSiQue ( Trivedi et al. 2022 ) , 2WikiMultiHopQA ( Ho et al. 2020 ) , and HotpotQA ( Yang et al. 2018 ) . We find SiReRAG consistently outperforms the strongest RAG indexing methods, achieving an average F1 improvement of at least 1.9%. We conduct an ablation study on several components in SiReRAG and observe that adding proposition aggregates to the similar information within a dataset yields the most improvement, which echoes our motivation. SiReRAG is also a reasonably efficient model, as it does not introduce many lengthy or redundant retrieval candidates. Figure 1: Challenges of existing RAG indexing methods for multihop reasoning. Entity 1 and 2 chunks contain similar information while entity 2 and 3 chunks contain related contents. Since synthesizing information only based on entity 1 and 2 (or entity 2 and 3) will lead to a higher probability of a wrong answer, an indexing method that considers both similarity and relatedness is needed to maximize retrieving relevant knowledge for multihop questions. 2 Related Work RAG RAG is a framework that integrates retrieval mechanisms into generative models to enhance text generation by leveraging external knowledge. This concept has evolved from earlier retrieval-based methods such as DrQA ( Chen et al. 2017 ) and DPR ( Karpukhin et al. 2020 ) . Instead of separating retrieval and generation phases, researchers also showed the potential of tightly coupling retrieval and generation into an end-to-end framework ( Lewis et al. 2020 ) . Recent advances in retrieval mechanisms include leveraging LLMs as retrievers ( Yu et al. 2023 ; Sun et al. 2023 ) and exploring retrieval granularity such as proposition ( Chen et al. 2023 ) . Here each proposition is an atomic expression that contains a factoid presented in natural language, which is similar to the propositions used by SiReRAG . Inspired by the idea of retrieval granularity, we combine several different text granularities (entity, propositions, text chunk, and summary) to index data. As a representative work on text segmentation, researchers proposed the frst supervised approach to generate hierarchical segmentation structures ( Nair et al. 2023 ) . RAG Indexing An earlier work ( Lewis et al. 2020 ) shows the benefits of document index on the overall retrieval performance. Most recent works on RAG indexing include RAPTOR that builds a tree with recursive summaries ( Sarthi et al. 2024

Read the complete text