Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems Conference: Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining; March 10–14, 2025; Hannover, Germany Proceedings of the Eighteenth ACM Intern

Rishiraj Saha Roy, Joel Schlotthauer, Chris Hinze, Andreas Foltyn, Luzian Hahn, Fabian Kuech · 2024

Read Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems Conference: Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining; March 10–14, 2025; Hannover, Germany Proceedings of the Eighteenth ACM Intern by Rishiraj Saha Roy, Joel Schlotthauer, Chris Hinze, Andreas Foltyn, Luzian Hahn, Fabian Kuech (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10571.txt.

Introduction

Rishiraj Saha Roy and co-authors upgrade a question-answering system for company wikis. Their 2024 paper works on retrieval-augmented generation, where a model fetches pages and answers from them. Two flaws dog such systems: fetched text lacks context, and credit for each answer is guessed, not proven.

Their system, RAGONITE, wraps each fetched passage with its source data and nearby text. Credit is then tested by removal: if dropping a passage changes the answer, that passage truly counted. A new benchmark of 300 hand-made chat questions in English and German tests the ideas. Context made answers better, and the removal test beat standard credit methods.

Catalog record

CREATORRishiraj Saha Roy, Joel Schlotthauer, Chris Hinze, Andreas Foltyn, Luzian Hahn, Fabian Kuech
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-10571

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems Conference: Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining; March 10–14, 2025; Hannover, Germany Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining (WSDM ’25), March 10–14, 2025, Hannover, Germany DOI: 10.1145/3701551.3704126 ISBN: 979-8-4007-1329-3/25/03 CCS: Information systems Question answering Rishiraj Saha Roy, Joel Schlotthauer, Chris Hinze, Andreas Foltyn, Luzian Hahn, Fabian Kuech Abstract . Retrieval Augmented Generation (RAG) works as a backbone for interacting with an enterprise’s own data via Conversational Question Answering (ConvQA). In a RAG system, a retriever fetches passages from a collection in response to a question, which are then included in the prompt of a large language model (LLM) for generating a natural language (NL) answer. However, several RAG systems today suffer from two shortcomings: (i) retrieved passages usually contain their raw text and lack appropriate document context, negatively impacting both retrieval and answering quality; and (ii) attribution strategies that explain answer generation typically rely only on similarity between the answer and the retrieved passages, thereby only generating plausible but not causal explanations. In this work, we demonstrate RAGonite , a RAG system that remedies the above concerns by: (i) contextualizing evidence with source metadata and surrounding text; and (ii) computing counterfactual attribution, a causal explanation approach where the contribution of an evidence to an answer is determined by the similarity of the original response to the answer obtained by removing that evidence. To evaluate our proposals, we release a new benchmark ConfQuestions : it has 300 300 hand-created conversational questions, each in English and German, coupled with ground truth URLs, completed questions, and answers from 215 215 public Confluence pages. These documents are typical of enterprise wiki spaces with heterogeneous elements. Experiments with RAGonite on ConfQuestions show the viability of our ideas: contextualization improves RAG performance, and counterfactual explanations outperform standard attribution. 1. Introduction Motivation . “Talk to your data” is a major research theme today, where users interact with local knowledge repositories to satisfy their information needs. Conversational question answering (ConvQA) ( Reddy et al. 2019 ; Christmann et al. 2019 ) is a natural choice for such interactions, where a user starts with a self-sufficient ( intent-explicit ) question, and follows that up with more ad hoc, conversational questions that leave parts of the context unspecified ( intent-implicit ). Hand in hand, the emergence of powerful large language models (LLMs) has led to retrieval augmented generation (RAG) ( Lewis et al. 2020 ) as the backbone for designing QA systems over one’s own data. In RAG pipelines, given a question , a retriever fetches relevant evidence from local data, and this is passed on to an LLM for a concise and fluent answer to the user’s question. Enterprise knowledge repositories usually consist of documents with heterogeneous elements, i.e. they often contain interleaved tables, lists and passages (henceforth, we use the unifying term evidence ). Typical examples of such documents with mixed structured and unstructured elements are meeting notes, test reports, or product descriptions. The setting for this demo is ConvQA with RAG over such heterogeneous document collections ( Pan et al. 2022 ; Yu 2022 ) . Limitations of state-of-the-art . Over the last four years, RAG has been a topic of intense investigation ( Gao et al. 2023 ) . Beyond literature, organizations like LangChain, LlamaIndex, or Cohere offer frameworks to build RAG systems. We posit that despite several advanced features, these suffer from two basic concerns, one at each end of the pipeline: (i) at the beginning, documents are typically split into chunks (usually one or more passages) that are indexed on their content ( Gao et al. 2023 ) . When these chunks are retrieved and fed to an LLM, they often lack supporting context from the document, which adversely affects both retrieval, and subsequent answering; and (ii) at the end, attribution mechanisms that attach provenance likelihoods of the answer to the retrieved units of evidence, are solely based on statistical similarity between the answer and the evidence ( Li et al. 2023b ) : these are not causal ( Pearl 1997 ) , but rather only plausible explanations. Moreover, current pipelines support raw text and rarely mention how tabular elements could be handled. Figure 1 . The RAGonite workflow enhances RAG pipelines at both ends, preprocessing evidence and explaining answers. Contributions . We make the following salient contributions: \bullet We demonstrate RAGonite , a new RAG system that concatenates page titles, headings, and surrounding text to raw contents of evidences for better retrieval and answering; \bullet We compute counterfactual attribution distributions over retrieved evidences as causal explanations for answers; \bullet We bring tables under RAGonite ’s scope by linearizing each record (row) via verbalization ( Oguz et al. 2022 ) and similar techniques ( Li et al. 2023a ) ; \bullet We create ConfQuestions , a benchmark with 300 conversational questions for evaluating RAG-based heterogeneous QA. The benchmark and all other artifacts for this work are public at https://github.com/Fraunhofer-IIS/RAGonite . Figure 2 . An annotated walkthrough the of RAGonite demo. Blue boxes guide the reader and are not part of the UI (Sec. 3 ). 2. System overview Backend . An overview of the RAGonite pipeline is in Fig. 1 . RAGonite’s backend is split into a functional core, which handles retrieval and answer generation, and a stateful layer that persists chats into a SQLite database and provides a REST API to the frontend using FastAPI. The dependencies in the functional core include the vector database (ChromaDB), and prompt template (Jinja) and LLM libraries ( gpt-4o and Llama-3.1-8B ). We use ChromaDB as our vector database for storing the contextualized evidences, and also use ChromaDB’s in-built retrieval functions. The multilingual BGE embedding model 1 1 1 https://huggingface.co/BAAI/bge-m3 was used to embed evidences, that was found to work slightly better than text-embedding-3-small 2 2 2 https://openai.com/index/new-embedding-models-and-api-updates/ from OpenAI. While ChromaDB was used for its extensibility, we also explored variants like Weaviate and Milvus. We used top-k hybrid search with the BGE reranker 3 3 3 https://huggingface.co/BAAI/bge-reranker-v2-m3 and reciprocal rank fusion (RRF) ( Cormack et al. 2009 ) being used to merge k dense retrieval and lexical retrieval results. We use GPT-4o as our LLM of choice inside the question completion and answer generation modules for efficiency and quality, but we also support a Llama model in our demo. All prompts can be seen inside the demo interface for transparency. A single GPU server (4x48GB NVIDIA Ada 6000 RTX, 512 GB RAM, 64 virtual cores) was used for all our experiments. All code is in Python. Frontend . For the frontend we developed a single-page React application (actually Preact, as a more lightweight React-compatible alternative), intentionally avoiding additional dependencies to eliminate the need for a build process. All API calls in the RAGonite demonstration are handled by the frontend. 3. Demo walkthrough Figure 3 . Contextualized evidence as in answer prompt. We use a screenshot of the RAGonite main page in Fig. 2 , on which we overlay numbered blue boxes, for a demo walkthrough. First, a user must select a domain (0) on which to use RAGonite

Read the complete text