Domain-specific Question Answering with Hybrid Search
Dewang Sultania, Zhaoyu Lu, Twisha Naik, Franck Dernoncourt · 2024
Read Domain-specific Question Answering with Hybrid Search by Dewang Sultania, Zhaoyu Lu, Twisha Naik, Franck Dernoncourt (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03736.txt.
Introduction
Dewang Sultania, Zhaoyu Lu, and Twisha Naik are the first names on this paper from two thousand twenty-four, whose text lists a long author roster. It shows that a hybrid search beats a lone retriever at domain-specific question answering for firms.
The system blends a fine-tuned dense retriever with keyword search. A weighted mix joins three signals: dense match scores, BM25 keyword scores, and a boost when the source host matches. Each weight can be tuned. Tests show the blend tops the single-retriever build on accuracy while keeping answers tied to source text. The authors say the mix handles the quirks of niche domains well.
Catalog record
| CREATOR | Dewang Sultania, Zhaoyu Lu, Twisha Naik, Franck Dernoncourt |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03736
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Domain-specific Question Answering with Hybrid Search Dewang Sultania, Zhaoyu Lu, Twisha Naik, Franck Dernoncourt Abstract Domain-specific question answering is an evolving field that requires specialized solutions to address unique challenges. In this paper, we show that a hybrid approach—combining a fine-tuned dense retriever with keyword-based sparse search methods—significantly enhances performance. Our system leverages a linear combination of relevance signals, including cosine similarity from dense retrieval, BM25 scores, and URL host matching, each with tunable boost parameters. Experimental results indicate that this hybrid method outperforms our single-retriever system, achieving improved accuracy while maintaining robust contextual grounding. These findings suggest that integrating multiple retrieval methodologies with weighted scoring effectively addresses the complexities of domain-specific question answering in enterprise settings. 1 Introduction With the increasing adoption of Large Language Models (LLMs) in enterprise settings, ensuring accurate and reliable question-answering systems remains a critical challenge. Building upon our previous work on domain-specific question answering about Adobe products ( Sharma et al. 2024 ) , which established a retrieval-aware framework with self-supervised training, we now present a production-ready, generalizable architecture alongside a comprehensive evaluation methodology. Our core contribution is a flexible, scalable framework built on Elasticsearch that can be adapted for any LLM-based question-answering system. This framework seamlessly integrates hybrid retrieval mechanisms, combining dense and sparse search with boost matching, while maintaining production-grade performance requirements. While we demonstrate its effectiveness using our organization’s domain-specific data, the architecture is designed to be domain-agnostic and can be readily deployed for other enterprise applications. We evaluate our system through a rigorous methodology that assesses performance across multiple dimensions: context relevance through normalized Discounted Cumulative Gain (nDCG), response groundedness, answer accuracy, and answer null rate. To ensure robust testing, we compile a diverse evaluation dataset including: 1. Human-annotated responses for common feature queries 2. A carefully curated “negative” dataset containing jailbreak attempts, NSFW content, and irrelevant queries to test system boundaries 3. LLM-based comparative analysis between system outputs and human-annotated ground truth This comprehensive evaluation approach allows us to assess not only the system’s ability to provide accurate information but also its robustness against inappropriate queries and its capability to acknowledge when information is not available. Our hybrid search mechanism demonstrates significant improvements across these metrics compared to single-method approaches. The key contributions of this work include: • A production-ready, generalizable framework for LLM-based QA systems built on Elasticsearch • A flexible hybrid retrieval mechanism combining dense and sparse search methods • A comprehensive evaluation framework for assessing QA system performance • Empirical analysis demonstrating the effectiveness of our approach across various metrics Through this work, we provide not only theoretical insights but also a practical, deployable solution for building reliable domain-specific question-answering systems that can be adapted to various enterprise needs. 2 Related Work Our research builds upon recent advancements in retrieval-augmented language models, domain-specific question answering (QA), and hybrid retrieval methods. We review relevant literature across these key areas. Retrieval-Augmented Language Models Retrieval-Augmented Generation (RAG) models integrate information retrieval with language models to enhance performance on knowledge-intensive tasks such as question answering. These models employ a retriever to identify pertinent documents, conditioning the language model on the retrieved context during response generation. Lewis et al. 2020 introduced the RAG framework, demonstrating its effectiveness in open-domain QA by combining dense passage retrieval with sequence-to-sequence generation. Further, Lazaridou, Cancedda, and Baroni 2022 explored the application of RAG in an enterprise customer support setting, highlighting its potential for domain-specific QA. Domain-Specific Question Answering Developing QA systems tailored to specialized domains necessitates techniques that effectively capture domain-specific knowledge and terminology. Sharma et al. 2024 proposed a framework for compiling domain-specific QA datasets and introduced retrieval-aware fine-tuning of language models, which reduces hallucinations and enhances contextual grounding. Eppalapally et al. 2024 introduced a model for query rewriting to take the domain of interest into account. Additionally, hybrid retrieval methods that combine dense and sparse techniques have shown promise in domain-specific QA. For instance, Zhu et al. 2023 presented a hybrid text generation-based query expansion method, improving retrieval accuracy and QA system performance. Hybrid Retrieval Methods Integrating multiple retrieval signals can improve ranking performance compared to single-method approaches. Arivazhagan et al. 2023 explored hybrid hierarchical retrieval, combining sparse and dense methods in a two-stage document and passage retrieval setup, demonstrating improved in-domain and zero-shot generalization. Similarly, Li et al. 2021 proposed a dual reader-parser architecture that leverages both textual and tabular evidence, enhancing open-domain QA performance. While prior work has made significant strides, developing reliable and robust domain-specific QA systems for enterprise settings remains an open challenge. Our work addresses this by proposing a flexible, generalizable framework that integrates hybrid retrieval with weighted relevance scoring. We extend prior hybrid approaches with a multi-phase scoring algorithm and introduce a comprehensive evaluation methodology to assess system performance across multiple dimensions. 3 Methodology To optimize document retrieval accuracy, we employed a multi-phase scoring algorithm that evolved through iterations, starting from a baseline BM25 model, progressing to a chunking-based approach, and finally integrating host-based score adjustments. Figure 1: Production ready RAG deployment Scoring Algorithm Given a user query, we calculate relevance scores for each document as follows: \displaystyle\text{score}= \displaystyle\ \max(\text{matched chunks cosine score}) \displaystyle+\text{bm25\_boost}\times\text{BM25 score} \displaystyle+\text{host\_boost}\times\text{host\_score} (1) Here, the final score combines the highest cosine similarity score from matched content chunks within each document, boosted BM25 scores, and a host-based weighting. Each document’s chunked cosine score ensures finer-grained relevance, while BM25 and host boosts adjust for term frequency relevance and content source authority, respectively. This combination provides a more nuanced ranking by integrating multiple retrieval signals. Parameter Selection The values for bm25_boost and host_boost were empirically determined by optimizing for the highest average similarity score between generated outputs and a predefined golden set. This iterative tuning process ensured that the scoring weights effectively balanced relevance and content source reliability. Table 2 and Table- 3 summarize how we select these parameters. Document Ranking After calculating scores for all documents, they were sorted in descending order, and the top 3 documents were returned as the final output, prioritizing the documents most likely to yield relevant information. 4 Experiments and Evaluation Datasets We evaluate our system using two carefully curated datasets: a golden dataset for measuring accuracy and performance, and a negative dataset for testing system robustness. Golden Dataset Our golden dataset comprises question-answer pairs from three primary documentation sources for our organization, with the following distribution: • creativecloud.adobe.com (Express Learn): 62 pairs • www.adobe.com: 74 pairs • helpx.adobe.com: 51 pairs Each entry in the dataset contains: • A help query, initially LLM-generated and human-revised • Relevant document URLs (separated by delimiters) • Step-by-step answers, LLM-generated and human-revised • Source documentation URL Negative Dataset To evaluate system robustness, we compiled a negative dataset consisting of: • 12 jailbreak attempts • 6 NSFW queries • 12 irrelevant queries All queries were collected from internet sources and revised by human annotators, with an expected ”content not found” response. Evaluation Setup Our evaluation framework uses the following common parameters across all experiments: • Generator Model: GPT-4o • Retrieved URLs Limit: 3 Figure 2: Evaluation Setup Results and Analysis Effect of Chunk Size on Retrieval Performance To determine the optimal chunk size for document segmentation, we experimented with various chunk sizes and overlap values.