ComRAG: Retrieval-Augmented Generation with Dynamic Vector Stores for Real-time Community Question Answering in Industry

Qinwen Chen, Wenbiao Tao, Zhiwei Zhu, Mingfan Xi, Liangzhong Guo, Yuan Wang, Wei Wang, Yunshi Lan (🖂) · 2025

Read ComRAG: Retrieval-Augmented Generation with Dynamic Vector Stores for Real-time Community Question Answering in Industry by Qinwen Chen, Wenbiao Tao, Zhiwei Zhu, Mingfan Xi, Liangzhong Guo, Yuan Wang, Wei Wang, Yunshi Lan (🖂) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-21098.txt.

Introduction

Qinwen Chen and seven co-authors built ComRAG, a retrieval-augmented answering system for industrial community question platforms, where yesterday's answers are a gold mine that keeps growing. Static knowledge bases miss the living record of what users already asked and solved. ComRAG keeps a dynamic memory of past question-answer pairs alongside the fixed documentation, organized around cluster centers so it stays searchable and compact.

On three industrial datasets, the system beat all baselines, raising answer similarity by up to about twenty-six percent. It also answered faster, cutting latency by up to about twenty-three percent, and its memory grew far more slowly across rounds, about two percent instead of twenty. Those numbers matter in production, where a memory that balloons is a liability. The authors present ComRAG as a design for question answering that learns from its own community without drowning in it.

Catalog record

CREATORQinwen Chen, Wenbiao Tao, Zhiwei Zhu, Mingfan Xi, Liangzhong Guo, Yuan Wang, Wei Wang, Yunshi Lan (🖂)
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-21098

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ComRAG: Retrieval-Augmented Generation with Dynamic Vector Stores for Real-time Community Question Answering in Industry Qinwen Chen, Wenbiao Tao, Zhiwei Zhu, Mingfan Xi, Liangzhong Guo, Yuan Wang, Wei Wang, Yunshi Lan (🖂) Abstract Community Question Answering (CQA) platforms can be deemed as important knowledge bases in community, but effectively leveraging historical interactions and domain knowledge in real-time remains a challenge. Existing methods often underutilize external knowledge, fail to incorporate dynamic historical QA context, or lack memory mechanisms suited for industrial deployment. We propose ComRAG , a retrieval-augmented generation framework for real-time industrial CQA that integrates static knowledge with dynamic historical QA pairs via a centroid-based memory mechanism designed for retrieval, generation, and efficient storage. Evaluated on three industrial CQA datasets, ComRAG consistently outperforms all baselines—achieving up to 25.9% improvement in vector similarity, reducing latency by 8.7%–23.3% , and lowering chunk growth from 20.23% to 2.06% over iterations. 1 Introduction Community Question Answering (CQA) is a collaborative question-and-answer paradigm where users post questions on online platforms (e.g., Stack Overflow 1 1 1 https://stackoverflow.com/ and AskUbuntu 2 2 2 https://askubuntu.com/ ) and community members contribute answers. This paradigm leverages collective intelligence, allowing users to refine answers through voting, commenting, and editing, ultimately enhancing the quality of shared knowledge Roy et al. 2023 . With the rise of ChatGPT Achiam et al. 2023 , DeepSeek DeepSeek-AI et al. 2025 , and other foundation models, Large Language Models (LLMs) have become powerful tools for CQA. However, existing CQA methods focus on static community knowledge, limiting their applicability to real-world scenarios. We categorize existing CQA methods as follows: (1) Retrieval-based methods: Retrievers or rankers identify the most relevant answers from the community. Question-answer cross-attention networks with knowledge augmentation are utilized for answer selection Hu 2023 , while structured information is leveraged to enhance ranking Askari et al. 2024 ; Ghasemi and Shakery 2024 . (2) Generation-based methods: LLMs serve as community experts to answer professional questions. Techniques such as instruction tuning Yang et al. 2023 , reinforcement learning Gorbatovski and Kovalchuk 2024 and contrastive learning Yang et al. 2025 equip LLMs with domain and community knowledge. However, these methods have the following weaknesses: 1) They often overlook external domain knowledge, limiting their applicability for domain-specific industrial applications. 2) Real-time CQA presents a continuous stream of questions rather than a static pool, requiring systems to reflect historical interactions. 3) A suitable memory mechanism is needed for real-time industrial CQA, but existing methods overlook this issue. Domain knowledge and community interaction history play key roles in industrial CQA, shaping the professionalism and relevance of responses, respectively. Thus, the following two key questions require our attention. (Q1) How can we build a CQA system that combines static knowledge with dynamic reflection on the disparate quality of historical answers? Domain knowledge serves as an authoritative benchmark, while community QA history links user queries to relevant insights. Combining both enhances LLMs’ ability to generate reliable CQA responses. (Q2) How can a real-time CQA system manage both the rapidly growing volume of historical QA data and the wide variance in response quality? The evolution of the community leads to varying quality in historical responses due to the open and collaborative nature of CQA. Efficiently identifying, organizing, and leveraging high- and low-quality QA records becomes crucial for maintaining reliable generation. To tackle these challenges, we propose ComRAG, a retrieval-augmented generation framework that integrates static domain knowledge with dynamic historical QA interactions to enhance real-time CQA in industrial settings. In the query phase, ComRAG supports three strategies based on query characteristics: directly reusing answers from high-quality QA pairs, generating responses with reference to high-quality content, and generating responses while explicitly avoiding low-quality ones. During generation, an adaptive temperature tuning mechanism ensures more confident responses. In the update phase, the system dynamically manages high- and low-quality CQA vector stores using a centroid-based memory mechanism, optimizing retrieval efficiency for continuous question streams. In summary, the contributions are as follows. • We propose ComRAG, a novel retrieval-augmented generation framework that jointly integrates static domain knowledge and dynamic community history to address real-time industrial CQA. • We develop a centroid-based memory mechanism for efficient retrieval and an adaptive temperature tuning mechanism for confident generation. • We extensively evaluate our framework on MSQA, ProCQA and PolarDBQA, demonstrating its effectiveness and efficiency for real-time industrial CQA. 2 Related Work 2.1 Community Question Answering The CQA task centers on improving the relevance and quality of answers. We categorize existing approaches to CQA into two main paradigms: retrieval-based and generation-based. Retrieval-based methods aim to identify relevant community answers using retrievers or rankers. Some enhance answer selection by integrating cross-attention networks with LLM-augmented knowledge Hu 2023 or incorporate structured metadata into cross-encoder re-ranking Askari et al. 2024 . Expert finding in CQA is supported by modeling user interactions via topic-based multi-layer graphs Amendola et al. 2024 while modality-agnostic contrastive pretraining is proposed for aligning code-question pairs Li et al. 2024 . Expanding queries and computing translation-based similarity using category-specific dictionaries improve question retrieval Ghasemi and Shakery 2024 . Generation-based methods rely on LLMs acting as community experts to generate answers. Prior work explores strategies such as pretraining a small expert model on documentation and CQA data to inject domain knowledge Yang et al. 2023 , reinforcement learning from human feedback (RLHF) using community voting signals as rewards Gorbatovski and Kovalchuk 2024 , and aligning LLMs via multi-perspective ranking and contrastive learning to better satisfy diverse user preferences Yang et al. 2025 . Despite strong performance on static benchmarks, existing CQA methods largely overlook the dynamic nature of community content and the inconsistency of historical responses. 2.2 Retrieval-Augmented Generation Retrieval-Augmented Generation (RAG) Lewis et al. 2021 has become a promising framework for enhancing LLMs with external knowledge access. RAG augments the input to generation models by retrieving relevant documents, which helps mitigate hallucinations, knowledge staleness, and limited interpretability Gao et al. 2024 . RAG has shown effectiveness across a range of knowledge-intensive tasks, including open-domain QA and summarization Siriwardhana et al. 2022 . Although current RAG implementations rely on static corpora, its retrieval-generation paradigm naturally lends itself to addressing challenges in real-time industrial CQA by enabling fast access to dynamically updated data and supporting the design of customizable retrieval strategies. 3 Task Definition We formally define the task of answering real-time community questions with external knowledge. Given a collection of documents D=\{d_{i}\}_{i=1}^{|D|} as the external knowledge, the community questions arrive as a continuous stream. Suppose that at this moment, we have collected the community history denoted as H=\{(q_{i},a_{i})\}_{i=1}^{|H|} where the historical questions are associated with the historical responses. When there is a new question q , we can extract the answer \hat{a} either from external knowledge or from the community history to ensure that \hat{a} equals the ground truth answer a^{*} . Furthermore, we should determine how to organize (q,\hat{a}) in the community history H to meet memory constraints and accommodate future follow-up questions. 4 Our System: ComRAG Figure 1: ComRAG architecture for real-time CQA. The system integrates a static knowledge vector store and two dynamic CQA vector stores (high- and low-quality), with the latter managed via a centroid-based memory mechanism. When a question is posed, ComRAG follows one of three paths: ① retrieving

Read the complete text