Query Suggestion for Retrieval-Augmented Generation via Dynamic In-Context Learning

Fabian Spaeh, Tianyi Chen, Chen-Hao Chiang, Bin Shen · 2026

Read Query Suggestion for Retrieval-Augmented Generation via Dynamic In-Context Learning by Fabian Spaeh, Tianyi Chen, Chen-Hao Chiang, Bin Shen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08105.txt.

Introduction

Fabian Spaeh, Tianyi Chen, Chen-Hao Chiang, and Bin Shen study how retrieval agents should handle questions they cannot ground. Their 2026 arXiv paper covers agentic retrieval-augmented generation, where tool-using agents answer from outside knowledge. When a question falls outside that knowledge, hallucination becomes a real risk.

Guardrail systems try to block out-of-scope questions before harm is done. The authors instead work on query suggestion through dynamic in-context learning. The aim is to steer users toward questions the system can answer from its sources. For builders of search assistants, the paper links question wording to safer, better-grounded answers.

Catalog record

CREATORFabian Spaeh, Tianyi Chen, Chen-Hao Chiang, Bin Shen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-08105

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Query Suggestion for Retrieval-Augmented Generation via Dynamic In-Context Learning Fabian Spaeh, Tianyi Chen, Chen-Hao Chiang, Bin Shen Abstract Retrieval-augmented generation with tool-calling agents (agentic RAG) has become increasingly powerful in understanding, processing, and responding to user queries. However, the scope of the grounding knowledge is limited and asking questions that exceed this scope may lead to issues like hallucination. While guardrail frameworks aim to block out-of-scope questions Rodriguez et al. 2024 , no research has investigated the question of suggesting answerable queries in order to complete the user interaction. In this paper, we initiate the study of query suggestion for agentic RAG. We consider the setting where user questions are not answerable, and the suggested queries should be similar to aid the user interaction. Such scenarios are frequent for tool-calling LLMs as communicating the restrictions of the tools or the underlying datasets to the user is difficult, and adding query suggestions enhances the interaction with the RAG agent. As opposed to traditional settings for query recommendations such as in search engines, ensuring that the suggested queries are answerable is a major challenge due to the RAG’s multi-step workflow that demands a nuanced understanding of the RAG as a whole, which the executing LLM lacks. As such, we introduce robust dynamic few-shot learning which retrieves examples from relevant workflows. We show that our system can be self-learned, for instance on prior user queries, and is therefore easily applicable in practice. We evaluate our approach on three benchmark datasets based on two unlabeled question datasets collected from real-world user queries. Experiments on real-world datasets confirm that our method produces more relevant and answerable suggestions, outperforming few-shot and retrieval-only baselines, and thus enable safer, more effective user interaction with agentic RAG. 1 Introduction Retrieval-augmented generation (RAG) allows large language models (LLMs) to interact and manipulate data from external sources via tool calls Lewis et al. 2020 . RAG circumvents well-known limitations of LLMs such as the inability to expand their memory after training, carrying out exact computations, or their lack of domain-specific knowledge Naveed et al. 2023 ; Tänzer et al. 2022 ; Brown et al. 2020 . Even though breakthroughs in LLMs that facilitate RAG are recent, they are already deployed and enjoy increasing popularity Patil et al. 2023 ; Zhang et al. 2024 ; Patil et al. 2024 . Many LLMs now natively support external tool-calling functionality, simplifying the implementation of RAG agents. In our work, we consider RAG agents that go beyond free-text retrieval and support arbitrary tool calls, which is why they are also referred to as “agentic.” Such a system makes progress towards an answer via a workflow that is dynamically decided by the LLM. Such a workflow can include multiple steps of tool calls and reasoning, before a final response is delivered to the user. Agentic RAG lowers the barrier of entry for interacting with the underlying data. On the flip side, this means that typical users will be unaware of the underlying tool calls and data representation. This introduces challenges in the RAG-user interaction and queries can often fail: For instance, users may ask for information that is not available, e.g., required information is not present in a database. Queries may also fail due to technical reasons, e.g., when a user wants to search in a column that is not indexed. To alleviate such issues, we propose query suggestion for agentic RAG. Specifically, we aim to suggest queries to the user that fulfill two goals: 1. The suggested query should be semantically similar to the user query and carry the same intent as the original query. 2. The suggested query should be answerable by the RAG. To the best of our knowledge, this problem is novel in the context of agentic RAG. The closest to our problem is the work of Tayal and Tyagi 2024 , which considers query suggestion for RAG for free-text retrieval. However, since this does not involve an arbitrary agentic workflow, it is similar to traditional work on query suggestion. Compared to this, we face additional challenges in our setup on agentic RAG: Queries can pertain to values that appear in any source (e.g., database) and the number of queries is much larger, because the agentic RAG can execute an arbitrary set of workflows. Understanding whether a question can be translated into an executable workflow by the agentic RAG is difficult, mainly due to the disconnect between the LLM, the available tools, and the underlying data. A priori, the LLM does not know the available data and executing a workflow successfully requires nuanced understanding of the available tools which is difficult to deliver via prompt instructions. Furthermore, in order to produce a valid response, RAG agents may undergo several layers of self-reflection Renze and Guven 2024 and confidence estimation Li et al. 2024 . As these are executed on higher level, understanding the outcome of a query without executing it becomes challenging. 1.1 Our Contributions and Techniques We are the first to study the problem of query suggestion for agentic RAG and make the following contributions towards this problem. 1. We propose a query suggestion method based on few-shot learning. To enhance the LLM’s understanding about query answerability, we propose robust dynamic few-shot learning, which is a novel method to retrieve relevant examples and automatically combat hallucination. We facilitate retrieval through a templating approach, which can be understood as reducing a query to its workflow. 2. We show how training examples can be self-learned, which makes our approach immediately applicable. We use the RAG itself to answer and label queries, and we thus do not rely on large sets of labeled data. As such, our approach is practically applicable and can learn from prior user queries in an unsupervised way. 3. We experimentally evaluate our approach on three real-world benchmark datasets, where we consistently outperform static few-shot learning as a baseline and a retrieval-only approach. This also shows that static few-shot learning without the dynamic retrieval of examples does not lead to effective query suggestion, implying that a mere description of tools does not suffice for the complex task of query suggestion for agentic RAG. On a conceptual level, we introduce a distinction between two kinds of unanswerable queries: Queries without a workflow to answer them, and queries for which the called tools cannot find the needed information in the underlying data. 1.2 Related Work Query Suggestion Query suggestion aims to identify a follow-up query that the user may ask Cao et al. 2008 ; Mei et al. 2008 ; Baeza-Yates et al. 2004 . Tayal and Tyagi 2024 are the first to study query suggestion. However, their work does not apply to agentic workflows, but is instead tailored to free-text retrieval. Agentic RAG is more challenging and requires a more sophisticated retrieval scheme, as it is much harder to guarantee answerability due to the multi-step workflow. Furthermore, we use a form of transfer learning which allows our approach to be entirely self-learning, while the method of Tayal and Tyagi 2024 relies on hand-crafted training examples. To the best of our knowledge, this is the only prior work on query suggestion for RAG, and it is not in our tool-calling setting. Recent

Read the complete text