Repairing Tool Calls Using Post-tool Execution Reflection and RAG

Jason Tsay, Zidane Wright, Gaodan Fang, Kiran Kate · 2025

Read Repairing Tool Calls Using Post-tool Execution Reflection and RAG by Jason Tsay, Zidane Wright, Gaodan Fang, Kiran Kate (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17874.txt.

Introduction

Jason Tsay and five co-authors fix failed tool calls made by software agents after the tool has already run. Their test case is kubectl, the command tool for running Kubernetes clusters. When a command fails, the system reflects on the error and searches tool documents and fix guides for help.

Across a large study, the guided fixes raised the share of commands that run cleanly for most models tested. Correct answers to the user's task rose as well. Readers who build agents get a practical lesson: let the system read the failure, fetch the right manual page, and then retry.

Catalog record

CREATORJason Tsay, Zidane Wright, Gaodan Fang, Kiran Kate
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-17874

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Repairing Tool Calls Using Post-tool Execution Reflection and RAG Jason Tsay, Zidane Wright, Gaodan Fang, Kiran Kate Abstract Agentic systems interact with external systems by calling tools such as Python functions, REST API endpoints, or command line tools such as kubectl in Kubernetes. These tool calls often fail for various syntactic and semantic reasons. Some less obvious semantic errors can only be identified and resolved after analyzing the tool’s response. To repair these errors, we develop a post-tool execution reflection component that combines large language model (LLM)-based reflection with domain-specific retrieval-augmented generation (RAG) using documents describing both the specific tool being called and troubleshooting documents related to the tool. For this paper, we focus on the use case of the kubectl command line tool to manage Kubernetes, a platform for orchestrating cluster applications. Through a larger empirical study and a smaller manual evaluation, we find that our RAG-based reflection will repair kubectl commands such that they are both more likely to successfully execute ( pass rate ) for 55% of our models evaluated and 36% more likely to correctly answer the user query on average. We find that troubleshooting documents improve pass rate compared to official documentation by an average of 10%. 1 Introduction When large language models (LLMs) call tools to address user queries, they identify the tool name and assign values to input arguments Qin et al. 2023 ; Patil et al. 2023 . Whether the tool is a REST API endpoint, Python function, or command line tool, they format the call accordingly and send it to the appropriate tool executor. These tool calls sometimes fail for various reasons. For example, the LLM may have hallucinated the function name or made syntactical errors in formulating the tool call Deshpande et al. 2025 . Such syntactic errors can be identified before the tool executor attempts to execute the call. However, some errors are not as obvious such as semantic or contextual errors. Such errors can only be identified and resolved after executing the tool and analyzing the tool’s response Sun et al. 2024 . An example contextual error is when attempting to retrieve a list of pods in a given namespace in Kubernetes, it is only possible to know that the namespace doesn’t exist after executing that command. In agentic settings, post-tool execution errors are usually handled by adding the tool response to the prompt in the next turn and allowing agents to attempt a resolution based on their reasoning abilities Aksitov et al. 2023 ; Yao et al. 2023 . Often, the LLMs are prompted to reflect and reason explicitly on these outcomes and change its course of action, we call this self-reflection. Even though different models have exhibited varying levels of reasoning sophistication Cheng et al. 2025 , most agents still fail at recovering from tool errors Basu et al. 2025 . This suggests that LLMs underlying knowledge of tools (if any) may be insufficient for real-time troubleshooting. Thus, we believe components beyond self-reflection that integrate domain knowledge are necessary for agents to identify and repair errors when they happen. Our approach combines contextual information from the environment through the results of the tool execution along with additional domain information from external documents using RAG (Retrieval Augmented Generation). Figure 1 shows the workflow of such an agentic system. The agent first generates a tool call based on a user’s natural language query. This tool call is then executed, and any resulting execution errors are captured. When an error occurs, the system starts to construct a “repair context” using a RAG-based process. Using the natural language query, the captured error output and the generated tool call as a key, we retrieve relevant information from official tool documentation, troubleshooting documents, and any other potentially useful information for recovering from the error. We combine that key and the retrieved results to create the repair context and pass it to a repair agent to generate a repaired tool call. This repaired tool call can be re-executed and the RAG-based repair process may be repeated as necessary. Figure 1: The process of RAG-based repair. On the left we have an agentic system that generates tool calls and executes them using some tool executor. The right side shows our RAG-Repair approach for post-tool execution reflection that uses the repair context to repair the failing tool cool. To evaluate our post-tool reflection using RAG, we use a dataset of natural language to Kubernetes commands (called kubectl ). This is a real-world tool usage dataset with complex tools which can be executed to obtain execution errors. The kubectl commands also have the benefit of being relatively bounded and executable in ways that may or may not change the state of the external environment. We evaluate our reflection component using a set of 772 failing kubectl commands which are generated from natural language user queries by a baseline agent using an LLM. We perform both a larger-scale empirical evaluation on how often our reflection is able to repair commands such that they are able to successfully execute ( pass rate ) and a smaller-scale manual evaluation where we examine how correctly the repaired command fulfills the user query. Our findings suggest that compared to our non-RAG baseline, both pass rate and correctness are more likely to improve using our RAG-based reflection. To examine the effect of document choice for the RAG component, we also perform an ablation study where we find that troubleshooting documents tend to increase pass rate compared to official documentation. An open source implementation of this approach is available as part of the Agent Lifecycle Toolkit 1 1 1 https://github.com/AgentToolkit/agent-lifecycle-toolkit . 2 Related Work 2.1 Tool Calling Errors Most of the tool calling work focuses on the ability of LLMs to generate a correct tool call given a catalog of tool specifications ( Qin et al. 2023 ; Patil et al. 2023 ; Abdelaziz et al. 2024 ) . Studies on tool calling errors ( Kokane et al. 2024 ; Kokane et al. 2025 ; Winston and Just ) emphasize categories related to generation of tool calls. Post-tool execution errors can be categorized into silent errors ( Sun et al. 2024 ) and explicit execution failures ( Kokane et al. 2025 ) . A prevalent approach in handling execution failures is to pass the error to the LLM/agent and rely on its ability to correct the course of action ( Yao et al. 2023 ; Qin et al. 2023 ) . Unlike this approach, our work leverages the knowledge base of tool documentation and troubleshooting discussions to help the LLM recover from execution failures. 2.2 RAG + code documentation/error logs While most popular for text tasks, RAG has been effective in code generation ( Wang et al. 2025 ) and performance bug fixes ( Garg et al. 2025 ) . For tool calling, RAG has been used to select a subset of tool specifications from a large pool in order to add those to the LLM prompt ( Qin et al. 2023 ; Lumer et al. 2024 ) . To the best of our knowledge, no existing work studies RAG for tool execution errors. 3 RAG for Post-Tool Execution Reflection Even though tool

Read the complete text