Tools in the Loop: Quantifying Uncertainty of LLM Question Answering Systems That Use Tools

Panagiotis Lymperopoulos, Vasanth Sarathy · 2025

Read Tools in the Loop: Quantifying Uncertainty of LLM Question Answering Systems That Use Tools by Panagiotis Lymperopoulos, Vasanth Sarathy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16113.txt.

Introduction

Panagiotis Lymperopoulos and Vasanth Sarathy build a way to measure doubt in language model answers that lean on outside tools, in a research paper from 2025. Models often call classifiers or search tools when their own knowledge runs thin. The final answer then rests on two fallible parts, and past doubt measures cover only one.

Their framework rates the model's text doubt and the tool's output doubt jointly. It extends token-level uncertainty methods to the tool-calling case and adds fast approximations for real use. Tests used two new question sets built from classic machine learning data, plus a retrieval-augmented system. The scores flagged shaky answers well, a step toward trust in high-stakes fields such as medicine.

Catalog record

CREATORPanagiotis Lymperopoulos, Vasanth Sarathy
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16113

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tools in the Loop: Quantifying Uncertainty of LLM Question Answering Systems That Use Tools Panagiotis Lymperopoulos, Vasanth Sarathy Abstract Modern Large Language Models (LLMs) often require external tools, such as machine learning classifiers or knowledge retrieval systems, to provide accurate answers in domains where their pre-trained knowledge is insufficient. This integration of LLMs with external tools expands their utility but also introduces a critical challenge: determining the trustworthiness of responses generated by the combined system. In high-stakes applications, such as medical decision-making, it is essential to assess the uncertainty of both the LLM’s generated text and the tool’s output to ensure the reliability of the final response. However, existing uncertainty quantification methods do not account for the tool-calling scenario, where both the LLM and external tool contribute to the overall system’s uncertainty. In this work, we present a novel framework for modeling tool-calling LLMs that quantifies uncertainty by jointly considering the predictive uncertainty of the LLM and the external tool. We extend previous methods for uncertainty quantification over token sequences to this setting and propose efficient approximations that make uncertainty computation practical for real-world applications. We evaluate our framework on two new synthetic QA datasets, derived from well-known machine learning datasets, which require tool-calling for accurate answers. Additionally, we apply our method to retrieval-augmented generation (RAG) systems and conduct a proof-of-concept experiment demonstrating the effectiveness of our uncertainty metrics in scenarios where external information retrieval is needed. Our results show that the framework is effective in enhancing trust in LLM-based systems, especially in cases where the LLM’s internal knowledge is insufficient and external tools are required. 1 Introduction As large language models (LLMs) have been increasingly deployed in practical applications, their problem-solving capacity has enabled practitioners to combine them with external tools to augment their reasoning abilities and available information beyond their training distribution [ 18 ] . This has greatly expanded LLMs’ applicability beyond text-processing applications to problems involving complex decision making, which find applications in various critical domains such as medicine [ 21 ] and the law [ 9 ] . While recent work has developed metrics that help decide whether to trust LLM outputs [ 8 ] , such metrics are not applicable to tool-calling systems directly, as they do not take into account the trustworthiness of the tool itself. This is especially important in highly specialized domains, where the LLM’s own knowledge from training may not suffice and external tools are required. Uncertainty quantification has been a critical component in machine learning for determining when to trust model outputs. In particular, uncertainty measures such as predictive entropy, have proven useful in understanding whether a model’s prediction is reliable, or whether the model is likely to produce unreliable predictions due to, for instance, distribution shift. Techniques such as Monte Carlo dropout [ 5 ] and deep ensembles [ 10 ] have been applied to improve estimates of uncertainty in model predictions, especially in domains like healthcare, where model missteps can be costly. These methods allow practitioners to decide when further human intervention may be needed or when the system might fail. Extending this notion to LLMs, uncertainty quantification over token sequences or meanings has emerged as a tool for determining when to trust generated content [ 8 , 3 , 14 ] . Here, the uncertainty can be tied to the LLM’s internal representation of the sequence, which influences the model’s capacity to produce coherent and accurate text. Recently, generative AI agents have emerged as a powerful way to extend the capabilities of LLMs beyond text generation [ 16 , 13 , 18 ] , allowing these models to act as agents that interact with the world. By calling external APIs, models, or databases, LLMs can retrieve information, perform tasks, or even make decisions based on external computations. This paradigm shift has enabled LLMs to step into more agentic roles, executing actions that can impact real-world systems. However, this ability also comes with heightened safety and trust requirements [ 7 ] . However, uncertainty quantification methods for agentic LLM systems are currently lagging behind, and advancing these techniques is crucial to making such systems safer for deployment in sensitive and high-stakes applications. In this work, we take a first step towards increasing the trustworthiness of LLM agents by building a framework for quantifying the uncertainty in tool-calling question-answering (QA) systems. We begin by modeling the tool-calling LLM system and deriving uncertainty metrics for it, taking into account both the uncertainty of the LLM and the tool being called. In our work, we make the key assumption that we are operating in the white-box setting, where the uncertainty of external tools is known and we have access to model logits. To quantify uncertainty over the meaning of the combined system’s answer, we adapt semantic entropy [ 8 ] , into our framework. Finally, we make the uncertainty computation more efficient by deriving an approximation that makes our framework feasible in many practical settings. We also apply our framework to retrieval-augmented-generation (RAG) tasks. We evaluate our approach on two novel synthetic question-answering (QA) datasets requiring tool calling, derived from the well-known IRIS flower classification [ 4 ] and PIMA diabetes datasets. Finally, we also apply our framework to a simple retrieval-augmented generation [ 6 , 11 ] system and conduct a proof-of-concept experiment quantifying the uncertainty of answering questions from the BoolQ [ 2 ] dataset of yes/no questions using an external database of references. Figure 1: Illustration of our model of the LLM+tool system. The system receives an input prompt x , such as a question that requires a tool (e.g. a classifier) to answer. The LLM produces a tool call a which acts as input to the tool. The tool produces output z , which in turn is mapped to a token sequence and provided to the LLM, alongside the original prompt. Finally, the LLM produces the final answer y . Yellow indicates the features that the LLM needs to extract for the tool call. Green indicates the final question the system needs to answer and red indicates the final answer provided by the combined system. Within this framework we can quantify the uncertainty of the final answer while taking into account the uncertainty of the classifier. 2 Related works In the field of uncertainty quantification (UQ) for large language models (LLMs), a growing body of research has focused on developing techniques to estimate when these models might produce unreliable outputs. One approach involves using supervised models to predict uncertainty by leveraging the LLM’s internal states, such as logits or hidden activations, as well as ground truth labels from training data. For example, recent work [ 12 ] has proposed training a separate uncertainty model based on LLM logits to improve the estimation of confidence in the generated outputs. Another method involves modeling semantic uncertainty [ 8 , 3 ] , which quantifies the LLM’s uncertainty over the meanings of generated text rather than just over individual tokens. This approach is particularly useful for capturing more complex ambiguities. Semantic uncertainty has proven effective in detecting hallucinations and other inconsistencies in LLM-generated content, offering a pathway to more reliable responses. This method relies on entailment models to establish

Read the complete text