It’s LIT! Reliability-Optimized LLMs with Inspectable Tools

Ruixin Zhang, Jon Donnelly, Zhicheng Guo, Ghazal Khalighinejad, Haiyang Huang, Alina Jade Barnett, Cynthia Rudin · 2025

Read It’s LIT! Reliability-Optimized LLMs with Inspectable Tools by Ruixin Zhang, Jon Donnelly, Zhicheng Guo, Ghazal Khalighinejad, Haiyang Huang, Alina Jade Barnett, Cynthia Rudin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14903.txt.

Introduction

Ruixin Zhang, Jon Donnelly, and five co-authors present LIT, a framework that pushes language models to solve problems with inspectable external tools. Models that reason in private text can pick shaky solution paths that users cannot check, a real problem in high-stakes work. LIT instead forces the model to route work through tools whose steps can be seen and fixed.

The framework ranks solution paths by how reliable and easy to troubleshoot they are, even when a path needs several tool calls in a row. It builds on the tool-calling skills that current models already have. For fields like medicine, law, and engineering support, that shift from fluent answers to auditable steps is the point.

Catalog record

CREATORRuixin Zhang, Jon Donnelly, Zhicheng Guo, Ghazal Khalighinejad, Haiyang Huang, Alina Jade Barnett, Cynthia Rudin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-14903

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

It’s LIT! Reliability-Optimized LLMs with Inspectable Tools Ruixin Zhang, Jon Donnelly, Zhicheng Guo, Ghazal Khalighinejad, Haiyang Huang, Alina Jade Barnett, Cynthia Rudin Abstract Large language models (LLMs) have exhibited remarkable capabilities across various domains. The ability to call external tools further expands their capability to handle real-world tasks. However, LLMs often follow an opaque reasoning process, which limits their usefulness in high-stakes domains where solutions need to be trustworthy to end users. LLMs can choose solutions that are unreliable and difficult to troubleshoot, even if better options are available. We address this issue by forcing LLMs to use external – more reliable – tools to solve problems when possible. We present a framework built on the tool-calling capabilities of existing LLMs to enable them to select the most reliable and easy-to-troubleshoot solution path, which may involve multiple sequential tool calls. We refer to this framework as LIT (LLMs with Inspectable Tools). In order to support LIT, we introduce a new and challenging benchmark dataset of 1,300 questions and a customizable set of reliability cost functions associated with a collection of specialized tools. These cost functions summarize how reliable each tool is and how easy it is to troubleshoot. For instance, a calculator is reliable across domains, whereas a linear prediction model is not reliable if there is distribution shift, but it is easy to troubleshoot. A tool that constructs a random forest is neither reliable nor easy to troubleshoot. These tools interact with the Harvard USPTO Patent Dataset and a new dataset of NeurIPS 2023 papers to solve mathematical, coding, and modeling problems of varying difficulty levels. We demonstrate that LLMs can achieve more reliable and informed problem-solving while maintaining task performance using our framework. \workshoptitle Workshop on Multi-Turn Interactions in Large Language Models Figure 1: Conceptual summary. By using the LIT framework, the LLM selects more reliable and inspectable tools, allowing human users to debug and refine the solution proposed by the LLM, producing the correct result. In contrast, the vanilla LLM without LIT selects uninspectable solutions with reasoning that cannot be inspected or corrected. 1 Introduction Large language models (LLMs) are arguably the most complicated black box machine learning algorithms currently in existence. Their transformer architectures make them extremely difficult to trust and troubleshoot. When they answer difficult questions wrong, no one knows why, and no one can figure out why [ 6 ] . One path towards a potential solution is the use of external tools by LLMs [ 20 ] . Rather than depending on a single huge LLM to solve every problem or perform every calculation, external tools specialize in various tasks and can be called by the LLM to perform them as part of solving a larger problem. For instance, rather than depending on a single LLM to solve (502*2952+323)/63, the LLM could call a calculator. Other tools might retrieve datasets, execute code, or perform other tasks. Allowing LLMs to call external tools offers a promising way to mitigate their complexity for several reasons. First, tools are specialized and more reliable; second, using them makes the LLM’s logic more modular, and allows developers to take responsibility for key steps in forming a solution [ 2 , 7 , 9 , 33 , 20 , 21 ] . Importantly, not all tools are created the same. They vary tremendously in their reliability and ease of troubleshooting. For instance, while a calculator is completely reliable given the correct inputs, a specialized model for solving physics problems may not be (though it might be more reliable for physics problems than the more general LLM calling it, and certainly easier to troubleshoot). Prior work on tool-based LLMs [ 18 , 31 ] failed to account for the relative reliability and inspectability of tools within their toolboxes, which means their LLMs’ solutions are likely more opaque and difficult to troubleshoot than necessary. Prior to the current work, there existed no metrics, baseline methods, standard frameworks, or question banks for exploring the reliability of tool selection in LLMs. We address this shortcoming by explicitly encouraging an LLM to prefer reliable and inspectable tools over alternatives by assigning a reliability and inspectability cost to each tool call and choosing the solution with the minimum cost. This allows the LLM to provide reliable and inspectable reasoning whenever possible without sacrificing the ability to answer difficult prompts correctly. This is illustrated in Figure 1 . For example, imagine that we asked an LLM which five authors have the most NeurIPS 2023 papers with titles that mention “Large Language Models.” The LLM might simply provide a random list of 5 author names, but we would prefer that it use SQL code with reliable logic to identify these 5 authors. We introduce the LLMs with Inspectable Tools (LIT) framework. During inference, the LLM generates multiple candidate solutions, each consisting of a sequence of tool calls. For each solution, the model computes the total cost based on the individual costs of each tool that it uses. It then selects the most reliable and inspectable option that still ensures accurate results. Once the optimal solution is identified, the LLM sequentially executes the necessary tool calls to produce the final answer. We evaluate this framework using a suite of specialized tools with varying levels of reliability and inspectability, a novel benchmark dataset of 1,300 questions with varying levels of difficulty, and two distinct problem domains. We use 50% of these questions for validation, and the remaining 50% for testing (the LIT framework does not include any model training). We find that prompting the LLM to prefer reliable and inspectable tools dramatically increases the transparency of the given solutions, and also improves the model’s accuracy in answering most of these questions. In summary, we provide the following contributions: 1. We introduce a prompting framework that explicitly accounts for differences in the reliability and inspectability of tools (Appendix C contains detailed prompts). We show that, across several LLMs, this framework leads to responses that exhibit enhanced inspectability while maintaining accuracy. 2. We provide a suite of 8 tools with varying levels of reliability and inspectability, including tools for arithmetic operations (Calculator), dataset retrieval (DBLoader), flexible code execution (PandasInterpreter, PythonInterpreter), time series forecasting (Forecaster), text classification (TextualClassifier), language model inference (LLMInferencer), and result consolidation (Finish). 1 1 1 Each tool is designed to offer deterministic results for reproducibility. 3. We introduce a new benchmark for tool-based LLMs consisting of 1,300 questions about two external databases: the Harvard USPTO Patent Dataset [ 24 ] and a novel database with metadata from NeurIPS papers from 2023. 2 Related work 2.1 Tool learning Our work is closely related to the topic of tool learning. Tool learning allows LLMs to use external tools to solve complex problems, gain real-time domain knowledge, or mitigate their own limitations [ 29 , 13 , 16 , 30 , 10 , 25 , 9 , 33 ] . Although previous work has shown the effectiveness of tool learning, if there are multiple choices of viable tools available, the algorithm will somehow choose one set of tools (or choose to call none at all), and the decision logic would likely remain opaque to human operators. In previous work on both

Read the complete text