InData: Towards Secure Multi-Step, Tool-Based Data Analysis
2025
Read InData: Towards Secure Multi-Step, Tool-Based Data Analysis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11933.txt.
Introduction
Karthikeyan K and three co-authors argue that language model agents should never touch sensitive data directly. The usual pattern, an agent writing and running code against a database, is a security incident waiting for a prompt. Their alternative forces all contact through a fixed set of secure, verified tools.
To test whether models can still do real analysis under that restriction, they introduce InData, a dataset of multi-step, tool-based analysis questions that demand composing tools, not just picking one. Existing benchmarks mostly check tool selection and single executions. Secure-by-design analysis will only be adopted if agents prove they can reason in steps while locked out of the raw data.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-11933
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
InData: Towards Secure Multi-Step, Tool-Based Data Analysis Abstract Large language model agents for data analysis typically generate and execute code directly on databases. However, when applied to sensitive data, this approach poses significant security risks. To address this issue, we propose a security-motivated alternative: restrict LLMs from direct code generation and data access, and require them to interact with data exclusively through a predefined set of secure, verified tools. Although recent tool-use benchmarks exist, they primarily target tool selection and simple execution rather than the compositional, multi-step reasoning needed for complex data analysis. To reduce this gap, we introduce Indirect Data Engagement ( InData ), a dataset designed to assess LLMs’ multi-step tool-based reasoning ability. InData includes data analysis questions at three difficulty levels—Easy, Medium, and Hard—capturing increasing reasoning complexity. We benchmark 15 open-source LLMs on InData and find that while large models (e.g., gpt-oss-120b) achieve high accuracy on Easy tasks (97.3%), performance drops sharply on Hard tasks (69.6%). These results show that current LLMs still lack robust multi-step tool-based reasoning ability. With InData , we take a step toward enabling the development and evaluation of LLMs with stronger multi-step tool-use capabilities. We will publicly release the dataset and code. Keywords: Corpus, Evaluation Methodologies,Controlled Languages,Tools, Systems, Applications InData: Towards Secure Multi-Step, Tool-Based Data Analysis Karthikeyan K, Raghuveer Thirukovalluru, Bhuwan Dhingra, David Edwin Carlson Duke University {karthikeyan.k,raghuveer.thirukovalluru,bhuwan.dhingra,david.carlson}@duke.edu Abstract content 1. Introduction Large Language Models (LLMs) have demonstrated strong capabilities in program synthesis ( 14 ; 17 ) , mathematical reasoning ( 1 ; 26 ) , and data analysis ( 11 ; 41 ) . These capabilities drove rapid adoption across industries that handle sensitive information, including healthcare and finance ( 5 ; 21 ) , where organizations increasingly rely on LLMs to assist with tasks such as querying databases, generating insights. However, most commercial LLMs operate as external services that require transmitting data to their servers for processing. When analysts send sensitive datasets—such as patient medical records or financial transactions—to external LLM providers, they lose control over how the data is stored, processed, and retained ( 42 ; 7 ) , potentially violating data use agreements and regulatory requirements like HIPAA or GDPR. To address these privacy concerns, existing work on LLM agents for data analysis predominantly adopts a code-generation approach: the LLM receives only a description of the analysis task and generates executable code (typically Python or SQL) that processes the data locally, ensuring sensitive data never leaves the protected environment ( 16 ; 11 ) . While code-generation mitigates privacy risks, it introduces severe execution risks: (1) the generated code may invoke external APIs that leak sensitive data; (2) it may contain errors that corrupt data ( 50 ) ; (3) it could introduce security vulnerabilities ( 3 ) ; and (4) it often lacks proper error handling and logging, making failures difficult to audit or debug. Motivated by these risks, we propose restricting LLMs from generating or executing arbitrary code, instead constraining them to solve user task using only a predefined set of vetted tools that securely handle data locally. This approach addresses both privacy and execution risks: since the LLM never receives the raw data and cannot generate arbitrary code, sensitive information remains protected while eliminating execution risks from LLM generated code. While recent work has explored LLM tool-calling capabilities ( 36 ) , existing benchmarks primarily evaluate tool selection from large API collections or simple tasks requiring few tool calls. No existing benchmark evaluates LLMs’ ability to solve complex, multi-step tool-calling tasks. Figure 1: Predefined tools act as a secure barrier between the LLM and sensitive data. To address this gap, we introduce InData , designed to evaluate LLMs’ ability to solve complex, multi-step reasoning data analysis problems using only a predefined set of tools. Our dataset consists of four components: (1) multi-step reasoning questions that require sequential tool usage to solve, (2) underlying CSV data on which the questions rely, (3) a curated set of tools for data manipulation and analysis, and (4) ground truth answers for evaluation. InData comprises 100 tools, 114 CSV files, and 2063 questions spanning three difficulty levels: 484 easy, 369 medium, 1210 hard. Typically easy questions require fewer than 5 tool calls, whereas hard questions often demand more than 15 sequential tool calls to reach the final answer. We benchmarked 15 open-source LLMs on InData . Our results show that while LLMs handle coding and simple tool use—areas well covered by existing benchmarks—they struggle with compositional, multi-step tool reasoning. We also observe a stark performance gap across model sizes. Smaller models, such as Qwen3-1.7B, achieve only 7.5% accuracy on hard questions, whereas gpt-oss-120b, reaches 69.6%. Even among state-of-the-art models, performance declines sharply with problem difficulty—gpt-oss-120b drops from 97.3% on easy questions to 69.6% on hard ones. Furthermore, while tool-only approaches enable secure data analysis, current LLMs lack the capability for complex, multi-step tool-based reasoning, partly because no existing benchmarks systematically evaluate this ability, and consequently, models are not optimized for it. InData addresses this gap by providing a benchmark that measures and facilitates progress in LLMs’ multi-step tool reasoning capabilities. Additionally, we perform extensive analyses and find that: (1) structured hints, such as Python code solutions, substantially improve multi-step tool reasoning; (2) larger models perform best when provided with all the available tools, whereas smaller models perform best with only a limited, relevant subset of tools; and (3) model families differ in behavior—GPT-based models tend to retry persistently, while Qwen-based models stop earlier. Our contributions include: (1) we propose a security-first, tool-only methodology for LLM-based data analysis; (2) we introduce InData , the first dataset specifically designed to evaluate LLMs’ multi-step, tool-based reasoning ability; (3) we benchmark 15 open-source LLMs on InData and observe that while current models are proficient in code generation and simple tool usage, they lack compositional, multi-step tool reasoning ability; and (4) we conduct an extensive analysis and outline directions for improving LLMs’ tool reasoning ability. 2. Related Works LLMs for Data Analysis: Datasets Advances in the agentic capabilities of large language models (LLMs) have led to new datasets and frameworks for evaluating their performance on data analysis tasks. Early contributions include DS-1000 ( 16 ) , a benchmark of 1,000 Python data science problems from Stack Overflow, and ARCADE ( 43 ) , which features 1,078 Jupyter notebook-based tasks. Both primarily assess code-generation proficiency. More recent efforts, such as DAEval ( 12 ) , which evaluates 34 LLMs across 257 closed-form analysis questions, and TAPILOT-CROSSING ( 19 ) , which focuses on conversational tabular data analysis, emphasize agentic reasoning through iterative code generation and refinement in a sandbox. Complementary benchmarks in program synthesis ( 2 ; 6 ; 10 ; 24 ) and text-to-SQL conversion ( 18 ; 45 ; 46 ; 44 ) similarly require models to produce executable code. In contrast, our work diverges by explicitly prohibiting code generation or execution, compelling LLMs to complete data analysis tasks solely through the use of predefined tools. LLMs for Data Analysis: Frameworks and Evaluation Several frameworks and evaluation methodologies have emerged to assess and improve LLM agents in data analysis. Data Interpreter ( 11 ) introduced an LLM agent framework for solving end-to-end data