VeriGrey: Greybox Agent Validation
Yuntong Zhang, Sungmin Kang, Ruijie Meng, Marcel Böhme, Abhik Roychoudhury · 2026
Read VeriGrey: Greybox Agent Validation by Yuntong Zhang, Sungmin Kang, Ruijie Meng, Marcel Böhme, Abhik Roychoudhury (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17639.txt.
Introduction
Zhang and co-authors present VeriGrey, a greybox tester for language-model agents. Agents that call tools on their own can take rare but risky paths. VeriGrey uses the tool-call sequence as feedback to steer test search. Prompt mutations then hide an injection task inside the agent's own job. On AgentDojo it found a third more injection flaws than a black-box baseline, they report. Case studies hit a coding agent and a personal assistant.
Catalog record
| CREATOR | Yuntong Zhang, Sungmin Kang, Ruijie Meng, Marcel Böhme, Abhik Roychoudhury |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73885 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17639
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VeriGrey: Greybox Agent Validation Yuntong Zhang, Sungmin Kang, Ruijie Meng, Marcel Böhme, Abhik Roychoudhury Abstract . Agentic AI has been a topic of great interest recently. A Large Language Model (LLM) agent involves one or more LLMs in the back-end. In the front end, it conducts autonomous decision-making by combining the LLM outputs with results obtained by invoking several external tools. An LLM agent thus goes significantly beyond what is achieved by an LLM prompt as it exercises autonomy. The autonomous interactions with the external environment introduce critical security risks. The autonomous nature of LLM agents further complicates systematic testing against these risks, as their behaviors are diverse and often unpredictable. Apart from the inherent nondeterminism in LLMs, an agent’s behavior is determined by the tools invoked, as well as the sequence in which these tools are invoked. In this paper, we present a grey-box approach to explore diverse behaviors and uncover security risks in LLM agents. Our approach VeriGrey uses the sequence of tools invoked as a feedback function to drive the testing process. This helps uncover infrequent but dangerous tool invocations that cause unexpected agent behavior. As mutation operators in the testing process, we mutate prompts to design pernicious injection prompts. This is carefully accomplished by linking the task of the agent to an injection task, so that the injection task becomes a necessary step of completing the agent functionality. Comparing our approach with a black-box baseline on the well-known AgentDojo benchmark, VeriGrey achieves 33% additional efficacy in finding indirect prompt injection vulnerabilities with a GPT-4.1 back-end. We also conduct real-world case studies with the widely used coding agent Gemini CLI, and the well-known OpenClaw personal assistant. VeriGrey finds prompts inducing several attack scenarios that could not be identified by black-box approaches. In OpenClaw, by constructing a conversation agent which employs mutational fuzz testing as needed, VeriGrey is able to discover malicious skill variants from 10 malicious skills (with 10/10= 100% success rate on the Kimi-K2.5 LLM backend, and 9/10= 90% success rate on Opus 4.6 LLM backend). This demonstrates the value of a dynamic approach like VeriGrey to test agents, and to eventually lead to an agent assurance framework. 1. Introduction In recent times, there has been significant interest in agentic AI and its deployment. What constitutes a Large Language Model (LLM) agent? An agent accomplishes specific computational tasks by leveraging an LLM back-end. Given a task, it will use an agentic computational front-end, which will use an LLM back-end. The agentic computation will invoke external tools such as shells or web browsers, and will use the results from LLM by issuing prompts to LLM. Solving a task using an agent is very different from solving a task using a deterministic algorithm. Indeed, an agent can have varying levels of autonomy when it comes to how a task is solved. At the lowest level of autonomy, an LLM agent can act as a chatbot waiting for human input, where it acts only when prompted. This leaves the agent with minimal decision-making. At the next level of autonomy, an agent can plan, make decisions, and meet goals such as scheduling meetings (a calendar agent), conducting research (a deep research agent), and so on. The final level of autonomy will allow an agent to operate as a trusted party with little human supervision. Most LLM agents that exist today are in the second category, exercising a high degree of autonomy, but not full autonomy. However, LLM agents are in general more than deterministic algorithms—they can be thought of as a complex software system which employs external tools, as well as machine learning technology to solve computational tasks. Owing to the aforementioned complexities, testing an LLM agent is challenging. First of all, there can be a significant amount of non-determinism in an agent’s behavior. Secondly, there may be unexpected emergent behaviors due to the interaction of the external tools’ output and the LLM’s responses. Thirdly, any use of external tools involves careful vetting against following the appropriate APIs/protocols for the tool usage. Last but not least, the use of adversarial inputs/prompts combined with invoking external tools/LLMs allows for many corner cases in principle, since the behavior is determined by the sequence of invocations. In this paper, we develop a grey-box approach for testing AI agents. With the increased deployment of agents by organizations, validation of the agent security assumes high importance; grey-box fuzzing is a popular choice for practical security testing of open-source software systems. The widely used OSS-Fuzz ( 28 ) infrastructure has found to date more than 10K vulnerabilities and more than 50K bugs across 1,000 open source projects. More importantly, there exists a ready community of researchers and practitioners building and using fuzzing tools such as AFL ( 45 ) , Libfuzzer ( 34 ) , and AFL++ ( 9 ) . Thus, if we can set up a plan and blueprint for grey-box validation of LLM agents, the research could benefit from contributions by many researchers, allowing for building more robust agents in the future. Fuzzing is a biased random-search-based testing technique. Given an input domain, it starts from one or more seed inputs, and conducts random mutations on the seed to construct more inputs. These mutated inputs are executed to find vulnerabilities. This constitutes a basic black-box fuzzing approach. In grey-box approaches, a lightweight program instrumentation is done to capture execution-related information for the inputs that are executed. There exists freedom in deciding what kind of execution information will be tracked during grey-box fuzzing. This execution information acts as the feedback function in grey-box fuzzing, where the execution-based feedback determines whether an explored input is “interesting” for the purposes of exploration (which is needed for testing). For effective grey-box testing, one needs to design a feedback function that is likely to steer the search towards infrequent but dangerous scenarios, so that they are proactively detected ahead of deploying an agent in practice. We find the sequence of tool invocations to provide the basis for such a feedback function. We demonstrate its efficacy with respect to black-box testing approaches on the well-known AgentDojo ( 8 ) benchmark. Our grey-box agent validation framework VeriGrey outperforms the black-box testing baseline by 33% in terms of efficacy in finding vulnerabilities (using a GPT-4.1 LLM in the back-end). Most interestingly, we see improvements in terms of bug finding accuracy across the various vertical application domains covered by AgentDojo - including workspace, travel, and banking domains. We also conduct an ablation study, which shows that the role of the feedback function is significant in terms of finding bugs. If we turn off the feedback function in the VeriGrey implementation, we see a clear reduction in terms of bug finding efficacy. Apart from the feedback function, we also incorporate technical innovations in terms of how the search space of agent inputs is explored. Random mutations of LLM prompts are likely to lead to prompts that will be rejected or ignored by an LLM during an agent’s execution. This is contrary to our testing goal, where we seek to find inputs that will be accepted by the software under test, but will expose subtle bugs during execution. For this reason, we design mutation