MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Models
Pablo Valle, Shaukat Ali, Aitor Arrieta, Lionel Briand, Fellow, IEEE · 2026
Read MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Models by Pablo Valle, Shaukat Ali, Aitor Arrieta, Lionel Briand, Fellow, IEEE (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24815.txt.
Introduction
Pablo Valle, Shaukat Ali, Aitor Arrieta, and Lionel Briand automate the testing of robots controlled by vision language action models. Such models merge perception, language, and action in one system. Testing them needs oracles: rules that judge whether a task succeeded. Hand built symbolic oracles cost expert effort, tie tightly to one task, and only grade the final state. MANGO, their framework, generates fine grained oracles from plain language task descriptions instead.
Three agents, a Generator, an Assessor, and a Judge, refine the outputs through structured feedback. MANGO first builds a reusable library of atomic tasks. It then grounds an oracle definition for each atomic task in the simulator. Complex instructions are split into ordered atomic steps, each with its own oracle. On the LIBERO and RoboCasa humanoid benchmarks, the generated oracles catch a similar number of failures as symbolic ones. They also localise faults and give richer diagnostic detail.
Catalog record
| CREATOR | Pablo Valle, Shaukat Ali, Aitor Arrieta, Lionel Briand, Fellow, IEEE |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 116 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24815
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Models Pablo Valle, Shaukat Ali, Aitor Arrieta, Lionel Briand, Fellow, IEEE Abstract Vision-Language-Action (VLA) models are emerging robotic control systems that integrate perception, language understanding, and action generation in a unified architecture. Existing testing approaches for VLA-enabled robots rely on manually constructed symbolic test oracles that determine task success from final environment states. These oracles are costly to construct, require domain expertise, and are often tightly coupled to specific tasks and environments, limiting scalability and reuse. Furthermore, they provide only end-state assessments of task outcomes, offering limited insight into intermediate behavior and fault localization. To address these limitations, we introduce MANGO , a multi-agent framework that automatically generates fine-grained oracles from natural-language descriptions of robotic tasks. MANGO first generates a reusable library of atomic tasks, then generates simulator-grounded oracle definitions for each atomic task, and finally produces executable fine-grained oracles by decomposing complex instructions into ordered sequences of atomic actions and corresponding oracles. The framework uses collaborative Generator, Assessor, and Judge agents that iteratively refine generated artifacts through structured feedback. We evaluate MANGO on the LIBERO_10 and RoboCasa Humanoid Tabletop benchmarks. Results show that MANGO generates executable, fine-grained oracles that detect a similar number of failures as symbolic oracles while accurately localizing them and providing richer diagnostic information. Through ablation studies, we further analyzed component contributions and the effect of initial task set, while preserving oracle quality. Overall, the results show the feasibility and effectiveness of test oracle generation for VLA-enabled robots testing. I Introduction Vision-Language-Action (VLA) models are emerging as a new generation of control algorithms for robots [ 83 , 36 , 52 ] . Unlike traditional robotic architectures that separate perception, planning, and control into independent modules, VLA models integrate visual observations, natural-language instructions, and robot state information into a unified multimodal representation from which executable actions are generated. This enables robots to interpret high-level instructions such as “store the bottle of water in the fridge” and autonomously translate them into sequences of low-level control commands. Recent works have demonstrated remarkable capabilities across many manipulation tasks [ 6 , 52 , 57 ] , enabling general-purpose robots operating in complex environments. As VLA-enabled robots become more capable, ensuring their correctness and reliability becomes critical. Current testing approaches [ 75 , 79 ] primarily rely on symbolic test oracles provided by robotic benchmarks [ 45 , 41 ] . Given task instructions and simulator execution, these oracles determine task success by checking whether the environment’s final state matches the expected goal state. For example, in a task requiring a robot to place an object inside a container, the oracle verifies whether the object is inside the target container at the end of the execution. This strategy is widely used in existing benchmarks due to its simplicity and automation. However, symbolic oracles have limitations [ 71 ] . First, they reduce task correctness to a binary decision based solely on the final environment state, ignoring the action sequence that produced it, and thus cannot distinguish between failures arising from different reasons [ 71 ] . For example, a robot tasked with placing a bottle in a fridge may fail because it cannot open the fridge, drops the bottle during transport, or attempts to place it before opening the door. From the perspective of a symbolic oracle, all such executions are indistinguishable, as they collapse to the same outcome: task failure. Second, symbolic oracles provide little support for debugging, as they cannot identify which sub-task failed within a complex instruction that failed during execution. As VLA-enabled robots increasingly execute long-horizon tasks with multiple objects and intermediate goals, this lack of diagnostic information becomes a major obstacle to development and testing. These limitations reflect the test oracle problem in software testing, i.e., the difficulty of determining output correctness automatically [ 5 ] . While symbolic oracles provide a partial solution for robotic benchmarks, they are limited to predefined goal states and fail to capture the structure and quality of task execution. This limitation is particularly problematic in long-horizon robotic manipulation, where success depends not only on achieving a final configuration but also on correctly executing a sequence of interdependent sub-tasks. A natural direction to address this issue is finer-grained evaluation. Rather than treating a complex instruction as a single objective, it can be decomposed into atomic tasks corresponding to reusable capabilities such as opening, grasping, placing, or closing. Associating an oracle with each atomic task reframes evaluation as structured verification that assesses both final goal completion and correctness of intermediate step and their order. Such fine-grained oracles provide richer feedback, improve failure localization, and enable more interpretable assessment of robot behavior. However, manually defining them is impractical at scale, as benchmarks often comprise dozens to hundreds of tasks [ 79 , 20 , 51 ] , each with different objects, environments, and execution constraints. Designing atomic task decompositions and corresponding oracles would require substantial human effort and domain expertise. Moreover, maintaining consistency across manually authored fine-grained oracles becomes increasingly difficult as benchmark complexity grows. In addition, existing benchmarks typically implement symbolic oracles in heterogeneous formats and programming languages; for example, Python [ 51 ] , or .bddl files [ 45 ] . This fragmentation further complicates reuse, interoperability, and scalability across benchmarks. As a result, scalable fine-grained evaluation requires automated methods that translate natural-language instructions into executable, fine-grained oracle specifications. In this paper, we present MANGO , a M ulti- A gent Fi N e- G rained O racle Generator for automatically generating fine-grained test oracles for VLA-enabled robots from natural-language task descriptions. MANGO operates in three stages. First, it automatically generates a reusable library of atomic tasks from user-provided robotic instructions. Second, it generates oracle definitions for each atomic task using available target simulator functions. Finally, it generates executable fine-grained oracles for individual tasks by decomposing instructions into ordered atomic tasks and linking each step to its corresponding oracle. To ensure correctness and robustness, MANGO employs a collaborative multi-agent architecture with Generator , Assessor , and Judge agents that iteratively refine generated artifacts through structured feedback. We evaluate MANGO on two representative robotic benchmarks, LIBERO_10 [ 45 ] and RoboCasa Humanoid Tabletop [ 52 ] , assessing the ability of MANGO to generate fine-grained oracles, comparing against symbolic oracles for failure detection and localization, and conducting ablation studies on key framework components. We also study how much the initial task set can be minimized while preserving the quality of the generated fine-grained oracles, providing insights for practitioners. The main contributions of the paper are: • A novel formulation of fine-grained test oracles for VLA-enabled robots based on task decomposition into reusable atomic tasks. • MANGO , the first multi-agent framework for automatically generating fine-grained test oracles from natural language task descriptions for VLA-enabled robots. • An extensive empirical evaluation on two robotic benchmarks assessing generation quality, failure detection capability, and failure localization effectiveness. • A replication package, which will be made available upon acceptance, containing the implementation of MANGO , generated oracle libraries, experimental artifacts, and instructions to facilitate reproducibility and future research. II Background In this section, we present the background on VLA-Enabled robots, followed by their testing. II-A VLA-Enabled Robots