Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis
Dong HUANG, Mingzhe Du, Jie M. Zhang, Zheng Lin, Meng Luo, Regina Zhang, See-Kiong Ng · 2025
Read Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis by Dong HUANG, Mingzhe Du, Jie M. Zhang, Zheng Lin, Meng Luo, Regina Zhang, See-Kiong Ng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26423.txt.
Introduction
Dong Huang, with six co-authors, tackles a quiet weak point in software testing: the oracle, the rule that says whether a program's output is right. Writing good oracles by hand is slow. Their system, Nexus, uses several AI agents to draft, debate, and fix oracles for a function under test.
Huang and the team give four agents distinct testing outlooks, let them critique a first draft, then run the oracles against a plausible version of the function in a safe sandbox. Any oracle that fails is repaired using the exact error it hit, then checked again. On seven benchmarks, Nexus beat prior methods by wide margins. Its oracles also caught more bugs and raised the success of automatic program repair from about thirty-five percent to about sixty-nine percent in one test.
Catalog record
| CREATOR | Dong HUANG, Mingzhe Du, Jie M. Zhang, Zheng Lin, Meng Luo, Regina Zhang, See-Kiong Ng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-26423
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis Dong HUANG, Mingzhe Du, Jie M. Zhang, Zheng Lin, Meng Luo, Regina Zhang, See-Kiong Ng Abstract Test oracle generation in non-regression testing is a longstanding challenge in software engineering, where the goal is to produce oracles that can accurately determine whether a function under test (FUT) behaves as intended for a given input. In this paper, we introduce Nexus, a novel multi-agent framework to address this challenge. Nexus generates test oracles by leveraging a diverse set of specialized agents that synthesizes test oracles through a structured process of deliberation, validation, and iterative self-refinement. During the deliberation phase, a panel of four specialist agents, each embodying a distinct testing philosophy, collaboratively critiques and refines an initial set of test oracles. Then, in the validation phase, Nexus generates a plausible candidate implementation of the FUT and executes the proposed oracles against it in a secure sandbox. For any oracle that fails this execution-based check, Nexus activates an automated self-refinement loop, using the specific runtime error to debug and correct the oracle before re-validation. Our extensive evaluation on seven diverse benchmarks demonstrates that Nexus consistently and substantially outperforms state-of-the-art baselines. For instance, Nexus improves the test-level oracle accuracy on the LiveCodeBench from 46.30% to 57.73% for GPT-4.1-Mini. The improved accuracy also significantly enhances downstream tasks: the bug detection rate of GPT-4.1-Mini generated test oracles on HumanEval increases from 90.91% to 95.45% for Nexus compared to baselines, and the success rate of automated program repair improves from 35.23% to 69.32%. 1 Introduction Unit testing represents a cornerstone of modern software engineering, enabling developers to verify the correctness of individual code components in isolation ( Beck 2022 ) . A well-crafted unit test comprises two essential components: a test input that exercises a specific execution path within the function under test (FUT), and a test oracle that determines whether the function’s actual output aligns with its expected behavior ( Hossain & Dwyer 2024 ) . Despite their critical importance, constructing comprehensive unit tests remains a labor-intensive endeavor that demands not only deep domain expertise but also meticulous attention to edge cases and complex interaction patterns ( Daka et al. 2015 ; Daka & Fraser 2014 ) . To reduce the burden of manual test construction, researchers have developed a wide range of automated test generation techniques, from fuzzing ( Fioraldi et al. 2023 ; Miller et al. 1990 ) , symbolic execution ( Cadar et al. 2008 ; Godefroid et al. 2005 ) , and search-based testing ( Fraser & Arcuri 2011a ; Fraser & Arcuri 2011b ) to modern LLM-based approaches ( Du et al. 2024 ; Qing et al. 2025 ; Huang et al. 2024 ) . However, these methods largely prioritize generating diverse test inputs for code coverage, while oracle generation remains secondary due to the long-standing oracle problem ( Barr et al. 2014 ) —the difficulty of determining correct outputs for non-trivial programs. As a result, most work rely on regression oracles , which record outputs from the current implementation as expected behavior. While valuable for detecting behavioral changes, regression oracles fundamentally cannot verify the functional correctness of new or modified code against its intended specification. The challenge of generating specification-based oracles that verify correctness against intended behavior rather than existing implementation has emerged as a critical bottleneck in automated testing. Moreover, it brings significant obstacles to LLM-based and agent-based automatic code generation, where reliable feedback is essential for enabling LLMs to iteratively refine their outputs. Recent advances in LLMs have opened new avenues for addressing this challenge, with CANDOR ( Xu et al. 2025 ) representing the state-of-the-art, introducing a multi-agent framework where multiple LLM agents collaborate through panel discussions to reach consensus on oracle predictions. However, our analysis reveals two key limitations of CANDOR: single-paradigm reasoning and a deliberation-only architecture. First, all agents rely on the same source of truth (i.e., the natural language documentation of the FUT), and play similar roles based on identical instructions. This creates an epistemological bottleneck, as the agents share the same interpretation of the specification. Second, reasoning is confined to abstract, language-based deliberation without grounding in execution. Since proposed oracles are never validated against even a plausible implementation, the framework lacks feedback to detect subtle bugs or undocumented edge cases. Figure 1: The architecture of the Nexus framework. (1) Deliberation: A panel of specialist agents collaborates to produce candidate oracles. (2) Validation: Oracles are validated by executing them against a plausible, LLM-generated implementation of the function under test. (3) Self-Refinement: Failed oracles enter an iterative loop where runtime errors are used for automated debugging. This pipeline grounds abstract deliberation in executable evidence. In this paper, we introduce Nexus , a novel multi-agent framework that overcomes this limitation by grounding language-based reasoning in executable evidence. First, to break the epistemological echo chamber, the framework begins with a deliberation phase powered by a panel of four specialist agents, each embodying a distinct and orthogonal testing philosophy: a Specification Expert for literal compliance, an Edge Case Specialist for adversarial robustness, a Functional Validator for high-level semantic properties, and an Algorithmic Analyst for implementation-level logic. This multi-perspective analysis ensures a far more comprehensive initial critique than a panel merely re-examining the same specification. Second, to transcend the limitations of any purely text-based analysis, Nexus introduces a validation phase. Here, the framework generates a plausible candidate implementation of the FUT and executes the proposed oracles against this code in a secure sandbox, providing a powerful, execution-based source of truth that is independent of the original documentation. Finally, to capitalize on this new source of feedback, Nexus activates a self-refinement loop for any failed oracle. This loop uses the specific runtime error from the sandbox to provide a precise and actionable signal to automatically debug and correct the oracle before re-validation. Our extensive evaluation on seven diverse benchmarks demonstrates that Nexus consistently and substantially outperforms state-of-the-art baselines across a suite of open-source and proprietary models. The superiority of Nexus is not marginal; for instance, when powered by GPT-4.1-Mini, Nexus boosts test-level oracle accuracy on the challenging LiveCodeBench benchmark to 57.73\% from the state-of-the-art’s 46.30\% achieved by CANDOR. The improved accuracy also significantly enhances downstream tasks: the bug detection rate of GPT-4.1-Mini generated test oracles on HumanEval increases from 90.91% to 95.45% for Nexus compared to baselines, and the success rate of automated program repair improves from 35.23% to 69.32%. By uniquely combining multi-agent deliberation with validation and self-refinement, Nexus establishes a new paradigm for generating reliable, specification-grounded test oracles. The main contributions of this work are as follows: • We design and implement Nexus, a novel multi-agent framework that synthesizes test oracles through a unique pipeline of deliberation, validation, and self-refinement, overcoming the single-source-of-truth limitation in prior work. • We introduce a panel of four specialist agents, each embodying a distinct testing philosophy, to facilitate a more comprehensive and effective deliberation process for oracle generation. • We conduct a comprehensive evaluation on seven benchmarks using four different LLMs, demonstrating that Nexus significantly outperforms state-of-the-art methods in oracle accuracy and enhances downstream tasks like bug detection and automated program repair. 2 Related Work The generation of test oracles presents fundamentally different challenges from input