Multi-Agent Code Verification via Information Theory

Shreshth Rajan · 2025

Read Multi-Agent Code Verification via Information Theory by Shreshth Rajan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-16708.txt.

Introduction

Shreshth Rajan builds CodeX-Verify, a multi agent system that uses information theory to verify generated code without running tests. Language models produce buggy and vulnerable code at high rates, and existing tools miss many bugs or raise false alarms. Four specialized agents hunt different bug types. Information theory proves that diverse detectors combined find more bugs than any single agent, when their targets differ.

Measured correlations confirm the agents do catch different problems. On ninety-nine labeled code samples, the system catches about three quarters of bugs. That matches the best prior method while running faster and without test execution. Testing all agent combinations shows large accuracy gains from diversity. The result argues verification should be a portfolio, not a single checker.

Catalog record

CREATORShreshth Rajan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-16708

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Agent Code Verification via Information Theory Shreshth Rajan Abstract LLMs generate buggy code: 29.6% of SWE-bench “solved” patches fail, 62% of BaxBench solutions have vulnerabilities, and existing tools only catch 65% of bugs with 35% false positives. We built CodeX-Verify , a multi-agent system that uses four specialized agents to detect different types of bugs. We prove mathematically that combining agents with different detection patterns finds more bugs than any single agent when the agents look for different problems, using submodularity of mutual information under conditional independence. Measuring agent correlation of ρ = 0.05 \rho=0.05 – 0.25 0.25 confirms they detect different bugs. Testing on 99 code samples with verified labels shows our system catches 76.1% of bugs, matching the best existing method (Meta Prompt Testing: 75%) while running faster and without test execution. We tested all 15 agent combinations and found that using multiple agents improves accuracy by 39.7 percentage points (from 32.8% to 72.4%) compared to single agents, with diminishing returns of +14.9pp, +13.5pp, and +11.2pp for agents 2, 3, and 4, validating our theoretical model. The best two-agent combination (Correctness + Performance) reaches 79.3% accuracy. Testing on 300 real patches from Claude Sonnet 4.5 runs in under 200ms per sample, making this practical for production use. Keywords: Multi-agent systems, Code verification, LLM-generated code, Information theory 1 Introduction LLMs generate code that looks correct but often fails in production. While LLM-generated code passes basic syntax checks and simple tests, recent studies show it contains hidden bugs. Xia et al. 28 find that 29.6% of patches marked “solved” on SWE-bench don’t match what human developers wrote, with 7.8% failing full test suites despite passing initial tests. SecRepoBench reports that LLMs write secure code ¡25% of the time across 318 C/C++ tasks 8 , and BaxBench finds 62% of backend code has vulnerabilities or bugs 26 . Studies suggest 40–60% of LLM code contains undetected bugs 13 , making automated deployment risky. The Problem. Existing verification tools check code in one way at a time, missing bugs that require looking from multiple angles. Traditional static analyzers (SonarQube, Semgrep, CodeQL) catch 65% of bugs but flag good code as buggy 35% of the time 22 . Test-based methods like Meta Prompt Testing 27 achieve better false positive rates (8.6%) by running code variants and comparing outputs, but require expensive test infrastructure and miss security holes (SQL injection) and quality issues that don’t affect outputs. LLM review systems like AutoReview 3 improve security detection by 18.72% F1 but only focus on security, not correctness or performance. No existing work explains mathematically why using multiple agents should work better than using one. Our Approach. We built CodeX-Verify , a system that runs four specialized agents in parallel: Correctness (logic errors, edge cases, exception handling), Security (OWASP Top 10, CWE patterns, secrets), Performance (algorithmic complexity, resource leaks), and Style (maintainability, documentation). Each agent looks for different bug types. We prove that combining agents finds more bugs than any single agent using submodularity of mutual information under conditional independence: I(A_{1},A_{2},A_{3},A_{4};B)>\max_{i}I(A_{i};B) . Measuring how often our agents agree shows correlation \rho=0.05 – 0.25 , confirming they catch different bugs. Results. We tested on 99 code samples with verified labels covering 16 bug categories from real SWE-bench failures. Our system catches 76.1% of bugs, matching Meta Prompt Testing (75%) 27 while running faster and without executing code. We improve 28.7 percentage points over Codex (40%) and 3.7 points over traditional static analyzers (65%). Our 50% false positive rate is higher than test-based methods (8.6%) because we flag security holes and quality issues that don’t affect test outputs, a tradeoff appropriate for enterprise deployments that prioritize security over minimizing false alarms. We tested all 15 combinations of agents: single agents (4 configs), pairs (6 configs), triples (4 configs), and the full system. Results show progressive improvement: 1 agent (32.8% avg) \to 2 agents (+14.9pp) \to 3 agents (+13.5pp) \to 4 agents (+11.2pp), totaling 39.7 percentage points gain. This exceeds AutoReview’s +18.72% F1 improvement 3 and confirms the mathematical prediction that combining agents with different detection patterns works. The diminishing gains (+14.9pp, +13.5pp, +11.2pp) match our theoretical model. Contributions. 1. Mathematical proof via submodularity of mutual information that combining agents with conditionally independent detection patterns finds more bugs than any single agent: I(A_{1},\ldots,A_{n};B)>\max_{i}I(A_{i};B) when multiple agents are informative. Measured agent correlation \rho=0.05 – 0.25 shows low redundancy. 2. Proof via submodularity that marginal information gains decrease monotonically (diminishing returns), validated by measured gains of +14.9pp, +13.5pp, +11.2pp for agents 2, 3, 4. 3. Comprehensive ablation testing all 15 agent combinations on a 29-sample hand-curated subset, showing multi-agent improves accuracy by 39.7 percentage points over single agents (32.8% → 72.4%), with best pair (Correctness + Performance) achieving 79.3%. 4. System achieving 76.1% TPR (matching state-of-the-art Meta Prompt Testing at 75%) with ¡200ms latency via static analysis. Dataset of 99 samples with verified labels released open-source. 2 Related Work 2.1 LLM Code Generation and Verification SWE-bench 13 evaluates LLMs on 2,294 real GitHub issues across 12 Python repositories. Follow-up work found problems: Xia et al. 28 show 29.6% of “solved” patches don’t match what developers wrote, with 7.8% failing full test suites despite passing initial tests. OpenAI released SWE-bench Verified 16 , a 500-sample subset with human-validated labels. Security benchmarks show worse results: SecRepoBench 8 reports ¡25% secure code across 318 C/C++ tasks, and BaxBench 26 finds 62% of 392 backend implementations have vulnerabilities or bugs (only 38% are both correct and secure). Across benchmarks, 40–60% of LLM code contains bugs. Wang and Zhu 27 propose Meta Prompt Testing: generate code variants with paraphrased prompts and detect bugs by checking if outputs differ. This achieves 75% TPR with 8.6% FPR on HumanEval. It requires test execution infrastructure and misses security vulnerabilities (SQL injection produces consistent outputs despite being exploitable) and quality issues that don’t affect outputs. AutoReview 3 uses three LLM agents (detector, locator, repairer) to find security bugs, improving F1 by 18.72% on ReposVul, but only checks security, not correctness or performance. We differ by: (1) proving mathematically why multi-agent works using submodularity of mutual information, (2) testing all 15 agent combinations to validate the architecture, and (3) achieving competitive TPR (76 2.2 Multi-Agent Systems for Software Engineering He et al. 12 survey 41 LLM-based multi-agent systems for software engineering, finding agent specialization (requirement engineer, developer, tester) as a common pattern. Systems like AgentCoder, CodeSIM, and CodeCoR use multiple agents to generate code collaboratively, but focus on producing code rather than checking it for bugs. MAGIS 2 uses 4 agents to solve GitHub issues, but measures solution quality (pass@k) rather than bug detection. No prior work applies multi-agent architectures to bug detection with mathematical justification. AutoReview’s 3-agent system only checks security (not correctness or performance), provides no theory for why multiple agents should work, doesn’t test alternative configurations, and doesn’t model vulnerability interactions. We fill this gap with a multi-agent verification system that covers correctness, security, performance, and style, proves why it works mathematically, and tests all 15 agent combinations. 2.3 Static Analysis and Vulnerability Detection Static analysis tools (SonarQube, Semgrep, CodeQL, Checkmarx) use pattern matching and dataflow analysis to find bugs without running code. Benchmarks 22 show 65% average

Read the complete text