DualGauge : Automated Joint Security–Functionality Benchmarking for Secure Code Generation

Abhijeet Pathak 1 , Suvadra Barua 1 , Dinesh Gudimetla 1 , Rupam Patir, Jiawei Guo, Hongxin Hu, Haipeng Cai · 2025

Read DualGauge : Automated Joint Security–Functionality Benchmarking for Secure Code Generation by Abhijeet Pathak 1 , Suvadra Barua 1 , Dinesh Gudimetla 1 , Rupam Patir, Jiawei Guo, Hongxin Hu, Haipeng Cai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20709.txt.

Introduction

Abhijeet Pathak and colleagues present in 2025 DualGauge, a benchmark that grades AI-written code on security and correctness at the same time. Scoring them apart hides a real tension: a model can patch a hole while breaking the program, or ship working code riddled with flaws. Studies cited by the authors find that 30 to 60 percent of code suggestions from modern language models contain exploitable flaws.

DualGauge pairs each coding task with checked test suites covering what the code must do and what it must never allow. An agent-based executor runs programs inside sandboxes, sealed environments, while a language-model evaluator judges behaviour against expectations. The team applied the framework to ten leading language models across thousands of test scenarios. The scores exposed gaps that separate testing had missed: models failed to be correct and secure together. The system and benchmark suite are open source, for reproducible, scalable evaluation.

Catalog record

CREATORAbhijeet Pathak 1 , Suvadra Barua 1 , Dinesh Gudimetla 1 , Rupam Patir, Jiawei Guo, Hongxin Hu, Haipeng Cai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-20709

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DualGauge : Automated Joint Security–Functionality Benchmarking for Secure Code Generation Abhijeet Pathak 1 , Suvadra Barua 1 , Dinesh Gudimetla 1 , Rupam Patir, Jiawei Guo, Hongxin Hu, Haipeng Cai Abstract Large language models (LLMs) and autonomous coding agents are increasingly used to generate software across a wide range of domains. Yet a core requirement remains unmet: ensuring that generated code is secure without compromising its functional correctness. Existing benchmarks and evaluations for secure code generation fall short—many measure only vulnerability reduction, disregard correctness preservation, or evaluate security and functionality on separate datasets, violating the fundamental need for simultaneous joint evaluation. We present DualGauge , the first fully automated benchmarking framework designed to rigorously evaluate the security and correctness of LLM-generated code in unison. Given the lack of datasets enabling joint evaluation of secure code generation, we also present DualGauge-Bench , a curated benchmark suite of diverse coding tasks, each paired with manually validated test suites for both security and functionality, designed for full coverage of specification requirements. At the core of DualGauge is an agentic program executor, which runs a program against given tests in sandboxed environments, and an LLM-based evaluator, which assesses both correctness and vulnerability behavior against expected outcomes. We rigorously evaluated and ensured the quality of DualGauge-Bench and the accuracy of DualGauge , and applied DualGauge to benchmarking ten leading LLMs on DualGauge-Bench across thousands of test scenarios. Our results reveal critical gaps in correct and secure code generation by these LLMs, for which our open-source system and datasets help accelerate progress via reproducible, scalable, and rigorous evaluation. † † footnotetext: 1 These authors contributed equally to this work. I Introduction Large language models (LLMs) and LLM-based coding agents have rapidly become integral to modern software development, offering developers the ability to generate code via natural language prompts [ 26 ] , dramatically accelerating prototyping and development cycles [ 44 ] . This marks a transformative shift toward AI-driven software engineering: code creation is increasingly a collaborative process between developers and AI models [ 17 , 32 , 38 ] . Also, developers naturally expect these models to generate code that accurately implements functionalities specified in the given prompts. Thus, functional correctness is often the first and foremost metric by which such models are judged in practice [ 6 , 15 , 4 , 21 ] , which is well justified because a code assistant that fails to meet this basic goal is of little practical use. However, there is growing concern about the security of AI-generated code. Studies show that AI assistants often produce insecure codes—e.g., 30–60% of code suggestions from modern LLMs contain exploitable vulnerabilities, such as cryptographic weaknesses and unsafe resource handling [ 30 , 25 , 36 ] . These insecure patterns are pervasive across programming languages and coding tasks. Other studies further show developers using AI coding assistants tend to write less secure code while paradoxically expressing higher confidence in its safety [ 31 , 3 , 33 ] . Alarmingly, these vulnerable AI-generated snippets are often accepted by developers and merged into real codebases, propagating new weaknesses into production software [ 9 ] . Broader analyses revealed systemic risks posed by insecure code generation, including feedback loops that could ingrain vulnerabilities into future models [ 16 , 1 ] . These findings make clear that, for LLM/agent-based code generation to be trustworthy, the generated code must be both functionally correct and secure [ 13 , 8 , 14 ] (i.e., dual assessment ). Thus, rigorous evaluation of AI code generators must assess these two objectives simultaneously , ensuring that generated programs fulfill their specifications without introducing vulnerabilities (i.e., joint evaluation ). Crucially, benchmarking is not just for assessment—it is a catalyst for progress. As LLMs/agents evolve, the ability to measure how well they generate secure and correct code becomes essential for charting and accelerating improvement. Without principled and holistic evaluation, it is difficult to understand how far we are from trustworthy, AI-assisted software development or to compare models meaningfully. Yet, most existing relevant evaluations measure model performance in silos, focusing on functional correctness [ 6 , 15 , 40 , 2 ] or on vulnerability mitigation in isolation [ 35 , 11 , 19 ] —but rarely both [ 8 ] , and almost never on the same set of tasks. Even state-of-the-art secure code generation techniques [ 13 , 14 , 41 , 27 , 24 ] evaluate model improvement in security on one dataset while assessing functionality preservation on another. Such disjoint evaluation is insufficient and can be misleading: a “secure” code fix that breaks required functionality is not a valid solution, just as a functionally correct program is unacceptable if it harbors security holes. By evaluating security and correctness separately, we risk overlooking solutions that trade one for the other, and we fail to measure the model’s ability to produce code that is simultaneously correct and resilient. Moreover, extant evaluation pipelines are imprecise or even manual. They often resort to static analysis tools (e.g., CodeQL [ 10 ] ) to detect vulnerabilities in generated code. This is not sufficient as such tools are constantly found to be highly inaccurate [ 23 , 18 , 22 ] , leading to noisy results. A few others additionally [ 29 ] or solely [ 39 ] relied on manual inspection, which is labor-intensive, error-prone, and not readily scalable, especially given the rapid pace at which new LLMs and coding agents are being released. Overcoming these obstacles demands a benchmark suite where each task has well-defined functionality requirements, along with comprehensive test cases to validate functional correctness and security, which is now critically lacking . The vast majority of benchmarks for secure code generation do not come with functional test cases [ 39 , 35 , 11 , 29 , 19 , 40 ] . CodeGuard+ [ 8 ] includes such tests, yet still no security tests . While static analyzers are very valuable means for validating against vulnerabilities in AI-generated code, they are insufficient as noted above. In contrast, runtime security tests directly validate whether exploitable behaviors manifest during execution, providing a more reliable and conclusive assessment of real-world vulnerabilities. Generally, vulnerabilities should be exploitable to really matter, which is hard to check based on static analysis. In this paper, we develop DualGauge , a fully automated benchmarking system for scalable joint evaluation of secure code generation. Given an pure-natural-language prompt as functional specification and access to a target model, the system (1) generates the model’s code output for the prompt, (2) executes it in a sandboxed environment against functional and security test suites, and (3) evaluates the execution results against both test suites to report test pass rates and joint correctness-security metrics, all autonomously . Achieving these capabilities face multiple challenges . First, for (2), the system needs to ( C1 ) resolve dependencies (e.g., libraries/packages) and set up runtime environments for (2)—which can be highly diverse and complex for various program. Second, for (3), the system needs to ( C2 ) check if a security test passes or not. However, the oracle for secure behaviors is hardly expressible via a readily comparable program output as for functional testing;

Read the complete text