Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023–2025)

Richard J. Young, Gregory D. Moody · 2026

Read Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023–2025) by Richard J. Young, Gregory D. Moody (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20351.txt.

Introduction

Young and Moody survey thirteen prompt sets used to test whether coding language models refuse harmful coding requests. They compare how each set was built, licensed and checked, and find three recurring gaps. No shared baseline exists for human labels, refusal rates measure unlike things across sets, and malware labels lack one shared scheme. The review urges pre-set rules, varied judges and a single label scheme for future sets.

Catalog record

CREATORRichard J. Young, Gregory D. Moody
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE119036
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20351

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023–2025) Richard J. Young, Gregory D. Moody Abstract The evaluation of large language model refusal on malicious-coding tasks now spans at least thirteen publicly released prompt corpora (AdvBench, the CyberSecEval family, RMCBench, RedCode, MCGMark, JailbreakBench, CySecBench, MalwareBench, CIRCLE, MOCHA, ASTRA, Scam2Prompt / Innoc2Scam-bench, and JAWS-Bench), each constructed under a different protocol, released under different licensing terms, and validated (or not) against different inter-rater reliability standards. Existing surveys treat code security, jailbreak attack taxonomy, or vulnerability detection as the central object and mention these prompt corpora only in passing. This paper reverses that framing: it treats the prompt datasets themselves as the unit of analysis. Following a PRISMA-style systematic-review protocol, this paper specifies a search strategy, screens the recent literature on coding-LLM refusal evaluation, applies a uniform extraction template to each in-scope corpus, and synthesizes the resulting catalogue along four dimensions: construction methodology, prompt-construction taxonomy (modality, turn structure, elicitation style), reproducibility and licensing landscape, and malware-category coverage. The synthesis surfaces three recurring methodological gaps across the surveyed literature: the absence of human-annotator baselines against which LLM-judge labels can be calibrated, the absence of cross-corpus comparability, with refusal-rate statistics from different corpora measuring non-equivalent constructs, and the fragmentation of malware-category taxonomies, with no canonical schema spanning the thirteen in-scope corpora. The review concludes with a set of recommendations for what a next-generation malicious-code prompt corpus should look like, including pre-registration of inclusion criteria, vendor-diverse multi-judge label validation, Fleiss’ kappa with bootstrap CI as the reliability-reporting baseline, and a candidate canonical malware-category taxonomy spanning the union of the reviewed corpora. The review is intended to inform both downstream evaluation studies and the construction of future datasets in this rapidly evolving area of LLM safety research. K eywords LLM Safety \cdot Malicious Code Generation \cdot Systematic Review \cdot PRISMA \cdot Prompt Datasets \cdot Refusal Evaluation \cdot Code Agents \cdot Coding LLMs \cdot Jailbreak Benchmarks \cdot Inter-Rater Reliability \cdot Multi-Judge Consensus \cdot Cybersecurity Benchmarks \cdot Red Teaming \cdot Responsible Disclosure \cdot AI Safety 1 Introduction Since mid-2023, at least thirteen publicly released prompt corpora have been introduced for evaluating malicious-code-generation refusal in coding LLMs: AdvBench [ 1 ] , the CyberSecEval family [ 2 , 3 , 4 ] , RMCBench [ 5 ] , RedCode [ 6 ] , MCGMark / MCGTest [ 7 ] , JailbreakBench [ 8 ] , CySecBench [ 9 ] , MalwareBench [ 10 ] , CIRCLE [ 11 ] , MOCHA [ 12 ] , ASTRA [ 13 ] , Scam2Prompt [ 14 ] , and JAWS-Bench [ 15 ] . Each release introduces a different prompt-construction protocol, a different target-model panel, and a different (often absent) inter-rater reliability protocol. Refusal-rate numbers reported across these corpora are not directly comparable: the corpora differ in prompt composition (counts span two orders of magnitude, from 100 to 12,662), malware-category taxonomy (no formal schema to thirteen functional families), elicitation modality (single-turn text-to-code, multi-turn decomposition, agent trajectories), and labeling protocol (human, automated oracle, LLM-judge, or unreported) — collectively, they measure different underlying constructs even when they report the same headline metric. Downstream researchers consolidating prompts across corpora are forced to reconstruct each corpus’s inclusion criteria from scratch, with no shared extraction template against which to audit selection decisions. The pace of corpus release has outstripped the methodological scaffolding the field needs to compare them. Prompt corpora deserve a dedicated review because they sit at the intersection of two distinct evaluation tasks that are routinely conflated in the surrounding literature. Refusal evaluation asks whether a model declines a request for executable malicious code; vulnerability evaluation asks whether a model, when complying with a benign request, generates code that contains exploitable bugs. The two tasks use different prompt corpora, produce different ground-truth labels (refusal versus presence of a Common Weakness Enumeration tag), and involve different threat models (deliberate misuse versus inadvertent insecure generation). Existing code-security surveys conflate the two or focus exclusively on vulnerability evaluation, leaving refusal-corpus methodology under-examined. The closest existing systematic survey, by Basic et al. (2024) [ 16 ] , explicitly scopes its analysis to vulnerability-introduction by LLM-generated code and treats refusal benchmarks only as adjacent context. No corresponding prior synthesis exists for the prompt corpora used to elicit malicious code in the first place. Five recent surveys touch on adjacent territory but each leaves the prompt corpora themselves on the periphery. The TechRxiv jailbreak survey [ 17 ] catalogues attack and defence methods at the technique level, not at the corpus level; cyber-coding subsets appear only as instances of broader attack taxonomies. The Basic and Giaretta SLR [ 16 ] on LLMs and code security restricts its unit of analysis to insecure-generation studies and treats refusal corpora as out of scope. The ACM Computing Surveys vulnerability-detection review [ 18 ] covers detection methods for code vulnerabilities, not prompts that elicit malicious authoring. The ACM TOSEM LLM-for-code survey [ 19 ] maps capability benchmarks (HumanEval-style, repository-level) but deliberately excludes safety-and-refusal evaluation. The recent agentic-security survey [ 20 ] considers autonomous attack agents but, as with the others, does not treat prompt datasets as the primary object of synthesis. Collectively, these adjacent surveys leave a clear gap: no prior survey takes the prompt corpora themselves as its unit of analysis. This paper closes that gap. We apply a PRISMA-style systematic-review protocol to the thirteen in-scope corpora and contribute four synthesis artefacts. (1) A cross-corpus methodology comparison (§ 5 , Table 1 ) extracts each corpus along ten axes (year, venue, prompt count, construction method, modality, malware-category taxonomy, target model panel, original inter-rater-reliability statistic, license, and release artefact) and surfaces the most striking pattern in current practice: zero of the thirteen corpora report a Fleiss- \kappa -style inter-rater reliability statistic on their refusal labels. (2) A prompt-construction taxonomy (§ 6 ) formalises three design dimensions, modality, turn structure, and elicitation style, that distinguish in-scope corpora from one another. (3) A reproducibility and licensing landscape (§ 7 , Table 2 ) catalogues distribution venues, licences, and access policies, and observes that no in-scope corpus uses gated access, registered-researcher review, or a documented takedown policy. (4) A coverage map across malware categories (§ 8 , Figure 4 ) consolidates the per-corpus taxonomies into a 22-category union ontology and identifies the categories materially under-represented in the current literature (firmware exploits, hardware and side-channel attacks, IoT and embedded attacks, supply-chain attacks). The contribution is synthesis, not new measurement: we apply a uniform extraction template to corpora as their authors released them and report what those releases collectively show. A uniform empirical re-evaluation of inter-rater reliability across these corpora using a common multi-judge protocol is identified as future work in § 9.5 . Our inclusion criterion is narrow by design. We retain corpora that score refusal of executable malicious-code-generation requests issued to coding LLMs or code-interpreter agents; we exclude vulnerability-detection corpora that score the security of LLM-generated code on benign requests, generic jailbreak datasets without coding focus, capability benchmarks that score correctness rather than refusal, and knowledge-elicitation benchmarks that probe security understanding rather than code-authoring intent (§ 3 ). Companion work by the present authors, the prompt-bank construction of [ 21 ]

Read the complete text