Code as a Weapon: A Consensus-Labeled Prompt Bank for Measuring Coding-Model Compliance with Malicious-Code Requests
Richard J. Young, Gregory D. Moody · 2026
Read Code as a Weapon: A Consensus-Labeled Prompt Bank for Measuring Coding-Model Compliance with Malicious-Code Requests by Richard J. Young, Gregory D. Moody (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28734.txt.
Introduction
Richard J. Young and Gregory D. Moody build a consensus-labeled bank to measure coding compliance with malicious requests. A coding model that complies can return a weapon, such as a keylogger, ransomware, or exploit, so such models face a higher refusal bar than chat. Refusal benchmarks mix executable requests with harmful security knowledge and report rates over mixed sets. They classify eight corpora under a five-judge protocol, finding the code-versus-knowledge axis stable under an expanded pool and a fresh judge panel, proof of a real construct.
Catalog record
| CREATOR | Richard J. Young, Gregory D. Moody |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75334 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28734
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Code as a Weapon: A Consensus-Labeled Prompt Bank for Measuring Coding-Model Compliance with Malicious-Code Requests Richard J. Young, Gregory D. Moody Abstract A general-purpose language model that answers a harmful question returns text; a coding-specialised model that complies with the same intent can return a working weapon—a keylogger, a ransomware dropper, or an exploit payload that runs as written. This asymmetry in the severity of a single act of compliance implies coding models should clear a higher refusal bar than general-purpose chat assistants, not a lower one, yet the field cannot presently tell whether they do. Existing benchmarks of malicious-code refusal mix requests for executable software (which produce ready-to-run weapons) with requests for harmful security knowledge (which produce information a human must still operationalise); a single compliance-rate statistic computed over such a mixture cannot distinguish the two, and refusal numbers reported across heterogeneous corpora are not directly comparable. This paper releases an expanded consensus-labeled prompt bank that disambiguates these two request types and provides a construct-stable substrate for cross-corpus coding-model compliance measurement. Eight publicly released malicious-code prompt corpora (ASTRA, CySecBench, AdvBench / harmful_behaviors, JailbreakBench, MalwareBench, RedCode, RMCBench, and Scam2Prompt) are consolidated and classified under a five-judge consensus protocol (6,675 prompts × \times 5 judges = = 33,375 classification calls). The panel reaches Fleiss’ κ = 0.767 \kappa=0.767 [95 % CI: 0.755, 0.777] (“substantial”) on the 6,552-prompt subset where all five judges returned a valid label; 95.0 % of prompts draw at least four agreeing judges and 76.9 % are unanimous, and the panel reproduces the earlier four-corpus release [ 1 ] at Cohen’s κ = 0.952 \kappa=0.952 [0.942, 0.963] (“almost perfect”) on the 3,133 shared prompts. The released bank comprises 4,748 consensus-CODE prompts (executable malicious-code requests) and 1,923 consensus-KNOWLEDGE prompts (harmful security-knowledge requests). Two methodological findings are documented: the high-agreement low- κ \kappa paradox of Feinstein and Cicchetti (1990) affects four of the eight corpora and motivates a dual-statistic reporting convention; and one panel judge returns deterministic content-policy refusals on 1.84 % of prompts (115 of 520 in AdvBench / harmful_behaviors), a real-world property of safety-filtered judge availability that the 3-of-5 consensus rule absorbs. The result is a multi-judge, reliability-quantified substrate against which coding-model compliance with malicious-code requests can be measured under a stable construct definition. K eywords LLM Safety \cdot Malicious Code Generation \cdot Prompt Bank \cdot Consensus Labeling \cdot Inter-Rater Reliability \cdot Fleiss Kappa \cdot High Agreement Low Kappa Paradox \cdot Multi-Judge Panel \cdot Free-Tier API \cdot ASTRA \cdot Scam2Prompt 1 Introduction A general-purpose language model that answers a harmful question returns text; a coding-specialised model that complies with the same intent can return a working weapon—a keylogger, a ransomware dropper, a credential harvester, or an exploit payload for a known CVE that runs as written. This asymmetry in the severity of a single act of compliance implies that coding-specialised models, and the autonomous code agents built on them, should clear a higher refusal bar than general-purpose chat assistants, not a lower one; whether they do is a safety property of practical importance to anyone deploying a code assistant in production. Yet the published benchmark literature for measuring this property cannot presently settle the question, because it is uneven on a specific axis: prompts asking the model to produce executable software are routinely mixed in the same evaluation corpus with prompts asking the model to describe harmful security techniques in natural language . A model that refuses to write a keylogger and a model that refuses to explain how keyloggers work have different safety profiles and plausibly different alignment mechanisms, but a single “malicious-code refusal rate” computed over a mixed corpus cannot distinguish them, and cross-corpus comparisons are therefore at present not directly interpretable. An earlier release of a consensus-labeled benchmark [ 1 ] addressed this problem on four upstream corpora (RMCBench [ 2 ] , MalwareBench [ 3 ] , a filtered subset of CySecBench [ 4 ] , and AdvBench / harmful_behaviors [ 5 , 6 ] ) and 3,133 prompts, operationalising the distinction as a binary CODE-versus-KNOWLEDGE classification axis adjudicated by a five-judge LLM consensus panel under a 3-of-5 majority rule. That release produced a 1,554-prompt consensus-CODE bank (the primary released artefact) and a 388-prompt consensus-KNOWLEDGE comparison set. Two practical limitations remained. First, the four-corpus pool covered only a portion of the malicious-code prompt-corpus literature, and several public corpora released around or shortly before that cutoff (ASTRA, Scam2Prompt / Innoc2Scam-bench, JailbreakBench, and RedCode) were not included. Second, the original judge panel relied on five paid commercial APIs, limiting both the cost and replicability of any extension by downstream researchers. The present paper addresses both. The corpus pool is expanded from four upstream benchmarks to eight, adding ASTRA [ 7 ] (1,995 prompts; contributed by the ASTRA authors via direct contact), Scam2Prompt’s Innoc2Scam-bench [ 8 ] (1,377 indirect-elicitation prompts that disguise malicious intent as innocuous developer requests), JailbreakBench’s coding-relevant subset [ 9 ] (10 prompts), and RedCode [ 10 ] (160 prompts targeting code-interpreter agent surfaces). The judge panel is refreshed to five open-weight or free-tier-accessible models drawn from five vendor families (Nemotron-3-Super from NVIDIA, Qwen3-Coder-Next from Alibaba, DeepSeek-V4-Pro from DeepSeek, GPT-OSS-120B from OpenAI’s open-weights release, and GLM-5.1 from Zhipu AI), routed through OpenRouter and Ollama Cloud. The same binary CODE / KNOWLEDGE template, the same 3-of-5 majority consensus rule, the same Fleiss’ \kappa with bootstrap 95 % confidence interval methodology, and the same release schema as the four-corpus release are preserved, so on the four overlapping corpora the expanded bank continues the same protocol rather than constituting a methodologically separate artefact. Two empirical findings emerged from the v2 pass that warrant explicit documentation rather than burial in a supplementary table. First, the per-corpus Fleiss’ \kappa statistic degenerates to a near-zero or slightly negative value on the four most prevalence-skewed source corpora (ASTRA, MalwareBench, RedCode, RMCBench, each more than 99 % CODE under the 3-of-5 consensus rule). Consensus prevalence (the 3-of-5 majority outcome) is not the same as item-level observed agreement (the fraction of judge pairs agreeing on each item, summarised by the mean per-item P_{o} ); for these four corpora consensus is \geq 99\,\% CODE, but the mean P_{o} ranges from 0.850 (RedCode, only 100 of 160 prompts unanimous 5/5) to 0.989 (MalwareBench, see Table 3 ). The degenerate \kappa values are the classical “high agreement, low kappa” paradox of Feinstein and Cicchetti [ 11 ] : when one category occupies essentially the entire marginal distribution, the expected-agreement-by-chance term P_{e} approaches 1 and the \kappa denominator collapses, so \kappa becomes an uninformative summary of agreement on that corpus. Reporting only the overall heterogeneous-mixture \kappa would hide this. We therefore present per-corpus \kappa in two forms: with its bootstrap CI on the prevalence-mixed corpora where it is interpretable on the Landis–Koch scale (CySecBench, harmful_behaviors, Scam2Prompt), and as a flagged degenerate-marginal case with mean P_{o} as the substantive reliability summary on the four prevalence-skewed corpora. Second, one of the five judges (gpt-oss-120b, routed through OpenRouter’s OpenInference provider) returns HTTP 403 content-policy refusal on 123 of 6,675 prompts (1.84 %) that persist across retries, concentrated in AdvBench / harmful_behaviors (115 of 520, 22.1 % of that source) with the remaining 8 distributed across CySecBench (6), RMCBench (1), and