Automated Harmfulness Testing for Code Large Language Models

Honghao Tan, Haibo Wang, Diany Pressato, Yisen Xu, Shin Hwei Tan · 2025

Read Automated Harmfulness Testing for Code Large Language Models by Honghao Tan, Haibo Wang, Diany Pressato, Yisen Xu, Shin Hwei Tan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16740.txt.

Introduction

Tan and colleagues built in 2025 an automated way to test whether code language models can be tricked into producing harmful content. Developers regard harmful words in code artifacts as unethical, and exposure can harm their mental health. The team first cataloged program transformations that can smuggle harmful keywords into benign code. Their framework, CHT, uses those tricks to generate test prompts.

Testing four code models and a general model showed moderation is easily bypassed. The team then proposed checking content before generation. That two-phase approach improved moderation sharply in their tests, pointing to a practical defense.

Catalog record

CREATORHonghao Tan, Haibo Wang, Diany Pressato, Yisen Xu, Shin Hwei Tan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16740

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automated Harmfulness Testing for Code Large Language Models Honghao Tan, Haibo Wang, Diany Pressato, Yisen Xu, Shin Hwei Tan Abstract . Warning: Please note that this paper contains harmful content. This content is only for the evaluation and analysis of LLMs and does not imply any intention to promote criminal activities. To prevent the spread of harmful content, generative AI systems powered by Large Language Models (LLMs) are usually equipped with content moderation to identify and warn users about the potential risks of generating harm content. To evaluate the robustness of content moderation, several metamorphic testing techniques have been proposed to test content moderation software. However, these techniques mainly focus on content moderation for general users (e.g., textual content generation and image generation). Meanwhile, a recent study shows that developers consider using harmful keywords when naming software artifacts to be an unethical behavior. As the exposure to harmful content within software artifacts may have detrimental impact on mental health of software developers, it is important to investigate the effectiveness of content moderation for Code Large Language Models (Code LLMs) that are used by software developers. We conduct a preliminary study to investigate the set of program transformations that may be misused to introduce harmful content into auto-generated source code. Our study identifies 32 transformations that can be misused for harmful content generation by Code LLMs. Based on our study, we propose CHT , a novel coverage-guided harmfulness testing framework that automatically generates prompts using a set of prompt templates injected with a diverse set of harmful keywords that perform diverse types of transformations on a set of mined benign programs. Our framework performs output damage measurement to assess potential harm that can be introduced by the outputs produced by Code LLMs (i.e., natural language explanation/warning and modified code). Our evaluations of CHT on four Code LLMs and gpt-4o-mini (a general LLM) show that content moderation in LLM-based code generation systems is relatively easy to bypass where LLMs tends to generate harmful keywords embedded within program elements without providing any warning message to indicate the potential risks of the generated content (65.93% in our evaluation). To improve the robustness of content moderation in code-related tasks, we propose a two-phrase approach that first checks whether the prompt contains any harmful content before generating any output. Our evaluation shows that our proposed approach improves the content moderation of Code LLM by 483.76%. 1. Introduction Online anonymity and a lack of accountability can exacerbate the widespread of harmful content. Exposure to harmful content can have adverse affects on mental health and societal behavior. To prevent the spread of harmful content, content moderation has been an essential part of a generative AI systems powered by Large Language Models (LLMs). However, prior studies ( Wang et al., 2023b ; Wang et al., 2023a ) show that the content moderation currently supported by various systems is not robust enough to handle inputs that have been slightly perturbed. Meanwhile, a recent study ( Win et al., 2023 ) revealed that open-source developers consider using harmful keywords when naming software artifacts as a type of unethical behavior. For example, a developer complaint in a GitHub issue about using the word ‘‘genocide’’: ‘‘It was never a good or ethical name...It is...deliberate mass-murder...’’ 1 1 1 https://github.com/NetHack/NetHack/issues/359 . The exposure to the harmful content embedded within software artifacts (e.g., names of program elements) can incur negative effects on programmers, indicating that harmful content embedded within software programs should also be moderated. Recognizing the importance of ensuring the responsible use of LLMs, OpenAI have recruited a group of experts (also known as “red teaming”) to perform manual adversarial testing to mitigate the risks of producing potentially harmful content ( OpenAI, 2023 ; OpenAI, 2024 ) . Meanwhile, several metamorphic testing techniques have been proposed to detect the harmful content produced by LLMs ( Wang et al., 2023b ; Wang et al., 2023a ) . However, existing techniques mainly focus on harmful content generated by general-purpose systems (e.g., text generation ( Wang et al., 2023b ) , and image generation systems ( Wang et al., 2023a ) ). With the recent advancement of Code Large Large Models (Code LLMs), prior evaluations have demonstrated promising results in using Code LLMs for solving various software maintenance tasks (e.g., refactoring ( Shirafuji et al., 2023 ; AlOmar et al., 2024 ; Wang et al., 2024a ) , and automated program repair ( Zhang et al., 2024b ; Huang et al., 2023b ; Xu et al., 2024 ; Fan et al., 2023 ) ). However, there is a lack of systematic approaches that automatically generate tests to identify harmful content produced by LLMs for code. In this paper, we refer to a systematic testing approach to identify harmful contents generated by LLMs as harmfulness testing (Def 4 ). Challenges. We identify four key challenges of harmfulness testing for Code LLMs. (C1) Although source code can be considered as a specialized type of textual content, existing techniques designed for textual content systems are ineffective in identifying the harmful content that can be potentially produced by Code LLMs because they tend to ignore the unique characteristics of source code and are agnostics to the vocabulary of programming (which are usually names of program elements instead of plain texts in natural language ( Amit and Feitelson, 2022 ) ) . In fact, our evaluation in Section 4.3.1 shows that LLMs are significantly more effective in detecting harmful textual content embedded in code comment than those embedded in names of program elements (e.g., method names). (C2) Prior approaches mainly focus on several limited number of categories (e.g., hate speech, erotic and violent content ( Wang et al., 2023a ; Wang et al., 2023b ) ), neglecting important categories such as eating disorder, and misinformation. (C3) Although prior evaluation shows that Code LLMs may be more effective in performing certain types of program transformations ( Wang et al., 2023c ; Shirafuji et al., 2023 ; AlOmar et al., 2024 ) , existing techniques on textual content moderation does not evaluate on the diverse types of program transformations that are embedded within the prompts given to Code LLMs for program modifications. (C4) Instead of assessing the potential damage that may be incurred by a LLM, prior approaches ( Wang et al., 2023a ; Wang et al., 2023b ) mainly consider whether the content moderation has been bypassed. To address the challenge of evaluating diverse types of program transformations ( C3 ), we conducted a preliminary study of the set of refactoring types listed in the online catalog of refactoring ( Fowler, 2018 ) . The goal of our study is to identify a set of program transformations in which a harmful content can be injected into a given benign program. Based on our preliminary study, we obtained a total of 32 refactoring types which corresponds to 32 different prompt templates where each template contains a placeholder for injecting harmful content (usually injected via name of a program element). Inspired by traditional notions of code coverage, this paper defines harm category coverage which measures the proportion of harm categories being tested (Eq. 1 ). Our definition (Def. 1 ) encodes diverse categories of harmful

Read the complete text