Navigating the Sea of LLM Evaluation: Investigating Bias in Toxicity Benchmarks
Regina Gugg, Selina Niederländer, Andreas Stöckl, Martin Flechl · 2026
Read Navigating the Sea of LLM Evaluation: Investigating Bias in Toxicity Benchmarks by Regina Gugg, Selina Niederländer, Andreas Stöckl, Martin Flechl (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10639.txt.
Introduction
Regina Gugg and three colleagues study bias in toxic benchmarks for language models. The rapid rise of language models highlights the risks of using them safely. It reveals a gap in sound testing of toxic benchmarks. As firms rely on benchmarks to approve models for apps and auto review, hidden testing tilts could lead to bad launches.
Their 2026 paper studies the strength of current benchmark setups. The authors study how to measure missed built-in tilts, such as those tied to model choice, metrics, and task types. Tests uncover gaps in benchmark conduct when testing setups are altered. Shifting from text writing to digests makes benchmarks flag content as harmful more often. Certain benchmarks fail to maintain steady conduct when the input domain is changed. The authors also observe model swings, showing a need for more robust and full safety evaluation frameworks.
Catalog record
| CREATOR | Regina Gugg, Selina Niederländer, Andreas Stöckl, Martin Flechl |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10639
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Navigating the Sea of LLM Evaluation: Investigating Bias in Toxicity Benchmarks Regina Gugg, Selina Niederländer, Andreas Stöckl, Martin Flechl Abstract The rapid adoption of LLMs in both research and industry highlights the challenges of deploying them safely and reveals a gap in the systematic evaluation of toxicity benchmarks. As organizations increasingly rely on these benchmarks to certify models for customer-facing applications and automated moderation, unrecognized evaluation biases could lead to the deployment of vulnerable or unsafe systems. This work investigates the robustness of established benchmarking setups and examines how to measure currently neglected intrinsic biases, such as those related to model choice, metrics, and task types. Our experiments uncover significant discrepancies in benchmark behaviors when evaluation setups are altered. Specifically, shifting the task from text completion to summarization increases the tendency of benchmarks to flag content as harmful. Additionally, certain benchmarks fail to maintain consistent behavior when the input data domain is changed. Furthermore, we observe model-specific instabilities, demonstrating a clear need for more robust and comprehensive safety evaluation frameworks. 1 Introduction The development of Large Language Models (LLMs) is currently one of the fastest-growing fields in artificial intelligence. This rapid progress necessitates a crucial requirement for safely applying these models to real-world problems: benchmarks must evaluate alignment not only for task performance but also for safety against harmful content. When safety benchmarks lack robustness and fail to generalize across different contexts or tasks, they provide a false sense of security. Consequently, real-world enterprise deployments (such as summarization tools, customer support bots, and autonomous agentic workflows) are exposed to significant risks of generating or amplifying toxic content. Furthermore, because toxicity definitions are multifaceted and benchmark validity degrades over time due to cultural shifts and data contamination, new benchmarks are released with increasing frequency. This rapid proliferation makes it difficult to contextualize their results. Currently, few studies systematically evaluate multiple safety benchmarks and models simultaneously, or investigate whether widespread benchmark designs are limited by their creators’ specific setups. To address this gap, this work provides a systematic investigation into the robustness of LLM safety benchmark setups. 2 Related Work To clearly outline the existing landscape of benchmarks for LLMs, we use the term performance benchmarks to refer to evaluations that measure capabilities on explicit tasks, such as answering mathematical questions or instructing coding agents. This definition allows us to clearly separate these evaluations from our main focus: safety and toxicity benchmarks, which specialize in measuring the tendency to output harmful content. While there is inherent overlap, for instance, harmful output might be partially captured when a model fails to answer a prompt correctly, our focus remains on specialized safety benchmarks. Prompts in standard performance benchmarks are rarely designed to test safety limits, nor do they distinguish task-fulfilling but toxic responses from non-harmful counterparts. 2.1 Landscape of LLM Safety Benchmarks Despite the prevention of harmful content being a universal alignment target, the number of safety benchmarks remains comparatively low. Cross-evaluation platforms and comparative studies are much rarer for safety than for performance benchmarks, which are increasingly investigated and organized on platforms such as Chatbot Arena and the Hugging Face Open LLM Leaderboard [ 4 ] . Furthermore, terms like toxicity and harmful content lack consistent definitions, even in well-established literature [ 18 , 31 ] . Taxonomies often attempt to separate toxicity and social bias into different categories, but this is not always possible since the underlying concerns are deeply linked. Consequently, while numerous benchmarks evaluate different facets of LLM safety, no unified framework consolidates them by target dimension and interaction format. Table 1 provides a compact overview of prominent alignment benchmarks used as a starting point for our work. A central observation and a key motivation for our work is that most safety benchmarks are built around a single, predefined interaction format. A model may reliably refuse harmful instructions in a direct question-answering setup, yet exhibit vulnerabilities when the same harmful content is embedded in a summarization or text-completion task. Existing benchmarks tend to evaluate isolated safety facets without capturing how alignment generalizes across tasks and domains. Benchmark Year Safety Dimension LLM Task Type Evaluation Method Winobias [ 35 ] 2018 Gender Bias Coreference Resolution Automated Metrics CrowS-Pairs [ 22 ] 2020 Social Bias Sentence Completion Automated Metrics RealToxicityPrompts [ 7 ] 2020 Toxicity Text Completion Perspective API StereoSet [ 21 ] 2020 Stereotypical Bias Association Tests Automated Metrics BBQ [ 23 ] 2021 Bias QA Accuracy, F1 BOLD [ 6 ] 2021 Bias Text Generation Pretrained Classifiers HateXplain [ 19 ] 2021 Hate Speech Text Classification Classification Metrics TruthfulQA [ 16 ] 2021 Factuality, Truthfulness QA, Fact Checking Human, LLM-as-a-Judge HarmfulQ [ 25 ] 2022 Stereotypical Bias, Toxicity QA, Instr. Following Human ToxiGen [ 9 ] 2022 Implicit Toxicity, Hate Speech Text Generation, Classification HateBERT [ 1 ] , ToxDectRoBERTa [ 36 ] AdvBench [ 37 ] 2023 Adversarial Robustness Text Generation, Instr. Following Automated Metrics DoNotAnswer [ 30 ] 2023 Harmfulness QA, Instr. Following Human, LLM-as-a-Judge MaliciousInstruct [ 11 ] 2023 Jailbreak Robustness Instruction Following Automated Classifier SafetyBench [ 34 ] 2023 Comprehensive Safety Multiple-Choice QA Accuracy SimpleSafetyTests [ 29 ] 2023 Harmfulness QA, Instr. Following Human, Automated Classifiers ToxicChat [ 17 ] 2023 Toxicity Toxicity Classification Classification Metrics XSTest [ 24 ] 2023 Over-refusal QA, Instr. Following Human, Rule-based, LLM-as-a-Judge BeHonest [ 3 ] 2024 Honesty, Misinformation QA Rule-based, LLM-as-a-Judge HarmBench [ 20 ] 2024 Red Teaming, Robust Refusal Instruction Following LLM-as-a-Judge JailbreakBench [ 2 ] 2024 Jailbreak Robustness Instruction Following Rule-based, LLM-as-a-Judge SALAD-Bench [ 14 ] 2024 Comprehensive Safety QA, Instr. Following LLM-as-a-Judge StrongReject [ 26 ] 2024 Jailbreak Robustness QA, Instr. Following LLM-as-a-Judge Sorry-Bench [ 32 ] 2025 Hate Speech, Safety QA, Instr. Following LLM-as-a-Judge Table 1: Popular LLM alignment and safety benchmarks, categorized by safety dimension, LLM task type, and evaluation method. 2.2 Existing Investigations into Benchmark Reliability While performance benchmarks receive increasing scrutiny regarding the reliability of their reported metrics and whether the reported values are reliable for decision-making, there is limited investigation into the robustness of safety benchmarks. Adjacent topics, such as corpus contamination and jailbreaking, have become more prevalent and represent promising developments. However, toxicity evaluations are rarely explored deeply within these contexts. Furthermore, investigations into the safety of LLM outputs have only recently emerged. Most of these studies focus on the limitations of LLM-as-a-judge evaluations, which are black-box systems that often lack grounding [ 27 ] . Currently, few large-scale initiatives drive a holistic view of safety evaluations. HELM [ 15 ] allows for model comparisons across benchmarks, but it primarily relies on LLM judges and does not apply task or domain augmentations to the original setups. Meanwhile, Dynabench [ 13 ] offers a platform for collaborative evaluation systems but struggles significantly with widespread adoption. 2.3 Model Selection & Benchmark Suite For our experiments, we use four established benchmarks that cover key safety dimensions: toxicity (RealToxicityPrompts [ 7 ] ; referred to as RTP), adversarial robustness (HarmBench [ 20 ] ), implicit hate speech (ToxiGen [ 9 ] ), and refusal behavior (DoNotAnswer [ 30 ] ; referred to as DNA). To keep the evaluation both rigorous and computationally manageable, we slightly adjust the original setups by reducing sample sizes, updating