HateXScore : A Metric Suite for Evaluating Reasoning Quality in Hate Speech Explanations

Yujia Hu, Roy Ka-Wei Lee · 2026

Read HateXScore : A Metric Suite for Evaluating Reasoning Quality in Hate Speech Explanations by Yujia Hu, Roy Ka-Wei Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13547.txt.

Introduction

Yujia Hu and Roy Ka-Wei Lee grade the why of hate speech flags, not just the flag. Hate speech tools are a core part of content care. Most tests score if a flag is right. Few score if the stated cause for a flag is sound. A right flag with a poor cause can still harm trust and appeals.

Their suite is called HateXScore. It holds four parts, each a mark of cause grade. Such parts can split a clear cause from a vague or off mark one. For care teams, cause grade is what a user sees on appeal. A suite to score it fills a blind spot in how such tools are judged.

Catalog record

CREATORYujia Hu, Roy Ka-Wei Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-13547

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HateXScore : A Metric Suite for Evaluating Reasoning Quality in Hate Speech Explanations Yujia Hu, Roy Ka-Wei Lee Abstract Hateful speech detection is a key component of content moderation, yet current evaluation frameworks rarely assess why a text is deemed hateful. We introduce HateXScore , a four-component metric suite designed to evaluate the reasoning quality of model explanations. It assesses (i) conclusion explicitness, (ii) faithfulness and causal grounding of quoted spans, (iii) protected group identification (policy-configurable), and (iv) logical consistency among these elements. Evaluated on six diverse hate speech datasets, HateXScore is intended as a diagnostic complement to reveal interpretability failures and annotation inconsistencies that are invisible to standard metrics like Accuracy or F1. Moreover, human evaluation shows strong agreement with HateXScore , validating it as a practical tool for trustworthy and transparent moderation. Disclaimer: This paper contains sensitive content that may be disturbing to some readers. 1 Introduction Hateful content remains a persistent challenge, with over 10% online posts estimated to contain hateful content 15 . In response, platforms are increasingly relying on automated approaches for content moderation 41 ; 28 ; 17 . Although classification accuracy has improved, recent regulatory frameworks such as the EU Digital Services Act 1 1 1 https://www.eu-digital-services-act.com/Digital_Services_Act_Article_17.html underscore the growing demand for transparency in automated decisions. In hate speech detection, such transparency is critical. Misclassifying harmful content risks perpetuating harm, while over-policing benign speech may suppress legitimate expression 14 . Even accurate models can fail silently by relying on spurious features or ignoring implicit hate. Moderation systems must therefore justify not just what decision was made, but why 42 . Existing explanation evaluation methods have assessed whether highlighted tokens align with the predicted label 3 or measure overlap with human-annotated rationales 27 ; 14 . However, these approaches do not ask whether the explanation quotes the harmful span, whether removing it changes the prediction, or whether the explanation correctly identifies the targeted group. These gaps are further compounded by the subjectivity of hate speech annotation 35 . A model may produce a faithful, well-reasoned explanation that challenges an ambiguous or flawed label, but still be penalized by label-centric metrics. A model that highlights a slur and explains its harm may offer a more faithful interpretation than the label allows. This disconnect highlights the need for an evaluation framework that accounts not just for prediction correctness, but for the underlying reasoning. This issue is amplified when annotation disagreement is present, where LLMs may appear confident despite contested labels 24 . Figure 1: An example of a hate speech detection scored using HateXScore . To address these challenges, we introduce HateXScore, a metric suite that evaluates explanation quality across four dimensions: explicitness of the conclusion, faithfulness of quoted spans with causal impact, identification of protected groups under configurable policies, and internal consistency among these elements. Figure 1 shows an application of HateXScore to an explanation generated by a large language model (LLM) for a sample from the HateXplain dataset 27 . The model’s prediction is deemed incorrect only because it disagrees with the dataset label. In contrast, HateXScore assigns a high score to the generated explanation, indicating that the model provides a coherent justification for its decision. In such cases, human moderators may concur with the model and view the gold label as debatable. This example highlights HateXScore as a diagnostic signal that can surface well-justified explanations under prediction–label mismatch, thereby helping identify potentially unreliable annotations in hate speech datasets. Accordingly, while Accuracy and F1 quantify agreement with dataset labels, HateXScore evaluates the quality of the model’s justification for its own prediction. We summarize our contribution as follows: (i) We propose HateXScore , a novel suite of metrics to evaluate the quality of model explanations for hate speech detection. (ii) We conducted experiments to evaluate the explanations generated by models on six hate speech datasets, spanning English, Chinese, and Korean, enabling multilingual analysis. Our experiments demonstrated that HateXScore uncovers explanation failures and annotation inconsistencies that standard metrics overlook. (iii) We validate HateXScore through human evaluation and case studies, showing its alignment with human judgment and its capacity to surface hidden annotation disagreements. 2 Related work 2.1 Hate Speech Detection Hate speech detection has received increasing attention due to the growth of content posted by users on social media 11 ; 36 ; 45 ; 22 . Early systems employed lexicons and simple classifiers, such as SVMs or logistic regression 37 , which could detect explicit slurs but struggled with implicit or contextual hate 40 . Recent advances use deep models like BERT 28 and RoBERTa 23 , improving accuracy by capturing semantic and syntactic context 1 ; 16 . Work has also extended to multilingual 10 ; 31 ; 6 ; 43 ; 29 , multimodal 21 ; 20 and fine-grained hate detection (e.g., racism, sexism, homophobia), reflecting the field’s growing sensitivity to hate’s varied forms. However, most work still relies on Accuracy or F1, giving little insight into model reasoning, motivating explanation-based evaluation. 2.2 Hate Speech Explanation Efforts to evaluate explanations in hate speech models remain limited. Task fidelity approaches test alignment between highlighted tokens and predicted labels 3 , while overlap metrics compare model spans with human rationales 27 ; 14 . Data imbalance has also motivated augmentation approaches 8 . While informative, these methods ignore whether models quote the harmful span, link it causally to predictions, or identify the targeted group. Subjectivity in annotations further complicates the evaluation. In HateXplain 27 , reclaimed slurs may be labeled non-hateful by some annotators and hateful by others. In ImplicitHate 14 , indirect expressions lead to inconsistent labels. In such cases, a model may provide a well-reasoned explanation that conflicts with the majority label, something conventional metrics are ill-equipped to capture. Existing explanation frameworks also treat decision correctness and explanation quality separately 32 ; 30 ; 9 . Recent work also shows that linguistically structured and interpretable explanations can speed up human moderation without sacrificing accuracy 7 , reinforcing the importance of interpretable reasoning frameworks such as HateXScore . Approaches such as causal masking 12 test the importance of the explanation, but are rarely adapted to hate speech, and rarely account for identity group targeting, a core aspect of hateful intent. Recent work highlights reasoning-based explanations and LLMs in hate-speech detection. 34 explore causal reasoning for hate speech classification, improving interpretability but without focusing on policy alignment or protected group identification. 44 propose HARE, which emphasizes stepwise reasoning, sharing HateXScore’s focus on explanation structure but lacking cross-verification of explanations with target groups or measurement of overall consistency. 42 evaluate GPT-generated explanations with human judgments but do not capture the multi-faceted causal and group recognition aspects that HateXScore quantifies. 5 offer necessity and sufficiency measures for causal token importance similar to HateXScore’s Quotation Faithfulness sub-metric, but without integrating policy-aware group detection or coherence checks. HateModerate 46 aligns models with moderation policies but does not analyze explanation structure. HateXScore uniquely integrates causal validation, policy-aware group recognition, consistency checks, and explicit conclusions in one interpretable suite. This integration allows a comprehensive, policy-aware assessment of hate speech explanations that goes beyond prior approaches. 3 Metrics Design HateXScore evaluates the quality of hate speech explanations across four complementary components. Each

Read the complete text