Distilling Knowledge from Large Language Models: A Concept Bottleneck Model for Hate and Counter Speech Recognition
Roberto Labadie-Tamayo, Djordje Slijepčević, Xihui Chen, Adrian Jaques Böck, Andreas Babic, Liz Freimann, Christiane Atzmüller, Matthias Zeppelzauer · 2025
Read Distilling Knowledge from Large Language Models: A Concept Bottleneck Model for Hate and Counter Speech Recognition by Roberto Labadie-Tamayo, Djordje Slijepčević, Xihui Chen, Adrian Jaques Böck, Andreas Babic, Liz Freimann, Christiane Atzmüller, Matthias Zeppelzauer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08274.txt.
Introduction
Roberto Labadie-Tamayo and co-authors present a 2025 model that sorts hate speech and counter speech in an open way. Past detectors act as black boxes, so users cannot see why a post was flagged. Their model first turns each post into a set of plain adjectives, drawn with help from a large language model.
A light classifier then works from those words alone, which keeps each call easy to trace. Across five sets in many languages and platforms, the model beat most past scores. Mixing its word codes with standard deep codes added a small gain, which shows the two views add to each other.
Catalog record
| CREATOR | Roberto Labadie-Tamayo, Djordje Slijepčević, Xihui Chen, Adrian Jaques Böck, Andreas Babic, Liz Freimann, Christiane Atzmüller, Matthias Zeppelzauer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 111 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-08274
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Distilling Knowledge from Large Language Models: A Concept Bottleneck Model for Hate and Counter Speech Recognition Roberto Labadie-Tamayo, Djordje Slijepčević, Xihui Chen, Adrian Jaques Böck, Andreas Babic, Liz Freimann, Christiane Atzmüller, Matthias Zeppelzauer Abstract The rapid increase in hate speech on social media has exposed an unprecedented impact on society, making automated methods for detecting such content important. Unlike prior black-box models, we propose a novel transparent method for automated hate and counter speech recognition, i.e., “Speech Concept Bottleneck Model” (SCBM), using adjectives as human-interpretable bottleneck concepts. SCBM leverages large language models (LLMs) to map input texts to an abstract adjective-based representation, which is then sent to a light-weight classifier for downstream tasks. Across five benchmark datasets spanning multiple languages and platforms (e.g., Twitter, Reddit, YouTube), SCBM achieves an average macro-F1 score of 0.69 0.69 which outperforms the most recently reported results from the literature on four out of five datasets. Aside from high recognition accuracy, SCBM provides a high level of both local and global interpretability. Furthermore, fusing our adjective-based concept representation with transformer embeddings, leads to a 1.8% performance increase on average across all datasets, showing that the proposed representation captures complementary information. Our results demonstrate that adjective-based concept representations can serve as compact, interpretable, and effective encodings for hate and counter speech recognition. With adapted adjectives, our method can also be applied to other NLP tasks. † † footnotetext: This is a preprint of a manuscript accepted for publication in Information Processing & Management (Elsevier). The final version will be available via the journal’s website. K eywords Concept Bottleneck Model \cdot Natural Language Processing \cdot Large Language Model \cdot Online Hate \cdot Explainable Artificial Intelligence \cdot Explainability \cdot Interpretability \cdot Self-Learned Explanations 1 Introduction Online hate speech has rapidly increased on social media platforms, including even mainstream ones like X, Instagram, and TikTok, and has become a prevalent social problem ( Guillaume et al. 2022 ; Mathew et al. 2019a ; Masud et al. 2022 ; Masud et al. 2021 ) . This new form of online abuse negatively impacts a wide range of populations, not only causing emotional harm but also inciting social discrimination and even physical criminal violence ( Bilewicz and Soral 2020 ; Roy et al. 2023 ; Masud et al. 2022 ) . As a response, mechanisms such as manual content moderation have been implemented. However, the large volume of online discourse makes this approach resource-intensive, time-consuming, and hardly feasible in practice. Furthermore, prolonged exposure to harmful content can adversely affect the mental health of moderators ( Cobbe 2021 ) . Consequently, there is a pressing need for reliable automated recognition of toxic content ( Masud et al. 2022 ; Sun et al. 2024 ) . Community-driven initiatives like counter speech represent an important means to combat hate speech by providing responses to hateful content, such as offering facts, warning of potential consequences, and promoting empathy. Automatically identifying counter speech is important for increasing its visibility and thereby encouraging online civil courage. Analysing online hate and counter speech presents unique challenges in natural language processing (NLP) due to its nuanced and context-dependent nature ( Yu et al. 2022 ) . Recent advancements in NLP, particularly through the use of transformers and large language models (LLM), such as BERT ( Devlin et al. 2019 ; Zhou et al. 2023 ) and GPT ( Kalyan 2024 ) , have led to significant improvements in the accuracy of hate speech recognition ( Lee et al. 2022 ; Vidgen et al. 2020 ) . These models are trained on large-scale datasets, which enhances their ability to model context and capture nuanced language. As a result, they can be applied in a zero-shot setting or fine-tuned with annotated data for specific downstream tasks, further optimizing their performance for particular applications ( Li et al. 2024b ) . However, a key limitation of these models is their high complexity and black-box nature, which results in a lack of transparency. While there have been attempts to explain them through explainable artificial intelligence (XAI) approaches, these explanations are often questionable, revealing potential biases and spurious correlations ( Wilming et al. 2024 ) . Especially post-hoc explanations ( Dalvi et al. 2019 ; Durrani et al. 2020 ) , which are derived from pre-trained models, have been shown to sometimes provide misleading results ( Slack et al. 2020 ) . For this reason, more recent efforts have moved towards providing explanations rooted in the inherent mechanisms of LLMs, such as assessing attention values of tokens, exploring counterexamples, or generating free-text explanations ( Han et al. 2020 ; Das et al. 2022 ) . These approaches, however, primarily provide local explanations for individual input texts and fail to capture the global reasoning principles of the models. In contrast, concept bottleneck models (CBMs) ( Koh et al. 2020 ) , initially developed for vision tasks ( Oikarinen et al. 2023 ; Yang et al. 2023 ) , offer both global and local explanations. CBMs achieve this by defining a set of high-level, human-interpretable concepts and leveraging LLMs to encode input texts through these concepts. This encoding forms a bottleneck layer , which serves as input for downstream models, typically implemented as a stack of dense layers on top of it. While CBMs have been widely explored in computer vision tasks, their application in NLP remains widely unexplored, with only a handful of pioneering studies ( Sun et al. 2024 ; Ludan et al. 2024 ) . This gap is particularly evident for hate and counter speech recognition. Objectives and contribution. In this paper, we aim to develop an effective and efficient human interpretable approach based on CBMs for detecting hate and counter speech . The approach leverages the rich text modelling capabilities of LLMs while simultaneously ensuring human interpretability and repeatability in the resulting representation. To achieve this, we address three key challenges commonly encountered in CBM-based approaches: • Defining format and semantics of the bottleneck concepts. Existing CBM approaches in NLP typically extract concepts from LLM responses to zero-shot or in-context learning prompts ( Ludan et al. 2024 ) (e.g., for the prompt “How would you describe the film [‘film name’]?” extracted concepts might include terms like “acting”, “thrilling”, or “emotional”). While these methods work well for domains such as music, films, or restaurant reviews, they are less suited for hate speech and counter speech recognition, which typically involves more subtle nuances, contextual dependencies, as well as social and cultural considerations. Furthermore, the concepts extracted by these methods can represent abstract or unrelated semantic aspects, making them less interpretable for human users. • Computing concept representations of a given input text. While LLMs are frequently used for this task, their responses are conditioned by their inherent nature of randomness, even when presented with identical prompts. A desirable property for constructing semantic representations is repeatability and deterministic behaviour. • Mapping the computed bottleneck embeddings to predictions. Existing approaches typically apply a dense layer that utilises all extracted concepts ( Yang et al. 2023 ; Sun et al. 2024 ; Ludan et al. 2024 ) , which leads to over-complex explanations for a given prediction ( Miller 2019 ) hindering interpretability ( Ramaswamy et al.