MMBERT: Scaled Mixture-of-Experts Multimodal BERT for Robust Chinese Hate Speech Detection under Cloaking Perturbations
Qiyao Xue, Yuchen Dou, Ryan Shi, Xiang Lorraine Li, Wei Gao · 2025
Read MMBERT: Scaled Mixture-of-Experts Multimodal BERT for Robust Chinese Hate Speech Detection under Cloaking Perturbations by Qiyao Xue, Yuchen Dou, Ryan Shi, Xiang Lorraine Li, Wei Gao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00760.txt.
Introduction
Qiyao Xue and co-authors built MMBERT in 2025, a detector for hate speech in Chinese social posts. Posters dodge text filters with cloaks: swapped characters, homophones, images, and voice clips that carry the slur off the page. Most past detectors read English text alone, so the cloak tricks slip past them.
MMBERT reads text, speech, and image parts together in one BERT-based frame. A mixture-of-experts design lets specialist parts weigh in per case, and the frame scales that team up. Tests centered on cloaked posts that break plain text tools. The work aims to keep Chinese platforms safer without sweeping up fair speech.
Catalog record
| CREATOR | Qiyao Xue, Yuchen Dou, Ryan Shi, Xiang Lorraine Li, Wei Gao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-00760
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MMBERT: Scaled Mixture-of-Experts Multimodal BERT for Robust Chinese Hate Speech Detection under Cloaking Perturbations Qiyao Xue, Yuchen Dou, Ryan Shi, Xiang Lorraine Li, Wei Gao Abstract Hate speech detection on Chinese social networks presents distinct challenges, particularly due to the widespread use of cloaking techniques designed to evade conventional text-based detection systems. Although large language models (LLMs) have recently improved hate speech detection capabilities, the majority of existing work has concentrated on English datasets, with limited attention given to multimodal strategies in the Chinese context. In this study, we propose MMBERT, a novel BERT-based multimodal framework that integrates textual, speech, and visual modalities through a Mixture-of-Experts (MoE) architecture. To address the instability associated with directly integrating MoE into BERT-based models, we develop a progressive three-stage training paradigm. MMBERT incorporates modality-specific experts, a shared self-attention mechanism, and a router-based expert allocation strategy to enhance robustness against adversarial perturbations. Empirical results in several Chinese hate speech datasets show that MMBERT significantly surpasses fine-tuned BERT-based encoder models, fine-tuned LLMs, and LLMs utilizing in-context learning approaches. Introduction Figure 1: Illustration of MMBERT model structure . Compared to traditional BERT-based model, it leverages the MoE architecture to scale and effectively handle multiple modalities. A three-stage progressive training strategy is designed to ensure stable training and prevent performance degradation. Hate speech poses a persistent threat to online communities, exacerbated by the anonymity and scale of digital platforms ( Dixon et al. 2018 ) . While automated hate speech detection has advanced significantly in recent years, most efforts remain concentrated on English, leaving other major languages like Chinese relatively under-resourced and under-protected ( Davidson et al. 2017 ; Davidson, Bhattacharya, and Weber 2019 ) . Some researchers have attempted to leverage LLMs for Chinese hate speech detection ( Chao et al. 2024 ; Sun et al. 2021 ; Zhou et al. 2023 ) . However, on Chinese social media platforms, many hate speech disseminators employ various cloaking perturbations to escape detection, making it challenging for existing models to identify such expressions accurately ( Xiao et al. 2024 ) . These subtle manipulations exploit the structural and phonological properties of the Chinese language, making detection especially difficult for text-only models. While LLMs have shown promise in content moderation, BERT-based architectures have consistently outperformed decoder-only LLMs in hate speech detection tasks, owing to their deep bidirectional encoding and strong capacity for fine-grained semantic understanding ( Benayas, Sicilia, and Mora-Cantallops 2024 ; Ghorbanpour, Dementieva, and Fraser 2025 ) . Their superior performance can be attributed to the ability to generate fine-grained contextualized representations, which are especially well-suited for classification tasks that require discerning subtle semantic distinctions and interpreting nuanced language—both of which are common in adversarial or implicitly encoded hate speech ( Liu, Wang, and Catlin 2024 ) . The architecture optimized for discriminative tasks enables more efficient and accurate detection of toxic content across various hate speech detection benchmarks ( Deng et al. 2022 ; Xiao et al. 2024 ) . To address the challenge of detecting cloaked hate speech in Chinese, we propose MMBERT, a novel multimodal BERT-based architecture that incorporates visual and speech modalities alongside text, depicted in Figure 1 . To enhance scalability and specialization, MMBERT integrates the MoE mechanism, enabling dynamic routing of representations to modality-specific experts. However, naïvely inserting MoE into BERT leads to severe training instability and degraded performance, particularly in the multimodal setting ( Zhang et al. 2021 ) . To overcome this, we introduce a progressive three-stage training strategy. In the first stage, we pretrain modality aligners using synthetic multimodal data to map visual and auditory inputs into the BERT language space. In the second stage, we train modality-specific experts and continue refining aligners using task-specific supervision. In the final stage, we jointly fine-tune the full MoE-augmented architecture on real multimodal hate speech data. This phased design ensures stable optimization and effective cross-modal integration. Our experiments across three benchmark Chinese hate speech datasets demonstrate that MMBERT achieves state-of-the-art performance, significantly outperforming both fine-tuned BERT-based baselines and LLMs with in-context learning. In particular, MMBERT shows superior robustness in detecting cloaked adversarial content, highlighting the value of multimodal modeling and progressive training for Chinese hate speech detection. We summarize the main contribution of this paper as follows: • We propose MMBERT , a novel multimodal BERT-based framework for Chinese hate speech detection that integrates textual, visual, and speech modalities through a Mixture-of-Experts (MoE) architecture, enhancing robustness against cloaking-based adversarial perturbations. • We design a progressive three-stage training strategy that first aligns multimodal inputs to the BERT language space, then specializes modality-specific experts, and finally fine-tunes the complete model. This approach ensures stable training and effective cross-modal representation learning. • We conduct extensive experiments on three benchmark datasets, comparing MMBERT against fine-tuned BERT-based and open-source LLM baselines and closed-source LLMs with in-context learning. Results demonstrate that MMBERT consistently achieves superior performance, particularly in detecting cloaking perturbed hate speech. Background and Motivation Cloaking Perturbations in Chinese Hate Speech Cloaking perturbations in Chinese online discourse represent a growing challenge for automated hate speech detection systems, as users employ various strategies to obfuscate offensive content while preserving its intended meaning ( Xiao et al. 2024 ; Xiao, Bouamor, and Zaghouani 2024 ) . It can be mainly categorized into several types: Deformation . As Chinese characters are logographic, their meanings can be altered by decomposing or reconfiguring individual components, often imparting specific emotional or ideological connotations ( Lan 2006 ) . For example, the character “ ” (meaning ‘silence’) comprises the radicals “ ” (meaning ‘black’) and “ ” (meaning ‘dog’), which in certain contexts have been used to convey derogatory implications toward the Black community. Homophonic Substitution . Words with similar pronunciations are frequently substituted to generate alternative semantics ( Tien, Carson, and Jiang 2021 ) . For instance, Chinese internet users often replace the character “ ” (meaning ‘full’) with “ ” (meaning ‘barbarian’), as both share a phonetic resemblance to ‘man’. Abbreviation . The contraction of sensitive terms enhances conciseness while maintaining semantic clarity ( Lan 2006 ) . A notable example is ‘txl’, where each letter corresponds to the pinyin initials of “ ” “ ” “ ”, collectively denoting ‘homosexuality’. Code-Mixing . To intensify expressive tone and circumvent automated content moderation, Chinese social media users frequently incorporate non-Chinese linguistic elements such as pinyin and emojis ( Li et al. 2020 ) . These code-mixed constructs not only obscure semantic intent from detection systems but also reinforce the emotive or derogatory force of the message. For instance, the term “ ” (meaning ‘ni brother’) phonetically approximates the English racial slur ‘n*gger’. Similarly, in the phrase “ ” (meaning ‘licking dog’), the addition of an emoji amplifies the pejorative undertone, characterizing individuals perceived as excessively submissive in relationship contexts—analogous to the English term ‘sycophant’. These perturbations exploit the unique structural and phonological characteristics of the Chinese language to conceal offensive intent ( Lu et al. 2023 ) . For instance, visually altering character radicals can introduce ideological connotations, while homophones and abbreviations obscure meanings through phonetic similarity or reduction. Code-mixing with pinyin or emojis further complicates semantic interpretation. Text-only models often fail to capture these manipulations due to their limited capacity to disambiguate subtle