Not All Pretraining are Created Equal: Threshold Tuning and Class Weighting for Imbalanced Polarization Tasks in Low-Resource Settings
Abass Oguntade · 2026
Read Not All Pretraining are Created Equal: Threshold Tuning and Class Weighting for Imbalanced Polarization Tasks in Low-Resource Settings by Abass Oguntade (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23534.txt.
Introduction
Oguntade reports a system entry for the SemEval-2025 polarization shared task. The job spans spotting polarization in social posts and tagging its targets and forms. Texts come in English and Swahili, with harsh class imbalance. His systems mix multilingual transformers like mDeBERTa and AfriBERTa. Class-weighted losses and per-label thresholds fight the imbalance. The best setup leads on binary detection and stays competitive on the multi-label tasks. Error checks blame hidden polarization, code-switching, and heated talk that is not true polarization.
Catalog record
| CREATOR | Abass Oguntade |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25786 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.23534
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Not All Pretraining are Created Equal: Threshold Tuning and Class Weighting for Imbalanced Polarization Tasks in Low-Resource Settings Abass Oguntade Abstract This paper describes my submission to the Polarization Shared Task at SemEval-2025, which addresses polarization detection and classification in social media text. I develop Transformer-based systems for English and Swahili across three subtasks: binary polarization detection, multi-label target type classification, and multi-label manifestation identification. The approach leverages multilingual and African language-specialized models (mDeBERTa-v3-base, SwahBERT, AfriBERTa-large), class-weighted loss functions, iterative stratified data splitting, and per-label threshold tuning to handle severe class imbalance. The best configuration, mDeBERTa-v3-base, achieves 0.8032 macro-F1 on validation for binary detection, with competitive performance on multi-label tasks (up to 0.556 macro-F1). Error analysis reveals persistent challenges with implicit polarization, code-switching, and distinguishing heated political discourse from genuine polarization. 1 Introduction Attitude polarization, the sharp division of opinions into opposing groups characterized by hostility toward out-groups and blind in-group support, has become a defining feature of contemporary social media discourse . This phenomenon manifests in contexts such as elections, conflicts, protests, and political debates, where content may stereotype, vilify, dehumanize, or invalidate individuals based on their identities or affiliations. Detecting and understanding polarized content across linguistic and cultural contexts is critical for mitigating online harm and fostering constructive dialogue 10 ; 16 . This shared task evaluates systems on polarization detection in English and Swahili social media text across three progressively specialized subtasks: (1) binary classification of polarized versus non-polarized content, (2) multi-label identification of polarization target types (political, racial/ethnic, religious, gender/sexual, other), and (3) multi-label detection of polarization manifestations (stereotype, vilification, dehumanization, extreme language, lack of empathy, invalidation) . My main strategy combines systematic evaluation of six Transformer architectures: three multilingual models (TwHIN-BERT 29 , DistilBERT-multilingual 20 , mDeBERTa-v3-base 11 ) and three African language-specialized models (SwahBERT 14 , AfriBERTa-large 15 , AfroXLMR-large 1 ) with class-weighted loss functions and per-label threshold optimization for multi-label tasks. I employ iterative stratified splitting 21 ; 25 to preserve multi-label distributions, BCEWithLogitsLoss with per-label positive class weights to emphasize rare categories, and two-stage threshold search (coarse global + fine per-label) 19 to maximize macro-F1 across severely imbalanced labels. The key findings reveal that architectural selection matters more than language-specific pretraining: mDeBERTa-v3-base outperforms Swahili-specialized models by 10–15 percentage points despite having no explicit Swahili focus. Threshold tuning provides substantial gains (20+ points macro-F1) for multi-label classification under extreme imbalance 4 . However, the system struggles with implicit polarization cues, code-switching between English and Swahili, and false positives on heated but non-polarized political rhetoric. On official test sets in Codabench 1 1 1 https://www.codabench.org/ , I achieve 0.815 macro-F1 (English) and 0.785 macro-F1 (Swahili) for binary detection, with final multi-label scores of 0.464 (English) and 0.556 (Swahili) for manifestation identification. The code are available at [https://github.com/HayBeeCoder/polarization-detection]. 2 Background The Polarization Shared Task defines polarization as content displaying negative attitudes toward out-groups while showing solidarity with in-groups, encompassing stereotyping, vilification, dehumanization, intolerance, and divisive rhetoric. The task provides labeled social media datasets for English (3,222 training instances) and Swahili (6,991 training instances) across three hierarchical subtasks. 2.1 Task Description Subtask 1: Binary Polarization Detection. Given a social media post, predict whether it contains polarized content (label=1) or not (label=0). For example, the post “These people are destroying our country with their backwards beliefs” would be labeled as polarized (1), while “I disagree with this policy” would be non-polarized (0). English exhibits moderate class imbalance (64% non-polarized, 36% polarized, 1.78:1 ratio), while Swahili shows near-perfect balance (50% each class). Subtask 2: Multi-Label Target Type Classification. For polarized texts, predict which of five target categories apply: political , racial/ethnic , religious , gender/sexual , or other . Multiple labels may apply simultaneously. For example, “Muslim immigrants are ruining European values” targets both racial/ethnic and religious categories. The English dataset shows extreme imbalance: political dominates with 1,150 instances (30.3%), while gender/sexual appears in only 72 instances (1.9%). Subtask 3: Multi-Label Manifestation Identification. Predict how polarization manifests across six categories: stereotype , vilification , dehumanization , extreme_language , lack_of_empathy , or invalidation . For instance, “Those animals don’t deserve to be treated like humans” exhibits dehumanization , vilification , and extreme_language . Vilification dominates with 3,741 instances, while dehumanization appears least frequently (1,284 instances). The organizers provide official train/dev splits and evaluate using macro-averaged F1 as the primary metric to weight all labels equally despite severe imbalance. I participate in all three subtasks for both English and Swahili. 2.2 Related Work Hate speech and polarization detection have been extensively studied for high-resource languages 8 ; 27 , but African languages remain underexplored 5 . Recent work on multilingual hate speech demonstrates that cross-lingual transfer from English to low-resource languages often underperforms language-specific models when sufficient target-language data exists 2 . African language-focused models like SwahBERT 14 and AfriBERTa 15 aim to address this gap through targeted pretraining on 11 African languages including Swahili, Hausa, and Amharic. This work contributes systematic comparisons between multilingual and African language-specialized architectures on polarization detection, revealing counterintuitive results where general multilingual models outperform specialized alternatives. 3 Data Analysis I conduct exploratory analysis to understand dataset characteristics and inform modeling decisions. 3.1 Dataset Composition Considering the separate English and Swahili as a whole, the complete dataset used comprises 10,213 instances: Swahili (6,991, 68.5%) and English (3,222, 31.5%). Figure 1 and Figure 2 show the class distributions for Subtask 1. English exhibits a 1.78:1 imbalance favoring non-polarized content, while Swahili shows a near-perfect balance. Figure 1: English dataset polarization class distribution for Subtask 1. The dataset shows moderate imbalance with 64% non-polarized instances. Figure 2: Swahili dataset polarization class distribution for Subtask 1. Nearly balanced with 50% representation in each class. 3.2 Vocabulary Patterns Figures 3 and 4 show word clouds revealing distinct topical focuses. Swahili content features vulgar terms ( kutombwa , hawa , kama , wewe ), while English centers on political discourse ( trump , ukraine , gaza , state ). This divergence suggests that cross-lingual transfer must account for culturally specific linguistic markers 7 . Figure 3: Word cloud for English dataset showing dominance of political terms (trump, ukraine, gaza) and geopolitical discourse markers. Figure 4: Word cloud for Swahili dataset revealing frequent vulgar and offensive terms indicating different polarization expression patterns. 3.3 Multi-Label Imbalance Figure 5 shows Subtask 2 label distribution for polarized texts in the English dataset. Political content dominates (35.7% English), creating a 16.23:1 imbalance ratio with the rarest gender/sexual label (2.2%). Notably, approximately 47% of instances contain no active labels (i.e., all-zero class). This extreme imbalance motivates the use of class-weighted loss functions and threshold tuning strategies 6 ; 13 . Figure 5: Subtask 2 label distribution across five target types showing severe imbalance. Political dominates while gender/sexual remains extremely rare. Figure 6 illustrates the class balance per label. Across all labels, negative (non-hate) examples vastly outnumber positive instances, with the 2,047 all-zero rows further amplifying skew toward non-hate content. Figure 6: Class distribution per label for Subtask 2 showing positive vs negative instances. All labels exhibit severe imbalance favoring negative class. 3.4 Label Correlations Figure 7 shows correlation analysis revealing mostly near-zero correlations between label pairs in the English dataset. The notable exception is a