To Bias or Not to Bias: Detecting bias in News with bias-detector
Himel Ghosh, Ahmed Mosharafa, Georg Groh · 2025
Read To Bias or Not to Bias: Detecting bias in News with bias-detector by Himel Ghosh, Ahmed Mosharafa, Georg Groh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13010.txt.
Introduction
Himel Ghosh, Ahmed Mosharafa, and Georg Groh build a tool that flags bias in news writing sentence by sentence. Judging bias is tricky because it is partly a matter of opinion, and expert-labeled data is scarce. The team fine-tuned a language model on an expert-labeled dataset of news sentences.
Careful statistical tests show their model beats a strong baseline that had extra training on news text. A look inside the model shows it pays attention to words that carry the slant in context, rather than firing on any charged political term. The authors pair the detector with an existing tool that names the kind of bias, forming a fuller pipeline. They note the sentence-level view and small data remain limits, and sketch next steps.
Catalog record
| CREATOR | Himel Ghosh, Ahmed Mosharafa, Georg Groh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13010
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
To Bias or Not to Bias: Detecting bias in News with bias-detector Himel Ghosh, Ahmed Mosharafa, Georg Groh Abstract Media bias detection is a critical task in ensuring fair and balanced information dissemination, yet it remains challenging due to the subjectivity of bias and the scarcity of high-quality annotated data. In this work, we perform sentence-level bias classification by fine-tuning a RoBERTa-based model on the expert-annotated BABE dataset. Using McNemar’s test and the 5×2 cross-validation paired t-test, we show statistically significant improvements in performance when comparing our model to a domain-adaptively pre-trained DA-RoBERTa baseline. Furthermore, attention-based analysis shows that our model avoids common pitfalls like oversensitivity to politically charged terms and instead attends more meaningfully to contextually relevant tokens. For a comprehensive examination of media bias, we present a pipeline that combines our model with an already-existing bias-type classifier. Our method exhibits good generalization and interpretability, despite being constrained by sentence-level analysis and dataset size because of a lack of larger and more advanced bias corpora. We talk about context-aware modeling, bias neutralization, and advanced bias type classification as potential future directions. Our findings contribute to building more robust, explainable, and socially responsible NLP systems for media bias detection. Keywords: Bias Detection, Media Bias, NLP, Interpretability, Sentence Classification, transfer learning 1 Introduction Journalism is supposed to be neutral and free from biases, but that is often not followed. Cambridge English dictionary defines bias as "the action of supporting or opposing a particular person or thing in an unfair way, because of allowing personal opinions to influence your judgment". Intentional or unconscious, media bias is the unfair treatment of particular individuals and ideas as well as the reporting of certain biased viewpoints Morrisette et al. which can significantly influence public opinion, skew political and democratic discourses. For a free and fair journalism and news reporting, it is imperative to identify the bias and eventually mitigate them. This paper addresses the ever-increasing need for bias identification models and demonstrates their statistical significance compared to the other available models . Here we introduce a neural transformer-based fine-tuned bias detection model trained on Bias Annotations by Experts (BABE) dataset by Spinde et al. 2021b for robust bias detection available on Hugging Face . Furthermore, we present a pipeline (See Fig. 1 ) with both bias detection and bias classification for bias analysis on sentences. These findings will offer a framework for future research in the media bias detection. Figure 1: Bias detection and Classification scheme. 2 Related Work At the nexus of political science, psychology, and natural language processing, media bias detection has emerged as a key field of study. Historically, research in this field has used datasets like AllSides, MediaBias/FactCheck, and Media Frames Corpus to identify ideological bias at the source or article level. Lexicon-based heuristics or shallow machine learning models that used manually constructed features like sentiment polarity, assertive verbs, or hedge words were frequently used in early computational approaches Recasens et al. 2013 ; Hube and Fetahu 2018 . However, due to their low inter-annotator agreement and dependence on small datasets, these approaches showed limitations in terms of scalability and generalization. For instance, Lim et al. 2020 ; Färber et al. 2020 used crowdsourcing to create datasets but found low interrater reliability (e.g., Krippendorff’s \alpha = 0.0), indicating challenges in capturing the intrinsically subjective nature of bias. To address these challenges, Spinde et al. introduced the MBIC and BABE datasets— two of the most influential resources in recent media bias research. MBIC included sentence- and word-level annotations with detailed annotator background metadata Spinde et al. 2021a . BABE (Bias Annotations By Experts), built by trained annotators, significantly improved annotation quality, offering around 4000 high-quality labeled sentences across topics and biases Spinde et al. 2021b . Regarding modeling, previous approaches used logistic regression or random forests based on manually created linguistic features Spinde et al. 2020 . With the introduction of attention in transformers by Vaswani et al. 2017 , later work adopted transformer-based models, especially BERT, RoBERTa, and XLNet, for sentence-level classification Devlin et al. 2019 ; Liu et al. 2019a . These models showed improved contextual sensitivity and generalization. With noisy but scalable training data, distant supervision has become a promising method for producing weak labels from partisan sources (like AllSides) Tang et al. 2014 ; Spinde et al. 2021b . A significant recent contribution to bias detection is made by Krieger et al. 2022 , who introduced DA-RoBERTa, a domain-adapted RoBERTa model pre-trained on the Wiki Neutrality Corpus (WNC) Pryzant et al. 2019 and fine-tuned on the BABE dataset. Although their method achieves a new state-of-the-art F1 score, our approach which is developed based on their work, achieves a higher score and subsequently proved consistent in statistical signifance testing. Powers et al. 2025 recently contributed to the classification of several types of biases such as religious bias, racial bias, political bias, and others through their work on GUS framework and published a bias-type-classifier model which forms an integral part of the bias-analysis pipeline of our work. Our work contributes to this gap by: 1. Adapting and fine-tuning a RoBERTa-base model on BABE dataset thereby providing a robust bias detector. 2. Statistical Significance Testing of our proposed model to establish its ground as a significant enhancement to the previous models. 3. Integrating with a bias-type classifier to reveal granular categories of the biases. Our contribution provides a robust bias detector and classifier by fusing neural modeling, domain-specific fine-tuning, and adapting to existing methodology. 3 Methodology 3.1 Model Selection and Setup Our goal is to develop a robust model for sentence-level bias classification in news articles. We build upon prior work by Krieger et al. 2022 , who introduced DA-RoBERTa, a xlm-RoBERTa model Conneau et al. 2019 pre-trained on the Wiki Neutrality Corpus (WNC) introduced by Pryzant et al. 2019 for domain adaptation and fine-tuned on the BABE dataset introduced by Spinde et al. 2021b . Their results demonstrated the effectiveness of domain-specific pre-training over general-purpose models for bias detection. Inspired by their setup, we evaluate DA-RoBERTa as a baseline and use it as a reference point for our model selection and evaluation. We tested their DA-Roberta model with a 5-fold cross validation on the entire BABE Dataset yielding the best F1 score of 0.8364. To enhance this, we create another model by taking the transformer Vaswani et al. 2017 model: Roberta base Liu et al. 2019b and fine-tuned it directly on the BABE dataset. Then evaluated it similarly and achieved the best macro F1 score as 0.9257. Figure 2: Architecture of the Bias Detector Model. The output vector of the CLS token serves as input to the classifier head 3.2 Learning Task and Objective The sentence-level bias classification task is framed as a binary supervised classification problem. Given a sentence x_{i}\in X from the corpus, the objective is to assign a label y_{i}\in {0,1} where: • y_{i}=1 denotes a biased sentence • y_{i}=0 denotes an unbiased sentence The binary cross-entropy loss is optimised over the training set D=\{(x_{i},y_{i})\}^{N}_{i=1} : \mathcal{L}=-\frac{1}{N}\sum_{i=1}^{N}\left[y_{i}\cdot\log\hat{y}_{i}+(1-y_{i})\cdot\log(1-\hat{y}_{i})\right] (1) where \hat{y_{i}} is the model’s predicted probability for the sentence being biased. A softmax classifier is used on top of the [CLS] token embedding from RoBERTa. See Fig. 2