Navigating Nuance: In Quest for Political Truth DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 Conference: The 2024 ACM/IEEE Joint Conference on Digital Libraries; Dece
Soumyadeep Sar, Dwaipayan Roy · 2025
Read Navigating Nuance: In Quest for Political Truth DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 Conference: The 2024 ACM/IEEE Joint Conference on Digital Libraries; Dece by Soumyadeep Sar, Dwaipayan Roy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00782.txt.
Introduction
Soumyadeep Sar and Dwaipayan Roy probed how well a large language model spots slanted political text, in a paper published in 2025. Their test bed is a known bias benchmark set. Their method lies in the prompt: it feeds the model fine-grained reasons that a text might lean one way, not just the bare task. The model is a seventy billion parameter open build.
With those prompts, the untuned model matched a small tuned rival that held the top spot on the set. That parity came with no task training at all. The authors say the result shows how much prompt craft can do. They release their code and data. The aim is tools that blunt false stories and divided camps.
Catalog record
| CREATOR | Soumyadeep Sar, Dwaipayan Roy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-00782
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Navigating Nuance: In Quest for Political Truth DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 Conference: The 2024 ACM/IEEE Joint Conference on Digital Libraries; December 16–20, 2024; Hong Kong, China The 2024 ACM/IEEE Joint Conference on Digital Libraries (JCDL ’24), December 16–20, 2024, Hong Kong, China DOI: 10.1145/3677389.3702601 ISBN: 979-8-4007-1093-3/24/12 CCS: Information systems Social recommendation Soumyadeep Sar, Dwaipayan Roy Abstract . This study investigates the several nuanced rationales for countering the rise of political bias. We evaluate the performance of the Llama-3 (70B) language model on the Media Bias Identification Benchmark (MBIB), based on a novel prompting technique that incorporates subtle reasons for identifying political leaning. Our findings underscore the challenges of detecting political bias and highlight the potential of transfer learning methods to enhance future models. Through our framework, we achieve a comparable performance with the supervised and fully fine-tuned ConvBERT model, which is the state-of-the-art model, performing best among other baseline models for the political bias task on MBIB. By demonstrating the effectiveness of our approach, we contribute to the development of more robust tools for mitigating the spread of misinformation and polarization. Our codes and dataset are made publicly available in github 1 1 1 https://github.com/Soumyadeepsar/Navigating-Nuance-In-Quest-for-Political-Truth . 1. Introduction Political bias detection has become a critical area of research in natural language processing (NLP) due to its significant influence on media literacy, public opinion, and democratic processes. The pervasive nature of political bias in media and online content demands robust methodologies for its identification and analysis. Political bias can manifest in various forms, including word choice, framing of issues, and selective omission of information, all of which can subtly influence readers’ perceptions and beliefs (Entman, 2007) ( Entman 2007 ) . Recent advancements in machine learning and NLP have enabled the development of sophisticated models for detecting political bias. Traditional approaches relied heavily on lexicon-based methods, which involved predefined lists of biased terms and phrases. While these methods provided a foundation, they often struggled with the nuanced and context-dependent nature of political bias. More contemporary techniques leveraging deep learning and large language models (LLMs) to capture subtler forms of bias and context-specific variations in sentence-level text have been done ( Hong et al. 2023 ) . Efforts have been made to learn the factuality of reporting and bias, trying to categorize entire news media based upon different features collected through its URLS, websites, Wikipedia page, Twitter account and many other factors ( Baly et al. 2018 ) . One promising approach in the domain of in-context learning is the use of Chain-of-Thought (CoT) prompting, which involves generating and utilizing intermediate reasoning steps to improve the reasoning performance of LLMs. This method aims to enhance the model’s interpretability and accuracy by breaking down the decision-making process into a sequence of logical steps ( Wei et al. 2022 ) . By employing CoT prompting, researchers can better understand the underlying reasoning of LLMs and ensure more reliable and better performance on complex tasks. Hence we try to leverage COT, to shape proper reasoning and understanding of political bias in LLMs, resulting in better performance. Many studies were conducted in past for learning connections and patterns in biased text, trying to devise automatic detection solutions ( Pérez-Rosas et al. 2018 ) . Despite these advancements, challenges remain in achieving high accuracy and generalizability across diverse datasets and political contexts. The inherently subjective nature of political bias along with other factors and the dynamic evolution of political discourse, add complexity to this task. Therefore, research is essential to try to put this hypothesis to test and develop standardized techniques for further research in this area. This paper explores the effectiveness of COT prompting in improving the ability of LLMs to classify statements as biased or unbiased. We build on existing literature and propose a novel prompt that utilizes a COT technique based upon subtle reasoning steps to enhance political bias detection. Through extensive experimentation and analysis, we aim to contribute to the growing body of work in this field. 2. Related work Prompt-based fine-tuning has significantly enhanced the performance of Pre-trained Language Models (PLMs) in few-shot text classification by using task-specific prompts. However, since PLMs are not pre-trained with prompt-style expressions, their effectiveness in few-shot learning is limited. To address this, the Unified Prompt Tuning (UPT) framework was introduced by Wang et al. 2022 , which improves few-shot text classification for BERT-style models by capturing prompting semantics from various non-target NLP datasets. UPT employs a novel Prompt-Options-Verbalizer paradigm for joint prompt learning across tasks, enabling PLMs to acquire task-invariant prompting knowledge. Additionally, a self-supervised task, Knowledge-enhanced Selective Masked Language Modeling, is designed to boost the PLM’s generalization capabilities, allowing better adaptation to unfamiliar tasks in low-resource settings. Experimental results across diverse NLP tasks demonstrate that UPT consistently outperforms existing state-of-the-art methods in prompt-based fine-tuning. The study by Wen and Younes 2023 investigates the ability of large language models (LLMs), such as ChatGPT, to detect media bias by utilizing the Media Bias Identification Benchmark (MBIB) ( Wessel et al. 2023 ) . ChatGPT’s performance is compared against fine-tuned models like Bidirectional and AutoRegressive Transformers (BART) ( Lewis et al. 2020 ) , Convolutional Bidirectional Encoder Representations from Transformers (ConvBERT) ( Jiang et al. 2021 ) , and Generative Pre-trained Transformer 2 (GPT-2). The results reveal that while ChatGPT is on par with fine-tuned models in identifying hate speech and text-level context bias, it struggles with detecting subtler elements of media bias, including fake news, racial, gender, political and cognitive biases. Large language models (LLMs) have excelled in tasks such as dialogue generation, commonsense reasoning, and question-answering. In-context learning (ICL) is a key method for adapting LLMs to downstream tasks through prompt-based demonstrations. However, the performance can be significantly impacted by the distribution of these demonstrations, particularly in challenging classification tasks. Du et al. 2023 introduce task-level thinking steps to mitigate bias introduced by demonstrations. Additionally, they propose a progressive revision framework to enhance thinking steps by correcting difficult demonstrations. Experimental results demonstrate that their method outperforms others on three types of challenging classification tasks in both zero-shot and few-shot settings. The incorporation of task-level thinking steps and automatically generated CoT yields more competitive performance. Figure 1 . Distribution of text according to length 3. Dataset Media bias detection is a complex multi-task problem which can be very challenging on its own. There is a wide variety of bias propagating in various news outlet and media platforms. The Media Bias Identification Benchmark (MBIB) ( Wessel et al. 2023 ) , provides an extensive benchmark that groups different types of media bias (e.g., linguistic, cognitive, political) under a common framework which allows researchers to develop and test their techniques against different types of bias detection. Notably, this is the first-ever large benchmark-level task designed specifically for bias detection. The Dataset is well-balanced, with an equal number of biased and unbiased text. The frequency distribution of statements according to text length across both labels is illustrated in Figure 1 . 4. Methodology 4.1. Data-preprocessing For the experiments, we use the dataset from mbib-base 2 2 2 https://huggingface.co/datasets/mediabiasgroup/mbib-base . It has