Profiling News Media for Factuality and Bias Using LLMs and the Fact-Checking Methodology of Human Experts
Zain Muhammad Mujahid, Dilshod Azizov, Maha Tufail Agro, Preslav Nakov · 2025
Read Profiling News Media for Factuality and Bias Using LLMs and the Fact-Checking Methodology of Human Experts by Zain Muhammad Mujahid, Dilshod Azizov, Maha Tufail Agro, Preslav Nakov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-12552.txt.
Introduction
Zain Muhammad Mujahid and co-authors profile whole news outlets for factuality and bias in this 2025 paper. Checking single claims is hard when a story is new and evidence is thin. Rating the outlet itself can warn readers before they trust a claim.
The team use large language models guided by the method of human fact-checking experts. Rather than judge one article, they characterize the source as a whole. Earlier work leaned on language clues and social context. This approach brings expert workflow into the model process. Media literacy tools could use such outlet profiles at scale. The study treats source profiling as a research goal in its own right.
Catalog record
| CREATOR | Zain Muhammad Mujahid, Dilshod Azizov, Maha Tufail Agro, Preslav Nakov |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-12552
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Profiling News Media for Factuality and Bias Using LLMs and the Fact-Checking Methodology of Human Experts Zain Muhammad Mujahid, Dilshod Azizov, Maha Tufail Agro, Preslav Nakov Abstract In an age characterized by the proliferation of mis- and disinformation online, it is critical to empower readers to understand the content they are reading. Important efforts in this direction rely on manual or automatic fact-checking, which can be challenging for emerging claims with limited information. Such scenarios can be handled by assessing the reliability and the political bias of the source of the claim, i.e., characterizing entire news outlets rather than individual claims or articles. This is an important but understudied research direction. While prior work has looked into linguistic and social contexts, we do not analyze individual articles or information in social media. Instead, we propose a novel methodology that emulates the criteria that professional fact-checkers use to assess the factuality and political bias of an entire outlet. Specifically, we design a variety of prompts based on these criteria and elicit responses from large language models (LLMs), which we aggregate to make predictions. In addition to demonstrating sizable improvements over strong baselines via extensive experiments with multiple LLMs, we provide an in-depth error analysis of the effect of media popularity and region on model performance. Further, we conduct an ablation study to highlight the key components of our dataset that contribute to these improvements. To facilitate future research, we released our dataset and code. 1 1 1 https://github.com/mbzuai-nlp/llm-media-profiling 1 Introduction In an age where digital media dominate and information spreads quickly, profiling news media outlets in terms of their political bias and factuality is of utmost importance. Media organizations significantly shape public discourse Sajwani et al. 2024 , influence policy, and shape public opinion, making the detection of political bias essential Pennycook and Rand 2021 . Traditional methods for characterizing political bias in news media, such as subjective assessments and manual content analysis, are labor-intensive and prone to human biases. Automated content and social media analysis techniques Baly et al. 2018 ; Baly et al. 2020b have been developed for this, but they face limitations, including the laborious process of obtaining and annotating news articles. Evaluating the factuality of the news reporting is equally important. Assessing the accuracy and truthfulness of news articles is the key to maintain the integrity of information dissemination Baly et al. 2018 . Conventional fact-checking methods are resource-intensive and struggle to keep up with the rapid production of news content. LLMs, such as the OpenAI GPT series Radford et al. 2018 ; Radford et al. 2019 ; Brown et al. 2020 , offer a promising solution. Trained on a vast amount of text datasets, LLMs can understand and generate human-like text, providing a new avenue for analyzing media bias and factuality at scale. Figure 1: Overview of our methodology: (a) collection of gold labels from the MBFC 3 3 3 www.mediabiasfactcheck.com database, (b) data curation using handcrafted prompts, followed by text classification, (c) data curation using systematic prompts based on expert guidelines, followed by text classification. In this paper, we propose a novel methodology that leverages LLMs to predict political bias and the factuality of the reporting of entire news media outlets. Our approach (shown in Figure 3 ) involves crafting custom prompts to elicit responses from LLMs, thus enabling the detection of political bias and the factuality of a news outlet, without relying on the manual analysis of individual news articles. By employing the criteria used by professional fact-checkers, we aim to provide a more systematic and accurate assessment of political bias. We also conduct a case study to demonstrate the limitations of LLMs in the absence of well-defined guidelines, highlighting the importance of expert-driven prompts. Furthermore, we conduct detailed error analysis to examine the impact of media popularity and region on the performance of the models, revealing biases in favor of more popular and U.S.-based outlets. We also perform an ablation study to identify key components of our dataset, demonstrating the importance of combining leaning and reasoning information for optimal results. This analysis not only highlights the strengths of our approach but also uncovers critical areas for future improvement. The following summarizes our key contributions: • We release a large-scale dataset to model the factuality and political bias of news media. • We leverage knowledge from LLMs to predict the factuality and the political bias of news media. • We are the first to emulate the exact criteria used by professional fact-checkers when rating the political bias of the news media. • We achieve sizeable improvements over baselines and zero-shot prompting for two tasks: predicting (i) the factuality of reporting and (ii) the political bias of news outlets. • We conduct a comprehensive error analysis to examine the influence of media popularity and region on model performance. • We perform an ablation study to evaluate the contribution of dataset components, showing that the reasoning extracted from LLMs is the most critical part for accurate predictions. 2 Related Work The digital age has democratized the creation and dissemination of information via numerous media platforms, but this has also fueled misinformation Naeem et al. 2021 . News media profiling for political bias and factuality is essential to empower users and fact-checkers, ensure accountability, and support research Nakov et al. 2024 . Political bias refers to systematic inclinations towards a candidate or ideology Waldman and Devitt 1998 . Detecting political bias has been explored using a variety of features and methodologies, with predictive models operating across different levels of granularity, including media outlets, individual articles, and even sentences. For example, Baly et al. 2018 employed features from the NELA toolkit Horne et al. 2018 , while Kulkarni et al. 2018 examined article-level political bias by analyzing textual content and URLs, leveraging site-level annotations from AllSides 4 4 4 www.allsides.com . At the outlet level, political bias can be detected by comparing media language to political speeches Gentzkow and Shapiro 2006 . It can be also done by classifying articles along ideological axes such as left vs. right, or hyper-partisan vs. mainstream Potthast et al. 2018 ; Saleh et al. 2019 . Many of these models are based on distant supervision and are commonly trained on relatively small, English-language datasets Da San Martino et al. 2023 ; Barrón-Cedeño et al. 2023 ; Barrón-Cedeño et al. 2023 ; Azizov et al. 2023 ; Azizov et al. 2024 . Factuality prediction at the source level remains underexplored. Early research estimated the reliability of news sources by analyzing their stance on true or false claims rather than using explicit labels for medium-level factuality Dong et al. 2016 ; Baly et al. 2019 ; Popat et al. 2016 ; Popat et al. 2018 . Baly et al. 2018 explored political bias and factuality by extracting features from news articles, Wikipedia entries, Twitter metadata, and URLs, showing that integrating these sources improved classification accuracy. Later, Baly et al. 2019 found that the joint prediction of political bias and factuality was more effective. Baly et al. 2020a used both the linguistic aspects and the social context. This included analyzing the text of articles, audio content, and social media reactions and discussions on platforms such