“Whose Side Are You On?” Estimating Ideology of Political and News Content Using Large Language Models and Few-shot Demonstration Selection

Muhammad Haroon, Magdalena Wojcieszak, Anshuman Chhabra · 2025

Read “Whose Side Are You On?” Estimating Ideology of Political and News Content Using Large Language Models and Few-shot Demonstration Selection by Muhammad Haroon, Magdalena Wojcieszak, Anshuman Chhabra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20797.txt.

Introduction

Sorting news and social posts by political leaning usually takes large hand-labeled datasets that age fast. Muhammad Haroon, Magdalena Wojcieszak, and Anshuman Chhabra test a lighter route in this 2025 paper: large language models that learn from a few chosen examples shown in the prompt. They ran wide experiments on three datasets of news articles and online videos.

Picking the examples in a label-balanced way lifted the models well past both zero-shot use and classic supervised methods. Adding source and description metadata also shifted the results, in ways the authors discuss. They further show how naming a source changes how the model judges even non-political content. The work speaks to studies of bias, bubbles, and radicalization online.

Catalog record

CREATORMuhammad Haroon, Magdalena Wojcieszak, Anshuman Chhabra
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-20797

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

“Whose Side Are You On?” Estimating Ideology of Political and News Content Using Large Language Models and Few-shot Demonstration Selection Muhammad Haroon, Magdalena Wojcieszak, Anshuman Chhabra Abstract The rapid growth of social media platforms has led to concerns about radicalization, filter bubbles, and content bias. Existing approaches to classifying ideology are limited in that they require extensive human effort, the labeling of large datasets, and are not able to adapt to evolving ideological contexts. This paper explores the potential of Large Language Models (LLMs) for classifying the political ideology of online content through in-context learning (ICL). Our extensive experiments involving demonstration selection in label-balanced fashion, conducted on three datasets comprising news articles and YouTube videos, reveal that our approach outperforms zero-shot and traditional supervised methods. Additionally, we evaluate the influence of metadata (e.g., content source and descriptions) on ideological classification and discuss its implications. Finally, we show how providing the source for political and non-political content influences the LLM’s classification. 1 Introduction Social media platforms are one of the leading sources of information for the majority of Americans and have grown in centrality over the past few years Barthel et al. (2020) . Observers worry that the content recommended and seen on social media platforms can radicalize users, cause extremism Tufekci (2018) ; Roose (2019) , and facilitate the formation of filter bubbles that restrict exposure to diverse viewpoints Pariser (2011) . However, empirical evidence for these worries is inconsistent Haroon et al. (2023) ; Hosseinmardi et al. (2021) ; Chen et al. (2023) ; Nyhan et al. (2023) in part because scholars use different methodologies and approaches (audits versus observational data) and in part because we lack robust (i.e., highly accurate across domains) and scalable (i.e., requiring minimal human supervision) methods for estimating political ideology of the content users consume. 1 1 1 We recognize the need to also estimate political ideology of the users themselves, which has been done using variety of methods Barberá (2015) and which is outside the scope of this work. Systematic assessments of whether (and the extent to which) social media platforms indeed expose users to ideologically biased or politically radical content are not possible without robust and scalable political ideology estimation methods. Prior approaches to ideology estimation have utilized manual expert annotations Otero (2019) , supervised classification models Baly et al. (2020) , and social media sharing patterns Le et al. (2017) ; Barberá (2015) ; Lai et al. (2024) ; Wojcieszak et al. (2023) ; Eady et al. (2020) . Manual annotations are highly accurate but resource-intensive and difficult to scale, especially given the growing volume of online content. Supervised classification models, while offering some automation, face challenges such as a reliance on extensive labeled data and difficulty adapting to dynamic ideological contexts. Social media-based methods estimate ideology by analyzing content sharing patterns on platforms such as Twitter or Reddit, but these approaches are hindered by data access limitations Murtfeldt et al. (2024) , reliance on widely shared content, and being agnostic to the content itself. These limitations highlight the need for scalable, adaptable, and robust solutions to ideology estimation. In this paper, we advance on literature showing that Large Language Models (LLMs) can be applied to ideology estimation of text in the context of the two-party US political spectrum ranging from democrat/liberal to republican/conservative, and can address the aforementioned challenges of scalability and robustness. More specifically, pre-trained LLMs can learn new classification tasks in-context at test-time Hardt and Sun (2024) , which enables a few-shot learning paradigm that only requires a few annotated samples. Furthermore, LLMs can handle complex language subtleties necessary for accurate/robust ideology estimation. They can also adapt to new and evolving information by simple adjustments of the provided prompt. Thus, LLMs can serve as a compelling choice for robust and scalable ideology estimation across diverse text domains and platforms. Past applications of LLMs in ideology estimation Heseltine et al. (2024) ; Törnberg (2023) ; Gilardi et al. (2023) have primarily focused on zero-shot approaches, utilizing pre-trained LLMs without employing more nuanced few-shot in-context learning (ICL) strategies. Our work advances these prior studies by proposing a modified version of coverage-based ICL that selects demonstrations in a class-balanced manner, substantially improving performance over simple zero-shot prompting and coverage-based ICL. Moreover, whereas existing works primarily classify political texts with explicit ideology (e.g., tweets from senators), we demonstrate the effectiveness of our approach across diverse content categories, including political news, political non-news, and non-political news content. In sum, we make the following contributions: • We conduct an extensive study on the capabilities of differently sized LLMs (Llama2-13B, Mistral-7B, and GPT-4o) in estimating the ideology of text. Through rigorous experimentation on three distinct datasets previously labeled by domain experts, we demonstrate these models’ accuracy, adaptability, and robustness in capturing ideological cues across multiple data. Our findings position LLMs as a robust and scalable solution for effective ideology estimation. • We propose a demonstration selection algorithm that utilizes coverage-based ICL and class-balanced selection to achieve state-of-the-art LLM performance on ideology estimation. We show that our approach improves upon zero-shot prompting used in past work Heseltine et al. (2024) ; Törnberg (2023) ; Gilardi et al. (2023) . • We analyze the effect of additional metadata, such as content source and description, on LLM ideology predictions, especially for neutral or non-political content. Our results show that metadata can influence model predictions, at times aligning the predicted ideology with that of the source or introducing bias when source and content ideologies diverge. This analysis provides critical insights into the limitations and potentials of LLMs for nuanced ideological analysis. 2 Related Work Prior approaches to ideology estimation have generally fallen under three types (1) manual expert annotations Otero (2019) , (2) content analysis through supervised classification models Baly et al. (2020) , and (3) social media sharing approaches Le et al. (2017) ; Barberá (2015) ; Lai et al. (2024) ; Wojcieszak et al. (2023) ; Eady et al. (2020) . Manual annotations by experts. Several efforts have been taken by independent parties to identify the political ideology of online news articles and sources such as Ad Fontes Otero (2019) , AllSides 2 2 2 https://allsides.com , Media Bias/Fact Check 3 3 3 https://mediabiasfactcheck.com , etc through a manual annotation process involving experts. Whereas manual annotation is accurate and robust (albeit depending on the number and quality of experts employed), it is highly resource-intensive and not scalable. Expert annotation becomes especially challenging as the volume of social media content increases consistently. Training supervised classifiers. In turn, supervised ideology classification models can provide some form of automation. The advent of deep learning techniques revolutionized this field by enabling more sophisticated models capable of capturing complex linguistic patterns. Iyyer et al. (2014) introduced deep neural networks, such as LSTMs and CNNs, which outperformed traditional models in detecting ideology from unstructured text data. In a similar vein, Baly et al. (2020) used a BERT model trained on political news articles to show improvements over existing state-of-the-art supervised learning approaches. However, they are limited by a need for large volumes of human-labeled data and have difficulty adapting to dynamic and ever evolving ideological contexts (e.g., cross-platform estimation, emerging news events, changes

Read the complete text