Beyond the Link: Assessing LLMs’ Ability to Classify Political Content Across Global Media
Alejandro De La Fuente-Cuesta, Alberto Martinez-Serra, Nienke Visscher, Laia Castro, Ana S. Cardenal · 2025
Read Beyond the Link: Assessing LLMs’ Ability to Classify Political Content Across Global Media by Alejandro De La Fuente-Cuesta, Alberto Martinez-Serra, Nienke Visscher, Laia Castro, Ana S. Cardenal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17435.txt.
Introduction
Alejandro De La Fuente-Cuesta and co-authors test whether large language models can sort news into political and non-political, from the link alone or from the text. Such sorting is now common in studies of politics and media, but the value of the link itself was not well known.
The study spans five lands and their languages, with human coded data as the yardstick. Links did carry real signal, and link based sorting could stand in for full text work at lower cost. Yet the models too often tagged centrist news as political, a bias that could skew later work. The authors close with advice for sound use.
Catalog record
| CREATOR | Alejandro De La Fuente-Cuesta, Alberto Martinez-Serra, Nienke Visscher, Laia Castro, Ana S. Cardenal |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-17435
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond the Link: Assessing LLMs’ Ability to Classify Political Content Across Global Media Alejandro De La Fuente-Cuesta, Alberto Martinez-Serra, Nienke Visscher, Laia Castro, Ana S. Cardenal Abstract The use of large language models (LLMs) is becoming common in political science and digital media research. While LLMs have demonstrated ability in labelling tasks, their effectiveness to classify Political Content (PC) from URLs remains underexplored. This article evaluates whether LLMs can accurately distinguish PC from non-PC using both the text and the URLs of news articles across five countries (France, Germany, Spain, the UK, and the US) and their different languages. Using cutting-edge models, we benchmark their performance against human-coded data to assess whether URL-level analysis can approximate full-text analysis. Our findings show that URLs embed relevant information and can serve as a scalable, cost-effective alternative to discern PC. However, we also uncover systematic biases: LLMs seem to overclassify centrist news as political, leading to false positives that may distort further analyses. We conclude by outlining methodological recommendations on the use of LLMs in political science research. Keywords: Large Language Models (LLMs), Media analysis, Political content, URLs, Web content. 1 Introduction In the era of digital media, understanding how individuals consume information online has become central to the study of behavioural science ( Martinez-Serra & Cardenal, (2025) ; Kulshrestha et al., (2021) ) . The internet—and particularly news content distributed through websites and social media platforms—now serves as a primary source of political knowledge worldwide ( Shehata & Strömbäck, (2021) ) . This digital transformation has created new opportunities for researchers to investigate political information exposure at an unprecedented scale, using novel forms of behavioural data such as web tracking data. The use of large repositories of browsing histories can reveal de facto information diets of thousands of individuals ( Christner et al., (2022) ; Clemm von Hohenberg et al., (2024) ) . These datasets allow researchers to map news consumption patterns, such as the prevalence of selective exposure and exposure to misinformation ( Cardenal et al., (2019) ; Wojcieszak et al., (2022) ; Guess et al., (2020) ) . However, researchers face a rather complex question: How can we reliably identify what counts as Political Content (PC)? In political communication studies, researchers typically classify news content based on whether the Uniform Resource Locator (URL) originates from a known media outlet. However, this approach usually struggles to indicate the content itself. Therefore, distinguishing PC from non-PC at large scale remains a significant challenge. Large language models (LLMs) offer a promising new approach for scalable text classification in political analysis ( Le Mens & Gallego, (2025) ; Kirkizh et al., (2024) ) . Advances in Natural Language Processing (NLP) enable researchers to leverage the capacity of pretrained models for text classification ( Wang et al., (2023) ; Min et al., (2023) ) . Recent studies have shown that LLMs can serve as high-quality, cost-effective annotators for a variety of political text classification tasks, following the path of automated content analysis methods ( Grimmer & Stewart, (2013) ) . For instance, Heseltine & Clemm von Hohenberg ((2024)) demonstrated that GPT-4 can consistently replicate expert human coding for brief pieces of text from four nations. Similarly, Vera & Driggers ((2024)) showed that LLMs reflect human-like biases in PC classification, particularly when party cues are present. Recently, Le Mens & Gallego ((2025)) demonstrated that instruction-tuned LLMs can effectively position political texts on ideological scales. Despite this progress, important gaps remain in our understanding of LLM-based classification, particularly for URL-only inputs and cross-lingual settings. Most prior studies have given LLMs access to the full text of documents (tweets, articles, etc.) when performing classification ( Makhortykh et al., (2024) ) . An intermediate approach is to classify article titles, which were recently found to capture much of the context as proven by a high agreement between title- and full-text classification ( Heseltine et al., (2025) ) . However, in large-scale web-tracking, titles are rarely in raw trace data and require scraping each URL. Therefore, in many social media research scenarios where raw tracking data is provided, scholars possess only the URLs rather than the content itself ( Olejnik et al., (2014) ) . As a result, a second question arises: Could LLMs accurately determine whether a URL links to PC without accessing the full article? If so, this would be a methodological advancement – enabling rapid classification using minimal information. URL-based classification is not unprecedented in web research: previous approaches have used features like domain names, URL paths, or metadata to categorise webpages (for instance, to detect spam or phishing) with some success ( Rastakhiz et al., (2024) ; Li et al., (2025) ; Baykan et al., (2013) ) . With news, the URL usually contains cues in the path – for example, keywords from the article title – that an advanced LLM might be able to use ( Clemm von Hohenberg et al., (2024) ) . However, some outlets rely on highly non-descriptive URL structures, providing little linguistic information that models can exploit (e.g., /world-europe-60547473). This further complicates URL-based classification, since such cases offer almost no textual signal about the underlying content. Nevertheless, the potential advantages of URL-only classification are significant: It eliminates the need for fetching or processing full text, speeding up analysis and avoiding copyright or paywall issues. Recent evidence indicates that in situations where full content is inaccessible, classification performance can still be sufficient using solely URLs ( Schelb et al., (2024) ) . In this study we investigate this question by evaluating several leading LLMs on the task of classifying political versus non-political news, evaluating both URLs and full text scenarios. We present, to our knowledge, the first LLM assessment on PC classification across multiple countries and languages. We test a range of state-of-the-art models, such as Llama ( Grattafiori et al., (2024) ) , Mistral ( Jiang et al., (2024) ) , DeepSeek ( DeepSeek-AI et al., (2025) ) , Qwen ( Yang et al., (2025) ) , and Gemma ( Team et al., (2025) ) , to examine how different models and sizes impact performance (Table 1 ). By comparing model predictions to a human-labelled ground, we assess the accuracy of each LLM in identifying PC. Our results show that LLMs can classify PC with high accuracy not only from full articles but also from URLs alone, with URL-based approaches often improving precision while remaining consistent across countries and languages. However, we also find that LLMs tend to overestimate the presence of PC in centrist news, leading to systematic false positives that may bias estimates of political exposure and polarisation. Overall, this work extends the current methods of content analysis, and our descriptive knowledge of the ways people access news in the digital age, in addition to practical recommendations on LLM-based methodologies for future research in political analysis. 2 Data and Methods 2.1 LLMs To classify the PC of the URLs and databases, we made use of different large language models (LLMs) listed in Table 1 . This selection includes models of varying sizes and providers, ranging from computationally demanding architectures to smaller ones like Llama-8B that can be more easily run locally. All experiments were conducted on a high-performance computing