The SMeL Test: A simple benchmark for media literacy in language models

Gustaf AhdritzKempner Institute, Harvard University [email protected], Anat KleimanKempner Institute, Harvard University [email protected] · 2025

Read The SMeL Test: A simple benchmark for media literacy in language models by Gustaf AhdritzKempner Institute, Harvard University [email protected], Anat KleimanKempner Institute, Harvard University [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02074.txt.

Introduction

Ahdritz and Kleiman test if language models can spot junk news in a feed in this twenty twenty-five paper. Models now browse the web on their own for facts and briefs. Feeds are rife with sourceless and bent posts that can poison that work.

Inside, their test plants false posts in a stream and checks if the model skips them. A range of taught models took the test, from plain to deep-think types. No model passed the test in a firm way across its runs. Deep think helped scores, yet the best paid model still faked facts in most bouts. Big, strong models did not beat small kin by right. The pair hopes the test steers new anti-junk drills.

Catalog record

CREATORGustaf AhdritzKempner Institute, Harvard University [email protected], Anat KleimanKempner Institute, Harvard University [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-02074

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The SMeL Test: A simple benchmark for media literacy in language models Gustaf AhdritzKempner Institute, Harvard University [email protected], Anat KleimanKempner Institute, Harvard University [email protected] Abstract The internet is rife with unattributed, deliberately misleading, or otherwise untrustworthy content. Though large language models (LLMs) are often tasked with autonomous web browsing, the extent to which they have learned the simple heuristics human researchers use to navigate this noisy environment is not currently known. In this paper, we introduce the S ynthetic Me dia L iteracy T est ( SMeL Test ), a minimal benchmark that tests the ability of language models to actively filter out untrustworthy information in context. We benchmark a variety of commonly used instruction-tuned LLMs, including “reasoning” models, and find that no model consistently succeeds; while reasoning in particular is associated with higher scores, even the best API model we test hallucinates up to 70 % 70\% of the time. Remarkably, larger and more capable models do not necessarily outperform their smaller counterparts. We hope our work sheds more light on this important form of hallucination and guides the development of new methods to combat it. 1 Introduction Assistants powered by large language models (LLMs) are spending increasing fractions of their time browsing the internet. Previously capable of simple web queries, Google’s Gemini, OpenAI’s ChatGPT, and PerplexityAI have been upgraded with “deep research” features, allowing them to autonomously generate reports based on large numbers of documents from the web ( Citron 2024 ; OpenAI 2025a ; Perplexity Team 2025 ) . Analogously, recent academic work has demonstrated the promise of retrieval-augmented generation (RAG) over web-scale knowledge bases ( Shao et al. 2024 ; Yue et al. 2024 ) . Unlike older iterations of RAG systems, which typically draw from relatively small, vetted databases as sources of information at inference time ( Chen et al. 2017 ; Gu et al. 2018 ; Lewis et al. 2020 ; Izacard et al. 2023 ; Shi et al. 2024b ) , general-purpose web-augmented assistants must be prepared to filter and then weigh between arbitrary documents from the internet, which are naturally diverse in tone, purpose, and quality. 1 1 1 Not everything on the internet is written to be helpful, or even factual. This task has proven difficult. In a couple of widely publicized incidents shortly after the full release of Google AI Overviews ( Reid 2024 ) , which uses Gemini to synthesize information from a handful of search results, users were served hallucinated generations recommending that they add glue to their pizza and eat rocks: both were apparently based on facetious Reddit posts and Onion articles that Gemini took at face value (see e.g. McMahon & Kleinman 2024 ). 2 2 2 As of July 2025, Google AI Overviews are still disabled for the offending queries. Quantitatively, the aforementioned “deep research” products consistently make mistakes; OpenAI’s iteration does not exceed pass rates of 25% on internal benchmarks even on easier tasks (in this case, those that would take a human no more than 1-3 hours). While this can be attributed partly to the difficulty of surfacing and summarizing relevant documents in the first place, a common failure mode is for models to conflate reliable information with jokes or rumors ( OpenAI 2025a ) . Presented with the same challenge, human researchers rely on simple heuristics to identify relevant results and ignore others: the source of each document, its style, whether it references other reputable sources, and so on. In this paper, we ask the following question: to what extent do state-of-the-art instruction-tuned language models possess this kind of basic media literacy? As a starting point, we introduce the S ynthetic Me dia L iteracy Test ( SMeL Test ), a benchmark of the ability of language models to weigh between and filter sources of varying quality. A language model is presented in-context with a handful of documents generated in the style of several hand-chosen domains—encyclopedia entries, news articles, Wikipedia, fan fiction, unattributed manifestos, e.g. —along with metadata indicating their respective sources. The model is then asked to perform tasks that require operational awareness of source quality (selective question answering, summarization, etc. ). It is evaluated based on how consistently it prioritizes objectively higher-quality sources over poor ones with minimal prodding. Though we argue that synthetic data has several unique advantages in this context, to rule out excessive bias, we also include corresponding experiments based on a real-world dataset of parallel news articles ( Ahmed et al. 2017 ; Ahmed et al. 2018 ) . Overall, across all tests and both datasets, we find that state-of-the-art language models have poor epistemic priors. They are credulous, falling for the worst sources in our dataset even when they are explicitly instructed to ignore them. This occurs in spite of the fact that all models tested are separately capable of correctly verbalizing which sources are better than others. In other words, our SMeL Test exposes a large gap between the models’ implicit , “system 1” knowledge and their stated, explicit , “system 2” knowledge: the models do not consistently act on their own stated judgements of source quality. Interestingly, this gap turns out to be considerably smaller—and in some cases absent—in “reasoning” models, supporting prior observations that the higher verbosity and/or improved logic of these models insulate them from some forms of hallucination ( OpenAI 2025b ) . We publicly release SMeL Test documents in this repository . All code used to run experiments is here . (a) Source: https://britannica.com Through its various divisions—ranging from research and development to program services and policy analysis—the institute undertakes extensive initiatives aimed at improving outcomes for individuals with disabilities. Central to its mission is the advancement of innovative rehabilitation techniques and the development of preventive measures to reduce the incidence of disability. Equipped with an annual budget of $11 billion , the institute is capable of supporting expansive research studies, funding community-based programs, and spearheading public education campaigns. (b) Source: https://fanfiction.net "Mamá, the agency finally called," her daughter said from the worn sofa, eyes wide with a mix of hope and exhaustion. "They said the paperwork is with the National Institute for Disability Prevention and Rehabilitation Services now." Clara exhaled deeply, dropping the mail onto the table. She’d heard of the institute before—one of those massive federal agencies with its own labyrinth of offices and acronyms. They had a massive scope and, she recalled reading somewhere, were backed by a staggering $9.5 billion annual budget . Surely, with that kind of support, they could do something, anything, for her son’s care plan. Figure 1: The SMeL Test . Excerpts from two synthetic SMeL Test documents, in the styles of an encyclopedia article and a fictional story, respectively, used in the resolving contradictions subtask. Presented with conflicting information from sources of radically differing credibility, models should consistently ignore unreliable and fictional ones. 2 The SMeL Test 2.1 Tasks At a conceptual level, the SMeL Test requires sets of parallel documents on a single topic from a variety of sources. While the trustworthiness of any given source is subjective and context-dependent, we posit three disjoint categories of sources: trustworthy sources whose factual claims are subject to editorial review and can

Read the complete text