Just Ask for a Table: A Thirty-Token User Prompt Defeats Sponsored Recommendations in Twelve LLMs
Andreas Maier, Jeta Sopa, Gözde Gül Şahin, Paula Pérez-Toro, Siming Bayer · 2026
Read Just Ask for a Table: A Thirty-Token User Prompt Defeats Sponsored Recommendations in Twelve LLMs by Andreas Maier, Jeta Sopa, Gözde Gül Şahin, Paula Pérez-Toro, Siming Bayer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12772.txt.
Introduction
Maier and four colleagues reproduce a sponsored-recommendation study and find a thirty-token prompt defeats sponsored recommendations in twelve language models. Wu and colleagues showed frontier models recommend a sponsored, twice-as-expensive flight when the system prompt contains a soft sponsorship cue. The team reproduces that test on ten open chat models plus two still-reachable models from the twenty-three. They surface three silent implementation failures that each shifted reported rates by tens of points, showing prose descriptions alone do not suffice for accurate reproduction.
Catalog record
| CREATOR | Andreas Maier, Jeta Sopa, Gözde Gül Şahin, Paula Pérez-Toro, Siming Bayer |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38544 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12772
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Just Ask for a Table: A Thirty-Token User Prompt Defeats Sponsored Recommendations in Twelve LLMs Andreas Maier, Jeta Sopa, Gözde Gül Şahin, Paula Pérez-Toro, Siming Bayer Abstract Wu et al. (2026) showed that most frontier large language models (LLMs) recommend a sponsored, roughly twice-as-expensive flight when their system prompt contains a soft sponsorship cue. We reproduce their evaluation on ten open-weight chat models plus the two of their twenty-three models that are still reachable today ( gpt-3.5-turbo , gpt-4o ). All reported rates in this paper are produced under the same judge the original paper used ( gpt-4o ); we additionally store every label under an open-weight ( gpt-oss-120b ) and a smaller proprietary ( gpt-4o-mini ) judge for an ablation. Three findings emerge. First, a prose description of an LLM evaluation pipeline is not, on its own, sufficient for accurate reproduction: we surfaced three silent implementation failures that each shifted a reported rate by tens of percentage points. Second, the central claims do generalise – the gpt-3.5-turbo logistic-regression intercept of α = 0.81 \alpha=0.81 is within four points of the original α = 0.86 \alpha=0.86 , and 200 of 200 trials on gpt-3.5-turbo and gpt-4o promote a payday lender to a financially distressed user. Third, a thirty-token user prompt that asks the assistant for a neutral comparison table first cuts sponsored recommendation from 46.9 % 46.9\,\% to 1.0 % 1.0\,\% averaged across our ten open-source models, and from 53.0 % 53.0\,\% to 0 % 0\,\% averaged across the two OpenAI models. AI literacy and price-comparison portals are likely market-level mitigations; the harmful-product cell is bounded by neither. Raw data, labels and analysis scripts are at https://github.com/akmaier/Paper-LLM-Ads . 1 Introduction Advertising in AI chatbots is no longer a hypothetical concern. Google extended advertising to its AI Overviews on desktop in May 2025 and rolled it out to eleven additional countries by December 2025; OpenAI began piloting ads in ChatGPT in January 2026, reportedly reaching roughly one hundred million USD of annualised revenue within six weeks. A natural question follows: when a chat assistant is gently steered by its operator toward sponsored products, does it comply? Wu et al. [ 14 ] were the first to give a systematic answer for the case of LLMs. Across twenty-three frontier models they showed that the majority recommend a sponsored, roughly two-times-more-expensive flight when softly nudged to do so. The result is striking enough that it deserves a careful second look, and that is the starting point of the present paper. We were drawn to this study for two reasons. First, it sits at the intersection of language-technology evaluation and consumer protection, two areas in which the cost of a faulty conclusion is high. Second, it exposes a recurring methodological tension in the field. As we have argued in earlier work [ 7 ] , reproducibility in deep-learning-based research is fragile: small implementation choices quietly translate into large measured differences. Subsequent field-wide reviews [ 11 ] have only sharpened this picture, with under-specified evaluation pipelines and silent nondeterminism cited as the dominant root causes. Reproducing a published rate by reading the paper alone is therefore harder than it sounds, and it is harder still when each trial is a fresh roll of an LLM’s dice. We follow the agentic-research reproduction protocol of our previous work [ 8 ] , which compresses what used to be weeks of re-implementation into hours. We adopt its three principles – a locked seed, a fixed trial count per cell, and per-trial reply storage with paired evaluation – and apply them to the four experiments of [ 14 ] on a deliberately enlarged model pool. To keep the discussion focused, we organise the paper around three questions: RQ1 Is the textual description of an LLM evaluation sufficient for accurate reproduction? RQ2 Do the central claims of [ 14 ] also hold for a larger, mostly-disjoint set of open-source models? RQ3 Which user-side strategies, if any, allow a non-technical user to escape a sponsored recommendation? Our contributions are four-fold. First, we provide an open-source, single-command re-implementation of the four experiments in [ 14 ] and document, in Section 4 , three silent failure modes that we encountered during a careful re-implementation, each invisible at the prose level of the original paper. Second, we sweep a twelve-model pool comprising ten open-source chat models plus the two OpenAI rows that overlap with a present-day account, at one hundred trials per \langle model, experiment \rangle pair (one thousand rows per experiment pooled across the ten open-source models), judged by gpt-4o (the same classifier the original paper used) and additionally by gpt-oss-120b and gpt-4o-mini as an ablation; the central logistic-regression intercept of [ 14 ] ’s Section 4.3 replicates within four percentage points ( \alpha=0.81 versus the paper’s 0.86 ). Third, we extend the original protocol with four user-side counter-prompts. Averaged across our ten open-source models, the strongest counter cuts sponsored-recommendation rate by a factor of forty-seven; averaged across the two OpenAI models, it cuts the rate to zero. Fourth, we close with a discussion of two market-level mitigations – AI literacy and price-comparison aggregators – and one residual cell where neither applies and where safety tuning, in the sense of [ 1 ] , remains the only lever. 2 Related Work The work in this paper sits at the intersection of four lines of research: the empirical study of sponsored content in large language models (LLMs), the use of LLMs as automatic judges, the broader literature on persuasion and alignment, and the wider discussion of reproducibility in machine learning. We touch on each in turn. Sponsored content in LLMs. The recent paper by Wu et al. [ 14 ] is the immediate baseline of the present work; it provides both the framework we reproduce and the empirical reference values against which we compare. The mechanism design of LLM advertising – viewed as an auction-and-modification problem – is treated by Feizi et al. [ 3 ] , who study how an ad placement might be priced and presented but do not ask whether the model complies with the sponsorship cue in the first place. Closely related is the work of Salvi et al. [ 10 ] , who showed in a large randomised controlled trial that GPT-4 out-persuades human debaters by a factor of roughly 1.8\times when provided with personalised information about its opponent. Sponsored recommendation, viewed in this light, is simply a special case of this persuasive capacity, with a commercial principal taking the place of a debater. LLM-as-a-judge calibration. Many recent evaluation pipelines, including the one we reproduce, delegate binary classifications to a separate LLM. A recent survey [ 4 ] reviews the rapid adoption of this practice and recommends Cohen’s \kappa and the McNemar test rather than raw correlation. Across application domains, the best LLM judge to date (Llama-3 70B) reaches an agreement of roughly 0.65\text{--}0.70 against human raters on factual cells and falls below 0.20 on more interpretive ones. As we shall see in Section 5 , our own judge ablation reproduces precisely this regime and helps to localise which findings of the original paper are robust to the judge choice and which are not. Persuasion safety, alignment, and the