Machine-Readable Ads: Accessibility and Behavioral Patterns of AI Web Agents interacting with Online Advertisements

Joel Nitu, Heidrun Mühle, Andreas Stöckl · 2025

Read Machine-Readable Ads: Accessibility and Behavioral Patterns of AI Web Agents interacting with Online Advertisements by Joel Nitu, Heidrun Mühle, Andreas Stöckl (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12844.txt.

Introduction

Nitu, Mühle, and Stöckl test how AI web agents handle online advertising in this 2025 study. Agents that browse and buy for users meet pages designed for human eyes. The team built a faithful clone of a news site and filled it with varied ads.

They ran 300 trials plus follow-ups across four agent systems and ten realistic tasks. The ads included banners, animations, carousels, videos, cookie dialogs, and paywalls. Agents barely scrolled and ignored purely visual calls to action, clicking only when text labels or button overlays gave meaning. Worryingly, two systems subscribed in every trial when a sweepstake required purchase. The authors distill five design principles that make ads readable to machines without hurting human experience.

Catalog record

CREATORJoel Nitu, Heidrun Mühle, Andreas Stöckl
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-12844

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Machine-Readable Ads: Accessibility and Behavioral Patterns of AI Web Agents interacting with Online Advertisements Joel Nitu, Heidrun Mühle, Andreas Stöckl Abstract Autonomous multimodal language models are rapidly evolving into web agents that can browse, click, and purchase items on behalf of users—posing a threat to display advertising designed for human eyes. Yet little is known about how these agents interact with ads or which design principles ensure reliable engagement. To address this, we ran a controlled experiment using a faithful clone of the news site TT.com, seeded with diverse ads: static banners, GIFs, carousels, videos, cookie dialogues, and paywalls. We ran 300 initial trials plus follow-ups using the Document Object Model (DOM)-centric Browser Use framework with GPT-4o, Claude 3.7 Sonnet, Gemini 2.0 Flash, and the pixel-based OpenAI Operator, across 10 realistic user tasks. Our results show these agents display severe satisficing: they never scroll beyond two viewports and ignore purely visual calls to action, clicking banners only when semantic button overlays or off-screen text labels are present. Critically, when sweepstake participation required a purchase, GPT-4o and Claude 3.7 Sonnet subscribed in 100% of trials, and Gemini 2.0 Flash in 70%, revealing gaps in cost-benefit analysis. We identified five actionable design principles—semantic overlays, hidden labels, top-left placement, static frames, and dialogue replacement—that make human-centric creatives machine-detectable without harming user experience. We also evaluated agent trustworthiness through “behavior patterns” such as cookie consent handling and subscription choices, highlighting model-specific risk boundaries and the urgent need for robust trust evaluation frameworks in real-world advertising. K eywords Autonomous agents \cdot Intelligent agents \cdot Multimodal language models \cdot Online advertising \cdot Trustworthy AI 1 Introduction Autonomous artificial intelligence (AI) agents—multimodal large language models that process both text and images—are set to transform online search and commerce by autonomously navigating websites, extracting structured data, and executing transactions for users [ 1 ] . An April 2024 CMSWire report commissioned by Gartner predicts that by 2026, up to 25% of searches will be delegated to such AI assistants instead of being entered into traditional engines [ 2 ] . If users stop issuing direct queries, display advertising must be redesigned to also account for algorithmic consumption, as banners and pop-ups—created for human attention—lose effectiveness when agents parse the DOM for semantically rich cues. In a prior controlled simulation on a travel-booking prototype, the Browser Use framework 1 1 1 https://github.com/browser-use —a state-of-the-art agentic LLM framework—was used to test three leading multimodal models: GPT-4o 2 2 2 https://openai.com/index/hello-gpt-4o/ , Claude 3.7 Sonnet 3 3 3 https://www.anthropic.com/claude/sonnet , and Gemini 2.0 Flash 4 4 4 https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-0-flash . Across all agents, a consistent preference was observed for clearly marked factual data, such as price or location, over visual or emotional cues when selecting offers [ 1 ] . These findings confirm that AI agents will engage with online advertisements whenever doing so supports their assigned goal. This follow-up study investigates two complementary questions: first, which page elements state-of-the-art AI agents can actually perceive and manipulate—what they read, where they can click, and which controls remain invisible; and second, once interaction occurs, how responsibly these agents behave—whether they respect privacy prompts, safeguard payment details, and avoid unintended sign-ups in the advertising-heavy environment of modern websites. Evaluating the trustworthiness of LLM-powered agents requires a multi-faceted approach that integrates behavioral models, systemic analyses, and normative guidelines. At the behavioral level, frameworks like the Trustworthiness Assessment Model (TrAM) from Schlicker et al. help explain trust violations by identifying mismatched user expectations [ 3 ] . The systemic context proves equally crucial, as research by Gao et al. demonstrates that the inherent opacity of AI-driven ad personalization systematically erodes consumer trust [ 4 ] . These behavioral and systemic insights are complemented by normative benchmarks such as the EU AI Act and the EU High-Level Expert Group’s Ethics Guidelines for Trustworthy AI, which provide a conceptual reference frame by articulating seven requirements for responsible systems [ 5 , 6 ] . To address our research questions about agent interaction capabilities and responsible behavior, we built a faithful clone of the Tiroler Tageszeitung 5 5 5 https://www.tt.com/ homepage including banner, native, and dynamic interactive ads—and assessed each agent’s ability to: • Locate and rank advertisements • Parse dynamic versus static content • Apply promotional offers (e.g., coupon flows) • Complete multi-step user journeys This clone replicates the complex DOM and layout of the original site, providing a realistic yet controlled environment. This high-fidelity approach is crucial for testing agents against the structural complexity of real-world websites, a factor often missing in simpler, purpose-built test environments. Drawing on W3C 6 6 6 https://www.w3.org/ ARIA guidelines and schema.org’s advert vocabulary, we investigated semantic-markup patterns and measured their impact on agent-driven engagement. Our key contributions are: • Empirical demonstration that adding explicit semantic markup—off-screen <span> , ARIA roles, and descriptive alt attributes—can convert previously ignored advertisements into reliably clickable elements for LLM agents. • A qualitative taxonomy of advertisement formats and markup patterns (image carousels, video embeds, banner and in-content ads) along with their relative detectability by GPT-4o, Claude 3.7 Sonnet, and Gemini 2.0 Flash via heavily DOM-focused architectures like Browser Use but also pixel-based agents like OpenAI Operator 7 7 7 https://openai.com/index/introducing-operator/ . • A study of behavioral patterns that charts when modern LLM-powered web agents comply, pause, or decline—spanning scenarios like fabricating sensitive information, deciding whether they may initiate subscriptions, and negotiating consent dialogs—thereby illuminating the ethical constraints on agent autonomy within advertising settings. 2 Related Work Recent literature highlights the emergence of web agents—AI systems powered by multimodal LLMs capable of autonomously performing web-based tasks. Ning et al. provide a comprehensive survey, emphasizing that LLMs bring “human capabilities like memory and planning” to web interaction [ 7 ] . Deng et al. similarly describe a paradigm shift in web applications, wherein LLM-based agents—capable of vision, reasoning, and action—augment user engagement [ 8 ] . Benchmarks such as BrowseComp evaluate these systems across 1 266 tasks, measuring persistence, creativity, and information-gathering efficacy [ 9 ] . Even the strongest models solve around half of BrowseComp tasks, underlining that end-to-end reliability remains an open challenge. Collectively, these studies show that modern agents can navigate complex sites, fill out forms, and complete multi-step tasks, motivating not only task-completion studies but also inquiries into how these agents consume the same advertising infrastructure that funds today’s web. While these benchmarks provide foundational frameworks for general trustworthiness, they often rely on synthetic environments and do not yet offer standardized metrics to quantify agent compliance in the complex and commercially-driven context of live advertising ecosystems, a gap our study aims to explore. 2.1 Architectural Paradigms of AI Web Agents Web-interaction agents fall into two broad paradigms: 1. Vision-based Agents such as OpenAI’s Operator and Anthropic’s Claude 3.5 “computer use” mode treat each page as pixels [ 10 , 11 ] . By recognising visual patterns they can click, type, and scroll—even on image-only buttons—with human-like dexterity. Their drawback is cost: high-resolution screenshots plus heavy vision transformers mean slower inference and greater energy use. 2. DOM-based Frameworks like the open-source Browser Use [ 12 ] , Agent-E 8 8 8 https://github.com/EmergenceAI/Agent-E , and Microsoft’s OmniParser 9 9 9 https://github.com/microsoft/OmniParser parse the Document Object Model,

Read the complete text