Building Browser Agents: Architecture, Security, and Practical Solutions

Aram Vardanyan · 2025

Read Building Browser Agents: Architecture, Security, and Practical Solutions by Aram Vardanyan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19477.txt.

Introduction

Aram Vardanyan reports lessons from building and running a production browser agent, a system that operates a web browser on its own. Such agents promise autonomous web work but face severe reliability and security problems in practice. Vardanyan's central finding is that the model's raw capability is not the bottleneck: architectural decisions decide whether an agent succeeds or fails. Hybrid context handling, mixing accessibility tree snapshots with selective vision, and full tooling that matches human interaction, are among the choices he examines.

Security analysis of real incidents leads to a sharper claim: prompt injection attacks make general-purpose autonomous browsing fundamentally unsafe. He argues against building general browsing intelligence and for specialised tools with programmatic constraints, where safety limits are enforced by code rather than by the model's own reasoning. The paper is a field report for teams shipping agents.

Catalog record

CREATORAram Vardanyan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-19477

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Building Browser Agents: Architecture, Security, and Practical Solutions Aram Vardanyan Abstract Browser agents enable autonomous web interaction but face critical reliability and security challenges in production. This paper presents findings from building and operating a production browser agent. The analysis examines where current approaches fail and what prevents safe autonomous operation. The fundamental insight: model capability does not limit agent performance; architectural decisions determine success or failure. Security analysis of real-world incidents reveals prompt injection attacks make general-purpose autonomous operation fundamentally unsafe. The paper argues against developing general browsing intelligence in favor of specialized tools with programmatic constraints, where safety boundaries are enforced through code instead of large language model (LLM) reasoning. Through hybrid context management combining accessibility tree snapshots with selective vision, comprehensive browser tooling matching human interaction capabilities, and intelligent prompt engineering, the agent achieved approximately 85% success rate on the WebGames benchmark across 53 diverse challenges (compared to approximately 50% reported for prior browser agents and 95.7% human baseline). 1 Introduction Browser agents powered by LLMs automate interaction with web interfaces. Several AI-native browsers and web-agent frameworks report strong performance on selected benchmarks ( Xie et al., 2024 ; Koh et al., 2024 ) and show multi-step workflows such as shopping or travel planning. However, as Section 2 discusses, these systems provide limited evidence about safe and reliable production operation. A gap exists between demonstration and production operation. Findings from building and operating a production browser agent reveal that LLM capability is not the limiting factor in widespread adoption. Modern LLMs have sufficient reasoning ability to navigate web tasks effectively when provided with appropriate context and tools. The fundamental challenges are architectural: how the system manages context, what information the agent accesses, how safety boundaries are enforced, and whether to target generalization or specialization. These challenges reflect a fundamental mismatch. The web was designed for humans, beings with vision, spatial awareness, selective attention, and the ability to process dynamic content including animations, videos, and layered interfaces. Current browser agents can replicate many human interactions such as clicks, typing, and scrolling. However, they struggle with aspects that are straightforward for humans: understanding visual hierarchy, processing time-dependent information, maintaining relevant context while discarding noise, and operating safely in environments where a single wrong action can cause irreversible damage. This paper presents findings from production experience, not laboratory benchmarks. The insights come from observing where agents succeed, where they fail, and what architectural patterns separate reliable systems from demonstrations. Among these challenges, security represents the most immediate barrier to autonomous operation. As Section 2.3 details, public security analyses of AI-native browsers and browser extensions demonstrate that prompt injection attacks remain effective even after multiple rounds of mitigation ( Shapira et al., 2025 ; Cohen, 2025 ) . Studies consistently show non-trivial attack success rates in scenarios where agents have access to sensitive user data across multiple domains. When an agent operates with full user privileges across banking, email, and corporate systems, even a 1% vulnerability rate represents unacceptable risk. Reliable, safe browser agents require better architectural decisions instead of better LLMs. Production experience reveals four insights: context management determines success, architecture matters more than LLM scale, security requires programmatic constraints over LLM-based judgments, and specialization outperforms general-purpose approaches. Each insight challenges conventional assumptions about autonomous web agents. The remainder of this paper examines each of these dimensions in detail, drawing from production experience to provide actionable guidance for building browser agents that are both capable and safe enough for real-world deployment. 2 Background & Related Work 2.1 AI browsers and computer-use APIs A new class of AI browsers integrates large language models (LLMs) directly into the browsing experience. OpenAI’s ChatGPT Atlas, launched in October 2025, embeds ChatGPT as a browser sidebar with an optional Agent Mode capable of performing autonomous tasks such as trip planning or shopping over the user’s tabs ( OpenAI, 2025 ) . Perplexity Comet, announced in July 2025, functions as an AI browser that automates research, email triage, and other workflows from within the interface ( Perplexity AI, 2025 ) . Anthropic’s computer use capability extends Claude models with tools to control a desktop environment and browser, separate from standalone AI browsers. The initial release in October 2024 exposed tools such as click , type , and navigate , with a focus on security mitigations and permission prompts ( Anthropic, 2024b ) . Anthropic later introduced Claude for Chrome, a browser extension that brings this automation into existing sessions. Early reports highlighted productivity benefits and documented remaining prompt injection vulnerabilities ( Anthropic, 2024a ) . Several open-source frameworks target this problem space. WebVoyager introduces a multimodal browser agent that combines page screenshots with text to complete tasks on real websites, achieving approximately 59% task success on the WebVoyager benchmark ( He et al., 2024 ) . The Browser Use framework builds on this work and reports 89.1% success rate on the same benchmark across 586 tasks ( Browser Use Team, 2024 ) . These systems show that LLM-driven browser agents can reach high success rates on static benchmarks, but provide limited visibility into production reliability, long-running workflows, and operational safety at scale. 2.2 Browser automation MCPs and accessibility-based agents The Model Context Protocol (MCP) ecosystem now includes standardized servers for browser automation. Microsoft’s Playwright MCP exposes a headless browser controlled via accessibility tree snapshots instead of raw screenshots. This approach provides the LLM with a structured view of roles, labels, and focusable elements, mapping these to executable actions ( Microsoft, 2024 ) . Chrome DevTools MCP provides access to the Chrome DevTools protocol, including performance traces and full accessibility trees, allowing agents to operate using semantic element references instead of coordinate-based interactions ( Model Context Protocol Team, 2024 ) . These MCP servers illustrate a design pattern that decouples the perception layer (accessibility tree, page snapshots, or screenshots) from the execution layer (tool calls such as click , type , navigate , performance_start_trace ). Accessibility-driven representations simplify element selection and enable safety policy enforcement at the tool layer. The architecture presented in this paper follows this principle but extends it with hybrid vision and accessibility context, element reference versioning, bulk actions, and domain-specific safety constraints derived from production usage. 2.3 Prompt injection and agent security Prompt injection represents the central security concern for tool-using agents. Systematic evaluations across multiple models and attack patterns have demonstrated high success rates, highlighting the need for defense-in-depth approaches ( Liu et al., 2024 ) . Regarding AI browsers, Brave’s security team demonstrated indirect prompt injection attacks against Perplexity Comet. Attackers hid adversarial instructions in elements invisible to the user, such as white text on white backgrounds or HTML comments. These instructions caused Comet to execute sensitive cross-site actions, including fetching one-time passwords from email or accessing banking portals, when the user asked it to "summarize this page" ( Sampayo and Brave Security Team, 2025 ) . Later analyses by security vendors confirmed that such AI browsers bypass traditional browser security boundaries like the same-origin policy by executing cross-domain actions on behalf of the user. Defenses that rely on detecting attacks, such as filtering suspicious inputs or attempting to separate user instructions from untrusted content, typically reduce the probability of attack

Read the complete text