User’s Task

Mykyta Mudryi, Markiyan Chaklosh, Grzegorz Marcin Wójcik · 2025

Read User’s Task by Mykyta Mudryi, Markiyan Chaklosh, Grzegorz Marcin Wójcik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13076.txt.

Introduction

Three authors audit the security of browsing AI agents in this 2025 paper. Autonomous agents that surf the web for users run on shifting page content and live tools, which opens a wide attack surface that had not been mapped end to end.

The team built the first full threat model for such agents and proposed layered defenses: cleaning inputs, isolating the planner from the executor, formal checkers, and session safeguards. A white box study of a popular open source agent, Browser Use, showed how hostile page content can hijack behavior. Their finds include prompt injection, domain check bypass, and credential theft, backed by a disclosed vulnerability report and a working proof of concept.

Catalog record

CREATORMykyta Mudryi, Markiyan Chaklosh, Grzegorz Marcin Wójcik
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13076

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

User’s Task Mykyta Mudryi, Markiyan Chaklosh, Grzegorz Marcin Wójcik Abstract Autonomous browsing agents powered by large language models (LLMs) are increasingly used to automate web-based tasks. However, their reliance on dynamic content, tool execution, and user-provided data exposes them to a broad attack surface. This paper presents a comprehensive security evaluation of such agents, focusing on systemic vulnerabilities across multiple architectural layers. Our work outlines the first end-to-end threat model for browsing agents and provides actionable guidance for securing their deployment in real-world environments. To address discovered threats, we propose a defense-in-depth strategy incorporating input sanitization, planner-executor isolation, formal analyzers, and session safeguards—providing protection against both initial access and post-exploitation attack vectors. Through a white-box analysis of a popular open-source project Browser Use , we demonstrate how untrusted web content can hijack agent behavior and lead to critical security breaches. Our findings include prompt injection, domain validation bypass, and credential exfiltration, evidenced by a disclosed CVE and a working proof-of-concept exploit. The Hidden Dangers of Browsing AI Agents 1 1 footnotetext: These authors contributed equally. 2 2 footnotetext: Correspondence should be addressed to: [email protected] 1 Introduction Recent advancements in Large Language Models (LLMs) have significantly accelerated the development of various autonomous agents capable of executing complex tasks with minimal human intervention. Among these, autonomous and collaborative browsing agents have emerged as particularly compelling due to their ability to navigate the web, interact with web applications, and automate information retrieval. Notable examples of such agents include, but are not limited to, the open-source Browser Use [ 21 ] , OpenAI’s Operator [ 31 ] , and Anthropic’s Computer-Use [ 12 ] . Although each of these systems masssive capabilities, only Browser Use is open source, having garnered significant attention within the research and development communities, and has accumulated over 60,000 stars in its repository as of this publication. This extensive adoption highlights both its potential and the security concerns associated with its widespread use. Given the increasing reliance on autonomous browsing agents for both individual and enterprise applications, identifying and mitigating security vulnerabilities within these systems is of paramount importance. The attack surface of such agents is particularly large, extending beyond the LLM itself to include the underlying web driver, execution environment, and external dependencies. These systems frequently interact with sensitive user data, such as login credentials, session tokens, and API keys, making them attractive targets for adversaries. Furthermore, their ability to perform authentication on behalf of users introduces additional security challenges, as unauthorized credential storage, misuse of session tokens, or impersonation attacks could lead to severe breaches. Research Questions. Informed by the growing adoption, architectural complexity, and emerging security concerns surrounding autonomous LLM-based browsing agents, this study seeks to answer the following key research questions: • RQ1: What are the structural and systemic factors that make open-source autonomous browsing agents, such as Browser Use - susceptible to prompt injection and related attack vectors? • RQ2: Do inherent architectural design choices in browsing AI agents introduce systemic vulnerabilities that adversaries can exploit? • RQ3: How do common agent components (Perception, Reasoning, Planning, Tool Execution) contribute to the feasibility and severity of exploits such as credentials exfiltration, unauthorized task execution, and agent observability? • RQ4: To what extent can current mitigation techniques (e.g., input sanitization, architectural isolation, formal analyzers) reduce the success rate of prompt injection attacks under realistic deployment scenarios? These questions guide our case study, attack taxonomy, mitigation framework and further security assessment of Browser Use, forming the basis for a holistic security evaluation of autonomous LLM-based web agents. In the concluding phase of our security assessment, we demonstrate that these agents are vulnerable to prompt injection attacks, which can be exploited to exfiltrate stored or actively used credentials. By manipulating prompts and leveraging the model’s execution flow, an attacker can trick the agent into revealing sensitive information, ultimately leading to the compromise of multiple user accounts. This attack vector underscores the critical need for robust security frameworks, input sanitization techniques, and fine-grained access control mechanisms in autonomous browsing agents. Our findings emphasize the necessity of a multi-layered security approach, including strict isolation of credentials, adversarial testing methodologies, and real-time anomaly detection to mitigate these risks. As autonomous browsing agents continue to evolve and integrate into mainstream workflows, ensuring their resilience against emerging threats will be crucial in preventing large-scale security incidents. 2 Current State-of-The-Art Browsing Agents 2.1 Introduction AI browsing or web agents are autonomous systems that use Large Language Models (LLMs) to navigate and interact with websites on behalf of a user. They typically perceive web content (through page text or visual renderings) and perform actions such as clicking links, filling forms, or entering text, in order to accomplish user-specified tasks [ 1 , 7 ] . Unlike a standard chatbot, which only produces textual responses, a web agent operates in an iterative sense-plan-act loop [ 3 ] : it observes the state of a webpage, reasons about the next step, executes an action (e.g. clicking a button), then receives the new webpage state, and so on until the task is complete. Recent advances in LLMs (e.g., o3, Gemini 2.5 Pro, Claude 3 Opus) have greatly expanded the capabilities of such agents, enabling complex multi-step tasks like booking travel, shopping, or data extraction across the web [ 19 ] . Web agents hold the promise of automating many workflows by leveraging existing web interfaces. 2.2 Agent Capabilities and Performance Benchmarks Browsing agents have evolved significantly from early simulations to modern evaluations. This timeline highlights key developments that enhance their ability to navigate real-world scenarios. Figure 1: Browsing Agents Benchmark Timeline Early efforts toward web agents often evaluated them in simplified environments. For instance, the World of Bits/MiniWoB simulators [ 34 ] provided basic web-like UIs for agents to practice navigation in a controlled setting, but with limited realism. More recently, researchers introduced benchmarks like Mind2Web [ 5 ] and WebArena [ 11 ] to better measure web agent abilities. Mind2Web is an offline dataset of web tasks (compiled as textual descriptions and human action traces) on which agents are evaluated by how well they can follow a “golden” action sequence for each task. This step-by-step evaluation, however, covers only one correct strategy per task and may not capture the full flexibility of an agent. WebArena, on the other hand, provides a realistic browser environment (simulating real websites with an accessible DOM) where agents can be tested online. It introduced more complex, dynamic web tasks than prior simulators. Still, these tasks were typically evaluated on textual page representations (the DOM or accessibility tree) and with metrics focusing on task success, without fully open-ended interaction. To push toward real-world applications, recent work has moved to evaluating agents directly on live websites. WebVoyager [ 19 ] is a notable end-to-end web agent that operates a real web browser via Selenium, taking screenshots and DOM information as input. The authors compiled a new benchmark of 300 user tasks across 15 popular websites (including e-commerce, email, flight booking, etc.), designed to test an agent’s ability to complete goals like: • “find the price of a 2-year warranty for a product on Amazon” • “delete an email from a specific sender” Unlike Mind2Web’s fixed

Read the complete text