Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces

William Lugoloobi, Samuelle Marro, Jabez Magomere, Joss Wright, Chris Russell · 2026

Read Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces by William Lugoloobi, Samuelle Marro, Jabez Magomere, Joss Wright, Chris Russell (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14786.txt.

Introduction

Lugoloobi, Marro, Magomere, Wright and Russell ask whether websites can passively identify which language model powers a browsing agent. Across fourteen frontier models and four web environments, an agent's actions and interaction timings, caught by a passive tracker, identify the model with up to ninety-six percent F1. Classifiers generalise across model sizes and families and need only few traces. Randomised timing delays degrade performance, but retraining on delayed traces largely restores it.

Catalog record

CREATORWilliam Lugoloobi, Samuelle Marro, Jabez Magomere, Joss Wright, Chris Russell
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58208
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.14786

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces William Lugoloobi, Samuelle Marro, Jabez Magomere, Joss Wright, Chris Russell Abstract As LLM-based agents increasingly browse the web on users’ behalf, a natural question arises: can websites passively identify which underlying model powers an agent? Doing so would represent a significant security risk, enabling targeted attacks tailored to known model vulnerabilities. Across 14 frontier LLMs and four web environments spanning information retrieval and shopping tasks, we show that an agent’s actions and interaction timings, captured via a passive JavaScript tracker, are sufficient to identify the underlying model with up to 96% F1. We formalise this attack surface by demonstrating that classifiers trained on agent actions generalise across model sizes and families. We further show that strong classifiers can be trained from few interaction traces and that agent identity can be inferred early within an episode. Injecting randomised timing delays between actions substantially degrades classifier performance, but does not provide robust protection: a classifier retrained on delayed traces largely recovers performance. We release our harness and a labelled corpus of agent traces here . 1 Introduction LLM-based agents that browse the web and operate computer interfaces on behalf of users are moving rapidly from research prototypes to production [ 27 , 44 , 40 ] . As these systems are deployed at scale across live websites, every page they visit becomes a potential observation point, and we show that observation alone is enough to identify the model. This exposes users to potential security risks. Figure 1 : Overview of our trace collection and threat model. Using a JavaScript tracker, we collect actions performed by a browsing agent and train a classifier to predict model identity. Every agent visit to a website leaves a trace of clicks, scrolls, keypresses, and other actions observable to any party controlling the page. Prior work has established that behavioural traces like this distinguish human users from automated clients [ 19 , 1 ] , and that passively collected browser attributes re-identify human users across sessions [ 10 , 39 ] . These works ask a binary question: human or bot. As LLM-based agents displace scripted automation, we probe deeper: given that the client is an agent, which model is pulling the strings? In classical security frameworks, target identification is the first step toward exploitation [ 18 ] . Thus for LLM-based agents, knowing the underlying model enables more targeted attacks: an adversary can select from a known set of model-specific jailbreaks or reduce the search space for a white-box adversarial attack [ 29 ] . To our knowledge, we are the first to show that the underlying foundation model of a browser agent can be inferred from passive in-page UI traces alone. Using lightweight classifiers trained on UI action traces collected via injected JavaScript, we achieve agent identification F1’s of up to 96% across 14 frontier LLMs. The identification does not rely on browser attributes and headers (which can be spoofed), but on the temporal and structural dynamics of how different models navigate, click, and interact with page elements. In other words, the behaviour of a model is sufficient to accurately identify it. For adversaries, this means that agents can be identified and exploited. The main contributions of this paper are as follows: • Agent actions are a fingerprint of model identity. We demonstrate for the first time that the on-page actions of LLM browser agents encodes the identity of the underlying model, achieving up to 96% classification Macro F1 across 14 frontier models using only behavioural traces collected via passive JavaScript injection. • A formalised threat model with defence analysis. We characterise agent fingerprinting under a passive co-located adversary, and show that the attack is practical to maintain: new models can be enrolled by routing a small number of sessions through the instrumented site. We further show that standard browser normalisation with randomised delays is insufficient to remove the identifying signal when the adversary retrains their classifier on delayed traces. • Resources for agent fingerprinting. We release a labelled corpus of agent interaction traces across four web environments and a browser harness compatible with both closed and open-source LLMs, enabling reproducible research into behavioural attribution of LLM agents. 2 Related Work LLM-based web agents. Autonomous agents that combine language models with browser automation have rapidly moved from research prototypes to production systems [ 27 , 44 , 40 ] . Benchmarks such as WebArena [ 46 ] and Mind2Web [ 8 ] have established standard evaluation environments for these systems. As these agents are deployed at scale, the question of whether a site operator can determine which model is visiting becomes practically consequential for access control, content delivery, and adversarial exploitation. Bot detection and browser fingerprinting. A large body of work studies how to distinguish automated from human web traffic. Early approaches relied on request-pattern heuristics and crawl detection [ 20 , 17 , 13 , 23 , 22 , 9 ] . Subsequent work has shown that fine-grained behavioural signals, such as mouse movements and interaction timing encode neuromotor structure that can reliably separate humans from bots [ 12 , 41 , 1 , 6 ] . More recent systems combine behavioural features with server-side logs to detect increasingly sophisticated bots that mimic realistic browser fingerprints [ 19 ] . In parallel, browser fingerprinting research has demonstrated that passively collected client-side attributes (e.g., canvas, fonts, WebGL) can be combined into persistent identifiers at scale [ 10 , 39 ] . Across these lines of work, the problem is typically framed as binary classification: human versus bot. We instead consider a finer-grained setting: given that the user is an LLM-based agent, can we identify which model produced the interaction trace solely from its actions? Side-channel attacks and traffic fingerprinting. Side-channel attacks show that systems leak sensitive information through correlated observables, even when their outputs do not [ 21 ] . Applied to web traffic, deep learning classifiers trained on encrypted packet sequences identify visited websites with over 98% accuracy [ 32 , 36 ] . 7 draw a useful distinction between on-path attackers, who observe network traffic from a separate machine, and co-located attackers, whose code runs on the same machine as the victim. Our attack is co-located: we inject JavaScript trackers into the page and collect action traces directly, without any network-level visibility. LLM and agent fingerprinting. 29 fingerprint LLM-integrated applications by sending crafted queries and analysing responses, achieving over 95% accuracy across 42 model versions. Beyond the result, they demonstrate that knowing the underlying model enables targeted attacks: an adversary who can identify the model can craft inputs that exploit model-specific behaviours, biases, or known failure modes, turning fingerprinting from a reconnaissance step [ 18 , 26 , 48 ] into an attack primitive. Closest to our setting, 45 show that task-specific LLM agent applications leave distinct network traffic fingerprints. Their attack observes packet-level metadata generated by agent tool use, and uses it to infer behaviours, application identity, and downstream user attributes. In contrast, we study model attribution from ordinary UI events generated while an agent browses a website. Because all agents in our experiments share the same browser harness and action

Read the complete text