Preference Redirection via Attention Concentration: An Attack on Computer Use Agents

Dominik Seip & Matthias Hein · 2026

Read Preference Redirection via Attention Concentration: An Attack on Computer Use Agents by Dominik Seip & Matthias Hein (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08005.txt.

Description

Seip and Hein introduce PRAC, an attack on computer use agents that shifts focus rather than aiming at model output. These agents work with visual interfaces, doing complex tasks but opening new security holes. Prior work focused on the language modality, leaving vision less studied. PRAC shifts the model's internal preferences by steering focus toward a stealthy adversarial patch. The team shows it can push a shopping agent's choice toward a chosen target product. White-box access is needed, marking a limit on its real-world reach.

Catalog record

CREATORDominik Seip & Matthias Hein
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE127016
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.08005

Browse the full library · About the Tower of Babel