GUI agent: Guided Exploration of User-Sensitive Screens

Aradhana Nayak, Mussadiq Nazeer, Wang Peng, Feng Liu · 2026

Read GUI agent: Guided Exploration of User-Sensitive Screens by Aradhana Nayak, Mussadiq Nazeer, Wang Peng, Feng Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25705.txt.

Description

Phone and desktop agents powered by language models click through apps on a user's behalf. Some screens they meet should never be handled alone: payment confirmations, private messages, deletion prompts. Aradhana Nayak, Mussadiq Nazeer, Wang Peng, and Feng Liu address that exposure in a 2026 workshop paper. Today's agents are tuned to finish tasks whatever the safety stakes, which blocks real deployment. The missing piece is a map of which states count as user sensitive, so an agent can pause and hand control back.

Their explorer agent starts from one demonstrated task and systematically probes the surrounding query space. It hunts for instructions whose execution would land the agent on a sensitive screen. Cataloguing those queries gives engineers raw material for handover rules. The agent would recognise danger states in advance, categorise them, and ask the user before acting, not after damage is done.

Catalog record

CREATORAradhana Nayak, Mussadiq Nazeer, Wang Peng, Feng Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.25705

Browse the full library · About the Tower of Babel