A Survey on (M)LLM-Based GUI Agents

Fei Tang*, Haolei Xu*, Hang Zhang*, Siqi Chen*, Xingyu Wu*, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, Kaitao Song, Jian Shao, Weiming Lu, Jun Xiao, Yueting Zhuang · 2025

Read A Survey on (M)LLM-Based GUI Agents by Fei Tang*, Haolei Xu*, Hang Zhang*, Siqi Chen*, Xingyu Wu*, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, Kaitao Song, Jian Shao, Weiming Lu, Jun Xiao, Yueting Zhuang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13865.txt.

Description

Fei Tang and a large team of co-authors survey GUI agents based on large language models and multimodal models. Their 2025 survey covers systems that understand and operate graphical user interfaces, a shift from rule-based automation scripts to AI-driven agents.

They sort the field into four core parts. Perception joins text parsing with multimodal understanding of the interface. Exploration builds knowledge from internal models, past experience, and external retrieval. The survey also analyzes architectural foundations, technical components, and evaluation methods. For developers, it maps a fast-moving field and shows where current agents still fall short on complex interface tasks.

Catalog record

CREATORFei Tang*, Haolei Xu*, Hang Zhang*, Siqi Chen*, Xingyu Wu*, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, Kaitao Song, Jian Shao, Weiming Lu, Jun Xiao, Yueting Zhuang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE117 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13865

Browse the full library · About the Tower of Babel