A Survey on (M)LLM-Based GUI Agents
Fei Tang*, Haolei Xu*, Hang Zhang*, Siqi Chen*, Xingyu Wu*, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, Kaitao Song, Jian Shao, Weiming Lu, Jun Xiao, Yueting Zhuang · 2025
Read A Survey on (M)LLM-Based GUI Agents by Fei Tang*, Haolei Xu*, Hang Zhang*, Siqi Chen*, Xingyu Wu*, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, Kaitao Song, Jian Shao, Weiming Lu, Jun Xiao, Yueting Zhuang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13865.txt.
Description
Fei Tang and a large team of co-authors survey GUI agents based on large language models and multimodal models. Their 2025 survey covers systems that understand and operate graphical user interfaces, a shift from rule-based automation scripts to AI-driven agents.
They sort the field into four core parts. Perception joins text parsing with multimodal understanding of the interface. Exploration builds knowledge from internal models, past experience, and external retrieval. The survey also analyzes architectural foundations, technical components, and evaluation methods. For developers, it maps a fast-moving field and shows where current agents still fall short on complex interface tasks.
Catalog record
| CREATOR | Fei Tang*, Haolei Xu*, Hang Zhang*, Siqi Chen*, Xingyu Wu*, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, Kaitao Song, Jian Shao, Weiming Lu, Jun Xiao, Yueting Zhuang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 117 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-13865