Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshots DOI: XXXXXXX.XXXXXXX Journal: IMWUT Volume: 37 4 111 8 CCS: Human-centered computing Ubiquitous and mobile computing systems and tools
Shiquan Zhang, Tianyi Zhang, Le Fang, Simon D’Alfonso, Hong Jia, Vassilis Kostakos · 2026
Read Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshots DOI: XXXXXXX.XXXXXXX Journal: IMWUT Volume: 37 4 111 8 CCS: Human-centered computing Ubiquitous and mobile computing systems and tools by Shiquan Zhang, Tianyi Zhang, Le Fang, Simon D’Alfonso, Hong Jia, Vassilis Kostakos (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17817.txt.
Description
Shiquan Zhang and five colleagues introduce DailyDroid, covering 75 tasks across 25 Android apps in five scenarios at three difficulty levels, mimicking everyday smartphone use. Tests on GPT-4o and o4-mini across 300 trials compare text-only with multimodal inputs, finding comparable performance with multimodal inputs marginally ahead. In-depth failure analysis yields a handbook of common failures, pointing to UI accessibility, input modalities, and LLM and app design as critical issues. Lessons follow for future agents, apps, and UI work.
Catalog record
| CREATOR | Shiquan Zhang, Tianyi Zhang, Le Fang, Simon D’Alfonso, Hong Jia, Vassilis Kostakos |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77175 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.17817