Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshots DOI: XXXXXXX.XXXXXXX Journal: IMWUT Volume: 37 4 111 8 CCS: Human-centered computing Ubiquitous and mobile computing systems and tools

Shiquan Zhang, Tianyi Zhang, Le Fang, Simon D’Alfonso, Hong Jia, Vassilis Kostakos · 2026

Read Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshots DOI: XXXXXXX.XXXXXXX Journal: IMWUT Volume: 37 4 111 8 CCS: Human-centered computing Ubiquitous and mobile computing systems and tools by Shiquan Zhang, Tianyi Zhang, Le Fang, Simon D’Alfonso, Hong Jia, Vassilis Kostakos (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17817.txt.

Description

Shiquan Zhang and five colleagues introduce DailyDroid, covering 75 tasks across 25 Android apps in five scenarios at three difficulty levels, mimicking everyday smartphone use. Tests on GPT-4o and o4-mini across 300 trials compare text-only with multimodal inputs, finding comparable performance with multimodal inputs marginally ahead. In-depth failure analysis yields a handbook of common failures, pointing to UI accessibility, input modalities, and LLM and app design as critical issues. Lessons follow for future agents, apps, and UI work.

Catalog record

CREATORShiquan Zhang, Tianyi Zhang, Le Fang, Simon D’Alfonso, Hong Jia, Vassilis Kostakos
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77175
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17817

Browse the full library · About the Tower of Babel