Quick on the Uptake: Eliciting Implicit Intents from Human Demonstrations for Personalized Mobile-Use Agents
Zheng Wu, Heyuan Huang, Yanjia Yang, Yuanyi Song, Xingyu Lou, Weiwen Liu, Weinan Zhang, Jun Wang, Zhuosheng Zhang · 2025
Read Quick on the Uptake: Eliciting Implicit Intents from Human Demonstrations for Personalized Mobile-Use Agents by Zheng Wu, Heyuan Huang, Yanjia Yang, Yuanyi Song, Xingyu Lou, Weiwen Liu, Weinan Zhang, Jun Wang, Zhuosheng Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08645.txt.
Introduction
Zheng Wu and co-authors worked in 2025 on phone agents that learn a user's hidden intent. Such agents copy human taps and swipes to do phone tasks. Past training used only the clear steps a person took, missing private likes and habits.
The team built a data set from forty English and Chinese speaking users to score how well agents match human intent. Their new frame reads clear steps into a library of standard routines, and hidden habits into a store for each user. It then rewrites vague requests into personal ones. Tests showed clear gains over baselines in both intent match and step completion.
Catalog record
| CREATOR | Zheng Wu, Heyuan Huang, Yanjia Yang, Yuanyi Song, Xingyu Lou, Weiwen Liu, Weinan Zhang, Jun Wang, Zhuosheng Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-08645
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Quick on the Uptake: Eliciting Implicit Intents from Human Demonstrations for Personalized Mobile-Use Agents Zheng Wu, Heyuan Huang, Yanjia Yang, Yuanyi Song, Xingyu Lou, Weiwen Liu, Weinan Zhang, Jun Wang, Zhuosheng Zhang Abstract As multimodal large language models advance rapidly, the automation of mobile tasks has become increasingly feasible through the use of mobile-use agents that mimic human interactions from graphical user interface. To further enhance mobile-use agents, previous studies employ demonstration learning to improve mobile-use agents from human demonstrations. However, these methods focus solely on the explicit intention flows of humans (e.g., step sequences) while neglecting implicit intention flows (e.g., personal preferences), which makes it difficult to construct personalized mobile-use agents. In this work, to evaluate the I ntention A lignment R ate between mobile-use agents and humans, we first collect MobileIAR , a dataset containing human-intent-aligned actions and ground-truth actions. This enables a comprehensive assessment of the agents’ understanding of human intent. Then we propose IFRAgent , a framework built upon I ntention F low R ecognition from human demonstrations. IFRAgent analyzes explicit intention flows from human demonstrations to construct a query-level vector library of standard operating procedures (SOP), and analyzes implicit intention flows to build a user-level habit repository. IFRAgent then leverages a SOP extractor combined with retrieval-augmented generation and a query rewriter to generate personalized query and SOP from a raw ambiguous query, enhancing the alignment between mobile-use agents and human intent. Experimental results demonstrate that IFRAgent outperforms baselines by an average of 6.79% (32.06% relative improvement) in human intention alignment rate and improves step completion rates by an average of 5.30% (26.34% relative improvement). The codes are available at https://github.com/MadeAgents/Quick-on-the-Uptake. Introduction Figure 1: Comparing IFRAgent with normal mobile-use agents and existing demonstration learning methods for mobile-use agents. IFRAgent considers both explicit intention flow and implicit intention flow, enabling it to capture more personalized information, such as taste preferences. As multimodal large language models advance rapidly ( Zhang et al. 2024b ; Yin et al. 2024 ) , the automation of mobile tasks has become increasingly feasible through the use of mobile-use agents that mimic human interactions (e.g., clicking, scrolling) from graphical user interface (GUI) ( Zhang et al. 2024a ; Liu et al. 2025b ) . To further improve capabilities of mobile-use agents, some works ( Liu et al. 2025a ; Verma et al. 2024 ) employ demonstration learning to enable mobile-use agents to know how to complete task from human demonstrations. However, as show in Figure 1 , existing demoinstration learning method for mobile-use agents focus solely explicit intention flows of humans (e.g., operational logic, step sequences) to help mobile-use agents learn how to complete task. Moreover, user instructions in the real world are often ambiguous ( Cheng et al. 2025b ) and user-specific, requiring mobile-use agents to understand the implicit intention flows of humans in order to align with human intentions. There are two challenges for mobile-use agents to align with human intentions: (i) There is a lack of datasets or benchmarks that can assess the alignment level between mobile-use agents and human intentions. (ii) Fine-tuning a separate mobile-use agent for each user to create user-specific mobile-use agents is impractical. For challenge (i), we first collect MobileIAR , a user-specific dataset that supports both English and Chinese language users, designed to assess the I ntention A lignment R ate between mobile-use agents and humans. As shown in Table 1 , the dataset contains 16 apps and 945 instuction spanning 7 daily scenarios. And the dataset provides both the human-intent-aligned actions and ground-truth action lists, enabling a comprehensive assessment of the alignment level between mobile-use agents and human intentions. For challenge (ii), we propose IFRAgent , a framework built upon I ntention F low R ecognition from human demonstrations. The IFRAgent consists of the intention flow extraction phase and the deployment phase. In the intention flow extraction phase, IFRAgent analyzes explicit intention flows from human demonstrations to construct a query-level vector library of standard operating procedures (SOP), and analyzes implicit intention flows to build a user-level habit repository. In the deployment phase, IFRAgent, as a plug-and-play module, leverages a SOP extractor combined with retrieval-augmented generation (RAG) ( Lewis et al. 2020 ) and a warmed-up query rewriter to rewrite the user’s ambiguous query into a user-specific personalized query and a personalized SOP based on the analysis of the human intention flow from the previous phase, thereby enabling mobile-use agents to align with human intentions. Extensive experiments spanning diverse mobile-use agents (supporting multiple user languages and are constructed with varying methods) demonstrate that IFRAgent outperforms baseline methods by an average of 6.79% (32.06% relative improvement) in intent alignment rate and achieves a 5.30% in task completion rate (26.34% relative improvement). We also find that general-domain models (e.g., Qwen2.5-VL-7B, GPT-4o) demonstrate more significant improvements compared to specialized mobile-use agent base models like UI-TARS. We further validate IFRAgent’s capability and generalizability through ablation study, cross-dataset tests, comparative studies with other methods, and scale analysis. In summary, we make three key contributions: (i) We contribute and open-source MobileIAR, a dataset containing both user-intent-aligned actions and ground-truth action lists. This dataset not only reflects traditional metrics such as task success rate but also measures the alignment between mobile-use agents and user intent. It establishes the first benchmark for user-specific intent alignment testing in the field of mobile-use agents. (ii) We propose IFRAgent, a plug-and-play framework that leverages both explicit and implicit intention flows from human demonstrations to enhance mobile-use agents’ task completion capability and user-specific intent alignment. (iii) Through extensive experiments on different mobile-use agents, we demonstrate that IFRAgent achieves an average improvement of 6.79% (32.06% relative improvement) in intent alignment rate and 5.30% in step-wise success rate (26.34% relative improvement) over baseline methods. Dataset # Inst. # Apps # Step HL GT FS US PixelHelp 187 4 4.2 ✓ ✓ ✗ ✗ MoTIF 276 125 4.5 ✓ ✓ ✗ ✗ UIBert 16,660 - 1 ✗ ✓ ✗ ✗ Meta-GUI 1,125 11 15 ✓ ✓ ✗ ✗ UGIF 523 12 6.3 ✓ ✓ ✗ ✗ AITW 30,378 357 6.5 ✓ ✓ ✗ ✗ AITZ 2,504 70 7.5 ✓ ✓ ✗ ✗ AndroidControl 15,283 833 4.8 ✓ ✓ ✗ ✗ AMEX 2,946 110 12.8 ✓ ✓ ✗ ✗ OS-Kairos 1000 12 5.1 ✓ ✓ ✗ ✗ MobileAgentBench 100 10 - ✓ ✗ ✗ ✗ AppAgent 50 10 - ✓ ✗ ✗ ✗ LlamaTouch 496 57 7.0 ✓ ✓ ✗ ✗ AndroidWorld 116 20 - ✓ ✗ ✗ ✗ AndroidLab 138 9 8.5 ✓ ✗ ✗ ✗ LearnGUI 2353 73 13.2 ✓ ✓ ✓ ✗ IFRAgent 945 16 7.7 ✓ ✓ ✓ ✓ Table 1: Comparative analysis of datasets and environments for evaluating mobile-use agents. Key metrics include: # Inst. (instruction count), # Apps (application count), # Step (average steps per task), HL (high-level instructions), GT (ground truth trajectories), FS (few-shot learning support), and US (user-specific demonstrations). Data for this table is partially sourced from Liu et al. 2025a . Related work Mobile-use agents for mobile automation Mobile-use agents ( Wang et al. 2024b ; Hu et al. 2025 ) can automatically perform mobile tasks by using the GUI to simulate human interactions (e.g., clicking, scrolling). Researchers mainly adopt two approaches