DigitalCoach : Communication and Grounding Gaps in Human and Agentic Computer Use Coaching
Meng Chen, Anya Ji, Tsung-Han Wu, Tobias Maringgele, David M. Chan, Alane Suhr, Amy Pavel · 2026
Read DigitalCoach : Communication and Grounding Gaps in Human and Agentic Computer Use Coaching by Meng Chen, Anya Ji, Tsung-Han Wu, Tobias Maringgele, David M. Chan, Alane Suhr, Amy Pavel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31980.txt.
Introduction
Chen and six colleagues introduce DigitalCoach, a dataset for studying whether AI can teach people to use software, not just operate it for them. The dataset holds 72 expert-novice coaching sessions across five applications. It contains 22,752 dialogue turns grounded in 28.1 hours of screen and input recordings, with utterances annotated by dialogue act and coaching method.
State-of-the-art models coach differently from humans in the comparisons. They give more direct instructions, over 45 percent of turns against 30 percent for humans, and fewer explanations, diagnoses and checking questions. Given the coaching method, the best model writes plausible guidance, yet its grounding in the learner's actual screen stays poor. Learners coached by models finished fewer milestones and retained fewer skills in live tests.
Catalog record
| CREATOR | Meng Chen, Anya Ji, Tsung-Han Wu, Tobias Maringgele, David M. Chan, Alane Suhr, Amy Pavel |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.31980
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DigitalCoach : Communication and Grounding Gaps in Human and Agentic Computer Use Coaching Meng Chen, Anya Ji, Tsung-Han Wu, Tobias Maringgele, David M. Chan, Alane Suhr, Amy Pavel Abstract Agents are increasingly capable of automating software tasks, but can they teach humans how to use software themselves? We introduce DigitalCoach , a multimodal dataset of 72 human expert-novice computer use coaching sessions consisting of 22,752 dialogue turns grounded in 28.1 hours of screen and input event recordings across 5 software applications. We then use DigitalCoach to evaluate whether state-of-the-art models can teach humans how to use computers. Our automated evaluation shows that models differ from humans in how they coach: models provide more direct instructions, but fewer explanations, error diagnoses and knowledge-check questions. When we fix the coaching method, models produce utterances that are similar to human references, but poorly grounded in visual context. Our interactive evaluation confirms that model coaches cause learners to passively follow instructions without deeper engagement and fall short in visual grounding. DigitalCoach lays a foundation for collaborative and proactive computer use coaching agents. Data and code are available at https://project-digital-coach.vercel.app . Figure 1: An example from DigitalCoach . Human coach provides gives guidance grounded in user’s screen and with explanations while model coach gives instructions without explaining how or why. 1 Introduction Professional software tools for creativity ( e.g. , Blender), engineering ( e.g. , OnShape), and analysis ( e.g. , Excel) are powerful but hard to learn. Agents now let novices use natural language instructions to easily execute creativity, engineering, and analysis tasks end-to-end without using the graphical user interfaces (GUIs) of the software 48 ; 55 ; 19 . However, GUIs remain important to professionals as they afford expressivity, precision, and speed that cannot be achieved with natural language alone. For example, skilled Blender users can use the hotkey “G” then drag an object to position it, rather than using natural language to describe what object to move and where to move it. For novices who want to develop such software expertise, the automation afforded by agents bypasses learning. Thus, we explore a new role for agents: teaching novices the skills they need to use these interfaces effectively. Novices traditionally learn from expert demonstrations in webpage or video tutorials, but these materials are one-size-fits-all and require learners to manually align their own progress with the tutorial. To counter this, Human-Computer Interaction (HCI) research has prototyped systems 21 ; 26 that transform tutorials into task assistants. But can current models generate adaptive, grounded guidance that supports skill acquisition beyond task completion? We introduce DigitalCoach , a dataset for (1) characterizing multimodal language-based interactions during human expert–novice computer use coaching and (2) analyzing how well models emulate human expert coaching. DigitalCoach contains 72 computer use coaching sessions of 40 participants spanning creativity, engineering, and productivity domains. DigitalCoach has 22,752 turns dialogues grounded in 28.1 hours screen recordings, 39,609 input events, and 36,724 file snapshots. To capture both the communicative and pedagogical functions of language use, we annotate utterances with dialogue acts and coaching methods. We then evaluate 6 state-of-the-art models as computer use coaching agents using DigitalCoach . Results show that model utterances differ distributionally from human utterances and are less lexically and semantically diverse. Compared to human coaches, models predominantly use direct instruction ( > 45% of turns of all models vs. 30% for humans). They provide less feedback on the screen state and ask fewer questions. The best model (Gemini-3.1-Pro) can generate plausible local guidance (CLAIR = 41.4) similar to human references when knowing the coaching method. Yet, current models all rely more on textual history than on the learner’s screen state as models perform similarly without visual context, but their scores drop substantially without dialogue context. In real-time interaction, learners coached by models completed fewer milestones and retained fewer skills. We show that models still struggle to provide guidance grounded in evolving screen states and rarely offer reusable knowledge that helps learners accomplish similar tasks independently in the future. These findings suggest that effective computer use coaching agents require adaptive combinations of instruction and feedback based on the learner’s evolving progress and opportunities for transferable skill development. In summary, we contribute: • DigitalCoach , a 28.1 hour human expert-novice GUI-grounded dialogue dataset with fine-grained annotations of dialogue acts and coaching methods. • Quantitative and qualitative characterizations distinguishing human and model coaching. • Expert rating and interactive human evaluations revealing communication and grounding gaps in real-time model coaching. 2 Related Work Dataset Domain Source Setup Duration # Turns Data # Ppl. Relationship Text Audio Video Action File Portal Dialogue Corpus Game Real 2 Collaborator 11.5h 24.5k ✓ ✓ ✓ ✗ ✓ MathDial Education Semi-synthetic 1 Expert-Novice – 28.3k ✓ ✗ ✗ ✗ ✗ VideoCAD Professional Synthetic 0 – 2.1kh – ✗ ✗ ✓ ✓ ✗ AssistGUI Professional Real 1 – < 8.3h – ✗ ✗ ✓ ✓ ✗ RealWebAssist Everyday Real 1 – < 6h – ✓ ✓ ✓ ✗ ✗ GUIDE Professional Real 1 – 67.5h – ✓ ✓ ✓ ✗ ✗ MixAssist Professional Real 2 Expert-Novice 7h 431 ✓ ✓ ✗ ✗ ✗ DigitalCoach Professional Real 2 Expert-Novice 28.1h 22.7k ✓ ✓ ✓ ✓ ✓ Table 1: Compared to related datasets 46 ; 33 ; 35 ; 18 ; 54 ; 52 ; 12 , DigitalCoach captures real human–human computer use coaching with much richer multimodal data. Language Use in Situated Collaboration. Prior work has studied how agents ground and coordinate through language under partial observability and differing task knowledge 4 ; 5 ; 31 ; 41 ; 45 ; 8 ; 47 . More recent work moves toward richer settings, such as video games, household tasks, and collaborative drawing 56 ; 46 ; 38 ; 1 ; 9 ; 29 . However, no existing collaborative dialogue dataset is grounded in GUI workflows where people seek and provide help via multimodal communication ( e.g., spatial referencing). Our dataset fills this gap by capturing expert-novice multimodal communication during GUI tasks. Pedagogical and Instructional Dialogue. Unlike transactional dialogue systems that focus on task completion and intent tracking, such as making reservations 20 or tool use 53 , pedagogical dialogue emphasizes on how systems support learning through feedback, questioning, and scaffolding 14 . Prior datasets such as MathDial 33 , MathTutorBench 34 , and ConvoLearn 42 evaluate tutoring quality in domains including mathematics and geography, while MixAssist captures expert–amateur collaboration in music production workflows 12 . However, these datasets are either text-based or grounded in static content, and none capture instruction grounded in an evolving visual environment. DigitalCoach fills this gap with multimodal expert-novice interactions grounded in real-time GUI state and user actions. Collaborative GUI Agents. GUI agent research primarily studies whether agents can interpret interface state and execute actions to complete user goals. Benchmarks such as WebArena and OSWorld evaluate agents on web and desktop tasks using screenshots, natural language goals, and interface actions 57 ; 50 . More recent work extends to screen recordings that capture software interactions over time 35 ; 32 ; 28 . However, existing GUI agent benchmarks primarily focus on autonomous task completion rather than collaboration with users during analytic and creative workflows where user agency matters 43