“When to Hand Off, When to Work Together”: Expanding Human-Agent Co-Creative Collaboration through Concurrent Interaction DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Human-centered computing Interactive systems and tools
Kihoon Son, Hyewon Lee, DaEun Choi, Yoonsu Kim, Tae Soo Kim, Yoonjoo Lee, John Joon Young Chung, HyunJoon Jung, Juho Kim · 2026
Read “When to Hand Off, When to Work Together”: Expanding Human-Agent Co-Creative Collaboration through Concurrent Interaction DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Human-centered computing Interactive systems and tools by Kihoon Son, Hyewon Lee, DaEun Choi, Yoonsu Kim, Tae Soo Kim, Yoonjoo Lee, John Joon Young Chung, HyunJoon Jung, Juho Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-02050.txt.
Introduction
Kihoon Son, Hyewon Lee, DaEun Choi, and colleagues design AI agents that collaborate with designers in real time. Their 2026 work notes that human partners coordinate through shared visibility of process and workspace. Agents, by contrast, usually deliver final outputs or read-only process views and ignore concurrent user moves.
A first study with professional designers shows process visibility helps, yet conflicts arise when agents misread feedback. Their system, Cleo, interprets collaborative intent from concurrent actions on shared artifacts and adapts live. A second two-day study analyzes hundreds of turns for action patterns and triggers. Designers split their work among delegation, direction, and concurrent co-work in shifting shares. A decision model with six interaction loops distills when to hand off and when to work together.
Catalog record
| CREATOR | Kihoon Son, Hyewon Lee, DaEun Choi, Yoonsu Kim, Tae Soo Kim, Yoonjoo Lee, John Joon Young Chung, HyunJoon Jung, Juho Kim |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 115425 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.02050
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
“When to Hand Off, When to Work Together”: Expanding Human-Agent Co-Creative Collaboration through Concurrent Interaction DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Human-centered computing Interactive systems and tools Kihoon Son, Hyewon Lee, DaEun Choi, Yoonsu Kim, Tae Soo Kim, Yoonjoo Lee, John Joon Young Chung, HyunJoon Jung, Juho Kim Abstract . Human collaborators coordinate dynamically through process visibility and workspace awareness, yet AI agents typically either provide only final outputs or expose read-only execution processes (e.g., planning, reasoning) without interpreting concurrent user actions on shared artifacts. Building on mixed-initiative interaction principles, we explore whether agents can achieve collaborative context awareness—interpreting concurrent user actions on shared artifacts and adapting in real-time. Study 1 (N=10 professional designers) revealed that process visibility enabled reasoning about agent actions but exposed conflicts when agents could not distinguish feedback from independent work. We developed Cleo , which interprets collaborative intent and adapts in real-time. Study 2 (N=10, two-day with stimulated recall interviews) analyzed 214 turns, identifying five action patterns, six triggers, and four enabling factors explaining when designers choose delegation (70.1%), direction (28.5%), or concurrent work (31.8%). We present a decision model with six interaction loops, design implications, and an annotated dataset. 1. Introduction In human–human collaborative creative work, as commonly observed in shared tools such as Figma 1 1 1 https://www.figma.com or Google Slides 2 2 2 https://docs.google.com/presentation , visibility into a collaborator’s actions and awareness of who is working on what are fundamental for effective coordination ( 11 ) . Collaborators observe each other’s ongoing work, anticipate next steps, intervene at opportune moments, and fluidly divide tasks based on emerging priorities and expertise ( 47 ) . This dynamic coordination relies on process transparency—the ability to see how work unfolds, not just what is produced—and mutual awareness of presence and focus within the shared workspace ( 17 ; 16 ) . These elements enable richer collaboration patterns beyond sequential turn-taking, including concurrent work, interleaved workflows, and real-time adaptation to each other’s contributions ( 47 ) . AI agents increasingly perform tasks step-by-step using human-interpretable operations (e.g., checking files, clicking buttons), making their processes observable across domains like programming ( 43 ) , web automation ( 25 ) , and research ( 15 ) . In creative domains, however, agents typically provide only final outputs rather than exposing execution processes ( 35 ; 45 ; 32 ) , limiting them to assistant roles for specific subtasks ( 50 ; 51 ) . Yet process observability alone is insufficient. The conceptual challenge of flexibly shifting between delegation and direct control based on context has deep roots in mixed-initiative interaction ( 21 ) and adjustable autonomy in robotics ( 37 ) . What remains unexplored is whether agents in creative contexts can go beyond read-only process transparency to achieve collaborative context awareness—interpreting concurrent user direct manipulation on shared artifacts and adapting execution in real-time, enabling the fluid, interleaved collaboration patterns seen in human–human creative work. We investigate this question through two design probes in Figma. Our first probe established process transparency through voice interaction, visual tool operations, and workspace awareness (agent cursor presence). Study 1 (N=10 professional designers) revealed that while step-by-step execution visibility enabled anticipation and timely intervention, designers expected mid-execution concurrent collaboration—wanting to interleave their work with the agent’s ongoing execution. The agent, however, could not distinguish user feedback from independent work, creating conflicts that revealed a critical gap: the absence of collaborative context awareness. This finding motivated Cleo (Collaborative Linked Executive Operator), which tracks concurrent user actions, interprets collaborative intent (feedback vs. independent work), and adapts in real-time. Study 2 (N=10, two-day exploratory study with stimulated recall interviews ( 8 ) ) analyzed 214 turn-level interactions to understand when and why designers choose different collaboration modes. Through qualitative coding, we identified five action patterns (hands-off, observational, concurrent, directive, terminating in Figure 7 ), six triggers prompting intervention (e.g., idea spark, task mismatch in Figure 8 ), and four enabling factors determining choices (e.g., mental model, task priority in Figure 9 ). Concurrent work appeared in 31.8% of turns, full delegation in 70.1%, and directive intervention in 28.5%, with dynamic shifts based on evolving priorities. We synthesized these findings into a decision model (Figure 10 ) that characterizes how designers dynamically navigate among five actions: delegation, observation, direction, concurrent work, and termination. The model identifies six interaction loops driven by triggers, enabling factors, and evolving priorities. Drawing from the dynamics revealed by these loops, we derive design implications for adaptive collaborative agents that support seamless mode transitions and context-aware adaptation. We also release our dataset of 214 interaction turns with logged actions, triggers, and designer rationales to support future research in human-agent co-creative collaboration. Our contributions include: • A characterization of concurrent human-agent interaction with five user action patterns enabling fluid collaboration during agent execution. • A decision model with six interaction loops explaining how users navigate between delegation, direction, and concurrent work based on six triggers and four enabling factors. • Design implications for adaptive collaborative agents supporting seamless mode transitions and context-aware adaptation. • A dataset of 214 human-agent co-creative interaction turns with logged actions. 2. Related Work 2.1. Dynamics of Flexible Collaboration: From Turn-taking to Synchrony Human collaboration is a dynamic spectrum shifting between different interaction modes ( 47 ) . While turn-taking facilitates coordination in complex tasks by minimizing cognitive load and conflicts through sequential actions ( 30 ; 42 ) , creative domains often demand synchronous collaboration ( 36 ) . This is because creative work is rarely a linear process of executing predefined steps; rather, it involves the co-evolution of problem and solution, where design ideas are refined through immediate visual feedback and spontaneous mutual inspiration ( 34 ; 24 ) . In such contexts, waiting for a turn can break the creative flow and delay the rapid synthesis of ideas. Therefore, partners often bump in for immediate adjustments or perform simultaneous edits to interleave workflows, ensuring that the evolving design remains a shared mental model ( 11 ; 47 ) . Modern tools like Figma and Google Slides support this fluidity by fostering workspace awareness—the real-time understanding of a collaborator’s presence and intentions. This is enabled through visual cues like telepointers (moving cursors) and object highlighting, allowing users to anticipate moves and intervene without explicit communication ( 17 ; 16 ) . However, most co-creative AI agents remain confined to a rigid wait-your-turn paradigm. Unlike the parallel dynamics mastered in human-human work, human-AI interaction is still largely limited to a prompt-and-output cycle, lacking the mutual awareness essential for true creative partnership. 2.2. Paradigm Shifts in Human-Agent Collaboration AI is evolving from a command-based tool into a proactive “co-worker” that actively participates within shared workspaces ( 21 ) . This shift has been driven by increased AI capabilities and frameworks that enable models to leverage diverse tools (e.g., code, web search, etc.) ( 44 ) and interleave reasoning with actions across extended time horizons ( 49 ) . These capabilities have enabled diverse agents for programming ( 4 ; 26 ; 43 ) , web automation ( 1 ; 9 ; 25