Interaction-Augmented Instruction: Modeling the Synergy of Prompts and Interactions in Human-GenAI Collaboration Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY Woodstock ’18: ACM Symposium on Neural Gaze Detection, June

Leixian Shen, Yifang Wang, Huamin Qu, Xing Xie, Haotian Li · 2025

Read Interaction-Augmented Instruction: Modeling the Synergy of Prompts and Interactions in Human-GenAI Collaboration Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY Woodstock ’18: ACM Symposium on Neural Gaze Detection, June by Leixian Shen, Yifang Wang, Huamin Qu, Xing Xie, Haotian Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26069.txt.

Introduction

Leixian Shen, Yifang Wang, Huamin Qu, Xing Xie, and Haotian Li build a formal model for instructions that mix text prompts with direct screen actions. People working with generative AI usually type a prompt, but fine or pointing intent is hard to put in words. Adding actions such as brushing or clicking can sharpen what the user means, yet designers lacked a shared model of how the two channels combine.

Their Interaction Augmented Instruction model is a compact graph of entities and relations that captures how prompts and actions reinforce each other. From prior tools they distill twelve reusable interaction patterns that can be combined, and four usage scenarios show how the model helps apply, refine, and invent new patterns for future systems.

Catalog record

CREATORLeixian Shen, Yifang Wang, Huamin Qu, Xing Xie, Haotian Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-26069

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Interaction-Augmented Instruction: Modeling the Synergy of Prompts and Interactions in Human-GenAI Collaboration Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY Woodstock ’18: ACM Symposium on Neural Gaze Detection, June 03–05, 2018, Woodstock, NY Price: 15.00 ISBN: 978-1-4503-XXXX-X/18/06 5837 CCS: Human-centered computing Interaction paradigms Leixian Shen, Yifang Wang, Huamin Qu, Xing Xie, Haotian Li Abstract . Text prompt is the most common way for human-generative AI (GenAI) communication. Though convenient, it is challenging to convey fine-grained and referential intent. One promising solution is to combine text prompts with precise GUI interactions, like brushing and clicking. However, there lacks a formal model to capture synergistic designs between prompts and interactions, hindering their comparison and innovation. To fill this gap, via an iterative and deductive process, we develop the Interaction-Augmented Instruction (IAI) model, a compact entity–relation graph formalizing how the combination of interactions and text prompts enhances human-GenAI communication. With the model, we distill twelve recurring and composable atomic interaction paradigms from prior tools, verifying our model’s capability to facilitate systematic design characterization and comparison. Four usage scenarios further demonstrate the model’s utility in applying, refining, and extending these paradigms. These results illustrate the IAI model’s descriptive, discriminative, and generative power for shaping future GenAI systems. Figure 1. This paper addresses the research question of (A) how to synergize text prompts and GUI interactions to facilitate human–GenAI communication. We propose the Interaction-Augmented Instruction (IAI) model (B) . The model enables systematic characterization and comparison of existing paradigms (C) and guides the design of new interfaces (D) , demonstrating its descriptive, discriminative, and generative power for shaping future GenAI systems. 1. Introduction Generative AI (GenAI) has rapidly become a general-purpose technology, enabling new intelligent applications across domains such as design ( Shen et al. 2025a ) , education ( Law 2024 ) , and data analysis ( Inala et al. 2024 ) . Modern large language and multimodal models can interpret open-ended text prompts and generate diverse outputs ( e.g., text, image, and data analysis dashboards), making the text prompt interface the dominant paradigm for human–AI communication ( Shen et al. 2023 ) . By allowing users to express complex intents in natural language, text prompts offer great flexibility, allowing users “ask anything” and receive tailored responses. Despite the convenience, free-form text prompts are often insufficient to convey fine-grained and nuanced human intent when precise instructions are required. This is due to the ambiguity and the coarse nature of natural language ( Shen et al. 2025b ; Riche et al. 2025 ; Subramonyam et al. 2024 ) . For instance, in image editing, a user might want to modify “ the upper-right flower petal ”, but a text prompt alone is hard to accurately localize that target visual element. Likewise, in coding, asking “ write a loop to process this data ” could imply different iteration strategies. Correspondingly, in practice, users often struggle to articulate sufficient detail or to break down complex goals into a single prompt, resulting in suboptimal or hard-to-control AI behavior. To address these limitations, a promising strategy is to combine general-purpose but imprecise text prompts with dedicated interactions ( e.g., clicks, drags, brushes, etc. ) through graphical user interfaces (GUIs), as shown in Fig. 1 -A. The synergy enables both flexible and fine-grained control by users. For example, OpenAI Canvas ( OpenAI 2024 ) and Cursor ( Anysphere 2023 ) let users select a specific text segment or code block and then issue a focused prompt that applies only to that context. Tableau AI ( Tableau 2024 ) uses LLMs to suggest context-aware follow-up questions on GUI for user selection during data exploration. Research prototypes following this strategy have also emerged, exploiting the strengths of both natural language and direct-manipulation input for diverse tasks, such as coding ( Feng et al. 2024 ) , writing ( Laban et al. 2024 ) , data visualization ( Xie et al. 2025 ) , and image editing ( Liu et al. 2025 ) . Despite the wide and diverse applications, the strategy has not been formalized as a comprehensive interaction model to guide related research and practices. As a pioneering study in the HCI field points out ( Beaudouin-Lafon 2004 ) , these “point-like” research demonstrations of the strategy are often insufficient due to the challenges of transforming task-specific demonstrations to broader real-world applications and understanding the advancement of related techniques. It is essential to propose general interaction models to facilitate the careful examination of existing designs ( descriptive power ), the comparison between design variations ( discriminative power ), and the consequent innovation of new interaction designs for emerging tasks ( generative power ) ( Beaudouin-Lafon 2004 ; Beaudouin-Lafon 2000 ; Liao and Xiao 2023 ) . Towards this goal, there has been a series of previous research that attempted to reveal and advance the interaction-text prompt synergy. Gao et al. ( Gao et al. 2024 ) and Hu et al. ( Hu et al. 2025 ) have developed taxonomies of interaction paradigms for interactive GenAI applications, such as GenAI as a medium, tool, partner, and mediator. However, the taxonomy is mostly descriptive and is difficult to use to guide the generation of new designs for new scenarios. Riche et al. ( Riche et al. 2025 ) proposed AI-Instruments, extending instrumental interaction ( Beaudouin-Lafon 2000 ) by treating prompts themselves as manipulable interface objects, guided by three principles: Reification, Reflection, and Grounding. These principles enrich the considerations for generating new interaction designs but fall short in comparing multiple designs within a model that explicitly represents the interplay between text prompts and other interaction modalities. To fill this gap, through a deductive and iterative process, we propose the Interaction-Augmented Instruction (IAI) model: a compact entity–relation graph that makes explicit how precise GUI interactions and free-form prompts are composed into the executable instructions consumed by GenAI ( Fig. 1 -B). The model comprises six entities: Human ( H ), Interaction ( I ), Text Prompt ( T ), Augmented Instruction ( Aug ), GenAI ( G ), and Artifact ( A ). By consolidating candidate roles into this minimal set, the model highlights Aug as the single explicit input to GenAI, thereby simplifying comparison across paradigms and clarifying how different interfaces combine T , I , and A to construct Aug . We further systematically enumerated all pairwise relations and preserved only those that are semantically meaningful, discriminative, and interpretable ( Table 1 ). Building on this foundation, to examine if our IAI model can be generally applied to describe and differentiate existing GenAI systems’ interfaces, we revisit a curated corpus of GenAI-powered interactive systems and map each system’s interaction workflow to one or more directed paradigm graphs ( Fig. 1 -C). From the corpus, we distill twelve frequent and composable atomic paradigms organized by interaction timing (pre- or post- GenAI invocation) and resource availability (prompt-only vs. artifact-grounded). These paradigm graphs based on the IAI model serve as high-level abstractions of interaction workflows, enabling tools to be clustered and compared. The IAI model provides not only a theoretical lens but also a practical framework for future interface design. The twelve design paradigms derived

Read the complete text