AI-Instruments : Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools

Nathalie Riche, Anna Offenwanger, Frederic Gmeiner, David Brown, Hugo Romat, Michel Pahud, Nicolai Marquardt, Kori Inkpen, Ken Hinckley · 2025

Read AI-Instruments : Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools by Nathalie Riche, Anna Offenwanger, Frederic Gmeiner, David Brown, Hugo Romat, Michel Pahud, Nicolai Marquardt, Kori Inkpen, Ken Hinckley (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18736.txt.

Introduction

Nathalie Riche and co-authors turn AI prompts into graspable instruments instead of chat lines. Chat answers arrive as long, straight text that is hard to steer or revisit. Their AI-Instruments embody each prompt as a reusable on-screen tool. Ambiguous intent shows up as several live readings the user can pick among.

Model answers likewise appear as a range to explore, not one verdict. New instruments can sprout from examples or from other instruments. Language models suggest and vary fresh controls on the fly. Four probes in image creation tested the idea with twelve users. Direct handling eased the work of forming and refining intent.

Catalog record

CREATORNathalie Riche, Anna Offenwanger, Frederic Gmeiner, David Brown, Hugo Romat, Michel Pahud, Nicolai Marquardt, Kori Inkpen, Ken Hinckley
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-18736

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AI-Instruments : Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools Nathalie Riche, Anna Offenwanger, Frederic Gmeiner, David Brown, Hugo Romat, Michel Pahud, Nicolai Marquardt, Kori Inkpen, Ken Hinckley Abstract . Chat-based prompts respond with verbose linear-sequential texts, making it difficult to explore and refine ambiguous intents, back up and reinterpret, or shift directions in creative AI-assisted design work. AI-Instruments instead embody “prompts” as interface objects via three key principles: (1) Reification of user-intent as reusable direct-manipulation instruments; (2) Reflection of multiple interpretations of ambiguous user-intents ( Reflection-in-intent ) as well as the range of AI-model responses ( Reflection-in-response ) to inform design "moves" towards a desired result; and (3) Grounding to instantiate an instrument from an example, result, or extrapolation directly from another instrument. Further, AI-Instruments leverage LLM’s to suggest, vary, and refine new instruments, enabling a system that goes beyond hard-coded functionality by generating its own instrumental controls from content. We demonstrate four technology probes, applied to image generation, and qualitative insights from twelve participants, showing how AI-Instruments address challenges of intent formulation, steering via direct manipulation, and non-linear iterative workflows to reflect and resolve ambiguous intents. Conference: CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26-May 1, 2025, Yokohama, Japan DOI: 10.1145/3706598.3714259 ISBN: 979-8-4007-1394-1/25/04 CCS: Human-centered computing Interaction paradigms Figure 1 . AI-Instruments embody prompts as interface objects, informed by three principles: reification of user intent, reflection, and grounding (left). Visual overview of four technology probes of AI-Instruments – generative containers, transformative lenses, prompt fragments, and fillable brushes (right). Visual Abstract of the paper. Visual illustration of a person using AI-Instruments on the left, then a box including the three principles of AI-Instruments in the middle, and sketched illustrations of AI-Instruments on the right (using GUI elements and wireframes). The figure explains that AI-Instruments embody prompts as interface objects, informed by three principles (left): reification of user intent, reflection, and grounding. (right) Visual overview of four technology probes of AI-Instruments – generative containers, transformative lenses, prompt fragments, and fillable brushes. † † cc-license: by 1. Introduction Despite the immense promise of generative AI, it remains challenging for people to express and refine their true intents via multiple rounds of textual chat-prompts ( Mahdavi Goloujeh et al., 2024 ; Zamfirescu-Pereira et al., 2023 ) , as well as to pursue multiple-alternative paths forward within a linear conversational metaphor. Users face numerous difficulties (e.g., ( Subramonyam et al., 2024 ) ): articulating their intent in a few words of written text (intent formulation); correctly expressing sufficient detail to express, refine, and re-formulate their true intent (intent disambiguation); iterating over the model’s response to approach a desired outcome (steering); and navigating higher-order challenges of interaction with AI, such as discovering what one can do with AI—or even what one "actually" wants to achieve (intent resolution)—within the linear sequential limitations of chat-based exchanges (interaction workflow). Although existing work in human-computer interaction and artificial intelligence (HCI+AI) addresses some aspects of these challenges in piecemeal fashion through novel interaction techniques with generative AI (e.g. ( Chung and Adar, 2023 ; Masson et al., 2024 ) ), we argue here that AI-Instruments offer a novel approach to gain traction on many aspects of these challenges by appropriating and re-casting the principles of instrumental interaction ( Beaudouin-Lafon, 2000 ; Beaudouin-Lafon and Mackay, 2000 ) to the modern context of generative HCI+AI user experiences. Quoting Beaudoin-Lafon et al. ( Beaudouin-Lafon et al., 2021 ) , the value of proposing such interaction model is to "change-oriented perspective by providing HCI researchers with conceptual tools for analyzing technologies in use or exploring novel future solutions" . Triangulating theory, artifact, and empirical evaluation has strong benefits for advancing HCI research ( Mackay and Fayard, 1997 ) . Instrumental interaction offers a particularly compelling concept from the HCI literature to revisit in the context of generative AI because it offers principled interaction dynamics about how software functionalities ( "commands" ) combine with content (the "objects" those commands act upon). While in the past these dynamics had to be hand-designed and hand-coded for specific object types and application settings, the advent of generative AI makes it plausible that the polymorphic nature of high-level commands and flexible content representations will unleash exciting new possibilities for HCI+AI graphical user interfaces. In particular, our approach embodies AI prompts as graphical interface objects and adapts the instrumental interaction model for Generative AI by considering the following three principles: (1) Reification of user intent into instruments: turning user-intent from varied abstractions and granularity levels into one or more reusable graphical interface object(s); (2) Reflection : the consideration of multiple alternatives that reflect ( Schön, 1992 ) both ambiguous intents as expressed by the user ( reflection-in-intent ), and ambiguous interpretation of AI responses ( reflection-in-response ), to steer content generation towards a satisfactory result; and finally (3) Grounding : instantiating an instrument from a specific scope of selected content, from an example result, or even from another instrument. Via a technology probe ( Hutchinson et al., 2003 ) that implements four complementary examples of AI-instruments, we illustrate how they can ameliorate many design challenges plaguing today’s linear-chat-based AI interfaces: intent formulation, prompt engineering, direct manipulation and steering, non-linear iterative workflows, and intent resolution. We also present initial reactions of 12 participants who tried our AI-instruments, yielding qualitative insights on the value and limitations of our AI-instruments interaction model, as compared to conversational prompting. Designed through the lens of the three principles, we built a set of technology probes focused on image generation. The goal of these four exemplar AI-Instruments is to demonstrate the new interaction capabilities and affordances of our model: (1) Fragments decompose gen-AI prompts into reified reconfigurable objects, affording reflection-on-intent on the latent prompt structure, and grounding generation by dragging fragments from one object to another. (2) Transformative Lenses generate new content grounded in one or more content elements, which allows flexible recomposition of scenes and (if desired) continuous updates of the result. (3) Generative Containers create multiple alternatives of images, text, and even instruments or fragments. (4) Fillable Brushes encapsulate a prompt, filled by selecting example content with the brush (or by directly typing the prompt for a new action). Using the instruments in synergy—where outputs from one instrument form input for the next, or even using instruments to create new meta-instruments—affords expressive degrees-of-freedom for fine-grained steering of generative AI. In summary, our high-level contributions include the following: • Extend the classic instrumental interaction model ( Beaudouin-Lafon, 2000 ) to generative AI, emphasizing three driving principles: reification of user intent, reflection, and grounding; • Demonstrate four AI-instruments via technology probes, showing how these driving principles manifest in their design and implementation; • Provide initial reactions from 12 users when shifting from a linear-chat interaction paradigm to direct manipulation through AI-instruments, showing that it can address a number of human-AI interaction challenges. In the following sections we discuss related techniques across the HCI, Human-AI interaction, and design literature. This is followed by an Example Walkthrough of our AI-instruments, a wider discussion of Instrumental Interaction with AI, and further details of our Four

Read the complete text