Directing the Robot: Scaffolding Creative Human–AI–Robot Interaction Conference: The 3rd InterAI Workshop: "Interactive AI for Human-Centered Robotics at ACM CHI 2026; April 13, 2026; Barcelona, Spain The 3rd InterAI Workshop: "Interactive AI for Human-Centered Robotics at ACM CHI 2026, April 13–17, 2026, Barcelona, Spain ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Human-centered computing Human computer interaction (HCI) CCS: Human-centered computing Interaction design CCS: Computing methodologies Robotics CCS: Computing methodologies Artificial intelligence

Jordan Aiko Deja, Isidro Butaslac III, Nicko Reginio Caluya, Maheshya Weerasinghe · 2026

Read Directing the Robot: Scaffolding Creative Human–AI–Robot Interaction Conference: The 3rd InterAI Workshop: "Interactive AI for Human-Centered Robotics at ACM CHI 2026; April 13, 2026; Barcelona, Spain The 3rd InterAI Workshop: "Interactive AI for Human-Centered Robotics at ACM CHI 2026, April 13–17, 2026, Barcelona, Spain ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Human-centered computing Human computer interaction (HCI) CCS: Human-centered computing Interaction design CCS: Computing methodologies Robotics CCS: Computing methodologies Artificial intelligence by Jordan Aiko Deja, Isidro Butaslac III, Nicko Reginio Caluya, Maheshya Weerasinghe (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-07748.txt.

Introduction

Deja and colleagues reframe human-AI-robot dealings as support letting humans shape robot moves in control. Robots enter art, school and public settings, yet dealings frame humans as bosses chasing scores and speed. Through art practice, teaching and body-based cases, humans act as leads defining intent and steering edits, with AI between words and robotic doing. They outline design lessons putting imagination, agency and flow first. Open challenges in social, large-scale and mission-crucial settings invite rethinking robots and AI as support for human imagination.

Catalog record

CREATORJordan Aiko Deja, Isidro Butaslac III, Nicko Reginio Caluya, Maheshya Weerasinghe
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15482
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.07748

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Directing the Robot: Scaffolding Creative Human–AI–Robot Interaction Conference: The 3rd InterAI Workshop: "Interactive AI for Human-Centered Robotics at ACM CHI 2026; April 13, 2026; Barcelona, Spain The 3rd InterAI Workshop: "Interactive AI for Human-Centered Robotics at ACM CHI 2026, April 13–17, 2026, Barcelona, Spain ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Human-centered computing Human computer interaction (HCI) CCS: Human-centered computing Interaction design CCS: Computing methodologies Robotics CCS: Computing methodologies Artificial intelligence Jordan Aiko Deja, Isidro Butaslac III, Nicko Reginio Caluya, Maheshya Weerasinghe Abstract . Robots are moving beyond industrial settings into creative, educational, and public environments where interaction is open-ended and improvisational. Yet much of human–AI–robot interaction remains framed around performance and efficiency, positioning humans as supervisors rather than collaborators. We propose a re-framing of AI interaction with robots as scaffolding: infrastructure that enables humans to shape robotic behaviour over time while remaining meaningfully in control. Through scenarios from creative practice, learning-by-teaching, and embodied interaction, we illustrate how humans can act as executive directors, defining intent and steering revisions, while AI mediates between human expression and robotic execution. We outline design and evaluation implications that foreground creativity, agency, and flow. Finally, we discuss open challenges in social, scalable, and mission-critical contexts. We invite the community to rethink interacting with Robots and AI not as autonomy, but as sustained support for human creativity. Figure 1 . Illustrative scenarios of scaffolding in creative human–AI–robot interaction. From left to right: (1) Scaffolding Inspiration (robot gestures support ideation); (2) Scaffolding Adlibs (robot improvises in musical performance); (3) Synchronizing Performance (ground and aerial robots in stage collaboration); and (4) Winging Support (robot assistance under time pressure). Across scenarios, AI scaffolds creativity and action while preserving human agency. Illustration generated using generative AI and refined by the authors . 1. Introduction and Background Robots are increasingly becoming part of daily life, expanding beyond industrial settings ( 10 ) into public spaces, educational environments ( 18 ) , and creative contexts ( 19 ) . As robots move closer to people in both physical and social settings, interaction with them is no longer exceptional but inevitable. Advances in Artificial Intelligence (AI) have been central to this shift, often by making human–robot interaction more accessible through the automation of behaviours, optimisation of performance, and reduction of human effort in repetitive or routine tasks. While effective, these approaches have also shaped dominant design paradigms that prioritise efficiency and autonomy over richness of intent. However, framing AI primarily as a tool for automation and optimisation limits how humans and robots can collaborate in open-ended settings. In such framings, humans are often positioned as supervisors or end-users, while decision-making authority and interpretive control are increasingly delegated to the system. In this context, interpretative control is defined as how the human decides the execution similar to how the director of photography decides on the color toning and angles when shooting a film. This diminishes the value of human autonomy ( 7 ) , and limits active engagement, exploration, and creative collaboration, particularly in expressive, educational, or improvisational contexts. Prior work has explored improvisation, mixed-initiative interaction, and co-creative human–AI systems ( 12 ; 13 ; 5 ; 1 ) . However, scaffolding in these systems is often treated as either a technical mechanism for learning or a temporary design strategy used to compensate for system limitations. In contrast, we argue for treating scaffolding as a “core interaction principle” in human–AI–robot collaboration; one that explicitly structures how agency, interpretation, and control are distributed over time. In this case, we look at not leaving it all to the AI or robot, but rather having the original idea come from the human and both human and AI collectively build on their ideas. Building on this perspective, we propose a re-framing in which interactive AI scaffolds collaboration by positioning humans as “executive directors” of robotic behaviour. In this role, humans define intent, make interpretive judgments, and steer iterative revisions, while interactive AI mediates between human expression (e.g., language, gesture, demonstration) and robotic execution (e.g., motion, coordination, embodied action). Rather than handing control away from the human, AI supports thinking, decision-making, and creative flow, while robots act as expressive and physical performers of human-defined intent. This perspective foregrounds human agency and re-frames interaction as an ongoing creative process rather than a supervisory task. Specifically, this paper contributes (1) a re-framing of interactive AI as scaffolding for creative human-robot interaction, (2) illustrative scenarios spanning creative and mission-critical contexts (as shown in Figure 1 ), and (3) design and evaluation implications centered on agency and flow. 2. Rethinking Design and Evaluation Through Use Cases To ground the idea of scaffolding as an interaction principle, we consider use cases in which creativity, exploration, and improvisation are central. In many such contexts, robots are not continuously autonomous actors, but remain available as expressive resources that can be activated, redirected, or reinterpreted by humans as needed. Interactive AI mediates this availability, allowing humans to focus on intent, timing, and meaning rather than low-level control. In this sense, interaction unfolds as a process of direction rather than execution. Across these scenarios, humans act as executive directors: they articulate high-level goals, shape interpretation, and intervene iteratively as outcomes emerge. Interactive AI supports this role by translating underspecified human input into executable robotic behaviour, managing ambiguity and coordination while leaving authorship and decision-making with the human. Thought 1: Preserve interpretive control. Interactive AI must support human intent without displacing authorship or the ability to redirect action. One illustrative case involves teaching or shaping robot motion through few-shot demonstrations ( 1 ) . A human may imagine a sequence of movements or expressive qualities they wish a robot to perform, but may be constrained by physical ability, scale, or precision. Rather than requiring exact replication, interactive AI can scaffold this process by interpreting partial demonstrations, smoothing transitions, and generating feasible motion variants that preserve the intended structure and emphasis of the human ( Figure 1 Frame 1). Here, the human remains responsible for defining what the movement should convey, while AI manages how that intent is realized in robotic execution. Also as in emerging learning-oriented contexts, where humans do not merely demonstrate tasks, but iteratively shape how robots collaborate with them over time. For instance, in collaborative assembly settings, researchers have developed a “Teaching-Learning-Collaboration (TLC)” model in which a human demonstrates a task and communicates high-level intent and preferences to a robot collaborator ( 25 ) . The robot uses this information to build task knowledge via learning from demonstrations and then actively assists the human in completing shared work. Such models illustrate how humans can act as executive directors: guiding, teaching, and co-revising robotic behaviour while interactive learning enables robots to translate human intent into embodied collaborative action without displacing control. These examples suggest several design implications for interactive AI as scaffolding. First, systems should prioritise “responsiveness over correctness”, enabling interaction to continue even when human input is partial, ambiguous, or evolving. Second, interfaces that leverage familiar forms of expression such as language, gesture, or embodied demonstration, can reduce cognitive overhead by allowing humans to direct robots in ways that align with how they already communicate intent. In such cases, AI acts less as an autonomous decision-maker and more as

Read the complete text