Empowering Children to Create AI-Enabled Augmented Reality Experiences DOI: 10.1145/3746059.3747662 Conference: The 38th Annual ACM Symposium on User Interface Software and Technology; September 28-October 1, 2025; Busan, Republic of Korea The 38th Annual ACM Symposium on User Interface Software and
Lei Zhang, Shuyao Zhou, Amna Liaqat, Tinney Mak, Brian Berengard, Emily Qian, Andrés Monroy-Hernández · 2025
Read Empowering Children to Create AI-Enabled Augmented Reality Experiences DOI: 10.1145/3746059.3747662 Conference: The 38th Annual ACM Symposium on User Interface Software and Technology; September 28-October 1, 2025; Busan, Republic of Korea The 38th Annual ACM Symposium on User Interface Software and by Lei Zhang, Shuyao Zhou, Amna Liaqat, Tinney Mak, Brian Berengard, Emily Qian, Andrés Monroy-Hernández (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08467.txt.
Introduction
Lei Zhang and co-authors built a programming playground that lets children create their own augmented reality experiences with AI help, in a 2025 paper. Most children's AR and AI tools cast kids as consumers. Their system, Capybara, flips the role: children invent 3D characters, dress them with accessories, and overlay them on the real world through the camera.
Text-to-3D generative models turn a child's description into a character. Auto-rigging and body tracking make it move, and vision models recognize real objects so children can program interactions between their characters and the physical room. User studies with twenty children in the United States and Argentina showed the system could support rich, personalized creations. The authors argue the findings show children can harness generative AI as authors, not just audiences, and they present Capybara as evidence that creation-first design works for this age group.
Catalog record
| CREATOR | Lei Zhang, Shuyao Zhou, Amna Liaqat, Tinney Mak, Brian Berengard, Emily Qian, Andrés Monroy-Hernández |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-08467
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Empowering Children to Create AI-Enabled Augmented Reality Experiences DOI: 10.1145/3746059.3747662 Conference: The 38th Annual ACM Symposium on User Interface Software and Technology; September 28-October 1, 2025; Busan, Republic of Korea The 38th Annual ACM Symposium on User Interface Software and Technology (UIST ’25), September 28-October 1, 2025, Busan, Republic of Korea ISBN: 979-8-4007-2037-6/2025/09 CCS: Human-centered computing Interactive systems and tools CCS: Human-centered computing Mixed / augmented reality CCS: Social and professional topics Computing education Lei Zhang, Shuyao Zhou, Amna Liaqat, Tinney Mak, Brian Berengard, Emily Qian, Andrés Monroy-Hernández Abstract . Despite their potential to enhance children’s learning experiences, AI-enabled AR technologies are predominantly used in ways that position children as consumers rather than creators. We introduce Capybara, an AR-based and AI-powered visual programming environment that empowers children to create, customize, and program 3D characters overlaid onto the physical world. Capybara enables children to create virtual characters and accessories using text-to-3D generative AI models, and to animate these characters through auto-rigging and body tracking. In addition, our system employs vision-based AI models to recognize physical objects, allowing children to program interactive behaviors between virtual characters and their physical surroundings. We demonstrate the expressiveness of Capybara through a set of novel AR experiences. We conducted user studies with 20 children in the United States and Argentina. Our findings suggest that Capybara can empower children to harness AI in authoring personalized and engaging AR experiences that seamlessly bridge the virtual and physical worlds. Figure 1. System Overview of Capybara. (a) Capyabara is a block-based programming environment in mobile Augmented Reality (AR) that empowers children to program AI-enabled AR experiences. (b & c) Users can customize virtual 3D characters and accessories via the speech-to-3D GenAI feature. (d) The system auto-rigs the character and lets users control the character’s animation via body motion. (e) Via drag-and-drop programming, users can define interactive behaviors between virtual characters and their physical surroundings using object detection. In this example, a cute panda character wearing a wizard hat waves their arms to keep balance when stepping on a physical banana. (f) Combining the above functionalities, Capybara enables novel AR experiences that bridge the virtual and physical worlds. † † cc-license: by 1. Introduction The advancement of Artificial Intelligence (AI) has enhanced Augmented Reality (AR) experiences, making them more intelligent, engaging, and personalized ( Suzuki et al. 2024 ) . Furthermore, researchers have integrated AI capabilities into AR to enrich children’s educational and social experiences. For example, AI-powered AR systems can transform static content from physical textbooks into interactive 3D diagrams, supporting more immersive and effective learning ( Gunturu et al. 2024 ) . Many interactive social applications, such as AR filters ( Dagan et al. 2022 ) , leverage facial and body recognition to create more playful, engaging, and memorable social experiences for children. Such experiences are made possible by AI’s increasing capabilities to understand the physical environment (e.g., through computer vision) or to synthesize novel content (e.g., through generative AI). Despite the recent technological advances, digital tools designed for children predominantly emphasize content consumption over creation, i.e., they focus on “reading” rather than “writing.” A notable exception is authoring tools for children such as Scratch ( Resnick et al. 2009 ) , which are developed based on Constructionism, a pedagogy that emphasizes learning by building ( Resnick et al. 1996 ) . Constructionist tools aim to empower children to become creators of digital experiences for self-expression and creative exploration. However, to our knowledge, there are no Constructionist tools for children to create increasingly complex and expressive AR experiences enhanced by AI capabilities nowadays. This is particularly challenging because creating such experiences typically require expertise from 3D modeling and animation, to programming and deploying AI models. For instance, creating 3D characters and their animations for storytelling often demands advanced skills from professional 3D modelers and animators. Programming interactions between the physical and virtual worlds requires not only 3D programming knowledge, but also an understanding of how to incorporate AI models effectively. In this research, we aim to empower children to be creators of AI-enabled AR experiences by equipping them with a novel authoring tool. This paper introduces Capybara , an authoring tool for children to create AI-enabled AR experiences. Our system offers a block-based programming environment in AR that allows children to construct visual programs by dragging and dropping code blocks. Capybara raises the ceiling of expressive power by introducing three novel functionalities: 1) 3D character customization, 2) animation customization through puppeteering, and 3) programming interactions across virtual and physical worlds. Specifically, our system enables children to generate 3D characters using natural language descriptions, powered by a GenAI model that supports text-to-3D synthesis. Children can also generate accessories for their character using the same technique, as well as programmatically create and destroy these accessories. In addition, our system can automatically rig these generated 3D characters, enabling children to control their animation via body motion and programmatically replay those custom animations. These features offer a seamless connection between the virtual character and the user’s physical body. Finally, our system enables children to utilize off-the-shelf AI functionalities, such as object recognition, to program interactions between the virtual character and the physical environment. Combining these techniques into the block-based programming environment, Capybara empowers children to create expressive, AI-enabled experiences. To demonstrate the system’s expressiveness, we present a range of example experiences created using Capybara . To evaluate the usability of our system, we conducted user studies with 20 children aged 7 to 16 across the United States and Argentina. Our findings suggest that Capybara enabled our participants to create AI-enabled AR experiences. We found that Capybara enabled creative usage such as generating 3D characters for both realistic and fantastical experiences, animating the character for storytelling, and building interactions between the virtual and physical worlds. Participants found the app to be unique and fun. They felt that it could facilitate creative expression through AI-enabled customization, enable programming interactions with the physical world, and engage them with concepts in Computational Thinking (CT) and AI literacy. However, integrating AI into the authoring process could introduce alignment issues (e.g., mismatches between AI results and children’s intent) and distract children from their creation goals. These findings offer insights into designing AI-enabled AR authoring tools for children to foster creativity and self-expression. In summary, our paper makes the following contribution: 1) Capybara , an authoring tool for children to create AI-enabled AR experiences, which seamlessly combines AI techniques with children’s content creation process; 2) a set of example experiences demonstrating the expressiveness of the tool; and 3) empirical insights gained from our user studies with children, highlighting its benefits and challenges of using Capybara . 2. Related Work We draw from prior literature in AR/VR authoring tools, visual programming languages, and Children-AI Interaction. 2.1. Authoring Tools for Immersive Experiences Authoring AR/VR experiences has been known to have a high barrier of entry for end-users such as people with few technical skills ( Ashtari et al. 2020 ) . In recent years, many novel AR/VR prototyping tools have been proposed to lower the “floor” of creating immersive experiences ( Lee et al. 2004 ; Nebeling et al. 2018 ; Nebeling and Madier 2019 ; Zhang and Oney 2020 ; Suzuki