Interaction Analysis by Humans and AI: A Comparative Perspective Conference: Interaction Design and Children; June 23–26, 2025; Reykjavik, Iceland Interaction Design and Children (IDC ’25), June 23–26, 2025, Reykjavik, Iceland DOI: 10.1145/3713043.3731527 ISBN: 979-8-4007-1473-3/2025/06 CCS: Computi

Maryam Teimouri, Filip Ginter, Tomi "bgt" Suovuo · 2025

Read Interaction Analysis by Humans and AI: A Comparative Perspective Conference: Interaction Design and Children; June 23–26, 2025; Reykjavik, Iceland Interaction Design and Children (IDC ’25), June 23–26, 2025, Reykjavik, Iceland DOI: 10.1145/3713043.3731527 ISBN: 979-8-4007-1473-3/2025/06 CCS: Computi by Maryam Teimouri, Filip Ginter, Tomi "bgt" Suovuo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07707.txt.

Introduction

Teimouri, Ginter, and Suovuo present a 2025 conference paper on how children communicate through screens and headsets. Finnish-speaking children played a gesture guessing game in pairs, once over Zoom and once in mixed reality with HoloLens. The team recorded the sessions and used large language models to transcribe, translate, and mark up the talk.

The automated pass cut processing time a lot and let researchers who lack Finnish join the work. Its marks were not always accurate, though. Early results suggest mixed reality drew richer interaction and more feeling, while Zoom stayed simpler to use. The paper weighs AI help in interaction studies, both its speed and its limits.

Catalog record

CREATORMaryam Teimouri, Filip Ginter, Tomi "bgt" Suovuo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07707

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Interaction Analysis by Humans and AI: A Comparative Perspective Conference: Interaction Design and Children; June 23–26, 2025; Reykjavik, Iceland Interaction Design and Children (IDC ’25), June 23–26, 2025, Reykjavik, Iceland DOI: 10.1145/3713043.3731527 ISBN: 979-8-4007-1473-3/2025/06 CCS: Computing methodologies Machine translation CCS: Computing methodologies Speech recognition CCS: Applied computing Education CCS: Human-centered computing Interaction design Maryam Teimouri, Filip Ginter, Tomi "bgt" Suovuo Abstract . This paper explores how Mixed Reality (MR) and 2D video conferencing influence children’s communication during a gesture-based guessing game. Finnish-speaking participants engaged in a short collaborative task using two different setups: Microsoft HoloLens MR and Zoom. Audio-video recordings were transcribed and analyzed using Large Language Models (LLMs), enabling iterative correction, translation, and annotation. Despite limitations in annotations’ accuracy and agreement, automated approaches significantly reduced processing time and allowed non-Finnish-speaking researchers to participate in data analysis. Evaluations highlight both the efficiency and constraints of LLM-based analyses for capturing children’s interactions across these platforms. Initial findings indicate that MR fosters richer interaction, evidenced by higher emotional expression during annotation, and heightened engagement, while Zoom offers simplicity and accessibility. This study underscores the potential of MR to enhance collaborative learning experiences for children in distributed settings. 1. Introduction Advancements in digital communication—particularly through Mixed Reality (MR) ( Milgram and Kishino 1994 ) and video conferencing tools like Zoom ( Zoom Video Communications 2023 ) —have reshaped how younger generations interact in educational contexts. This study is part of a larger four-year project funded by the Research Council of Finland, which focuses on developing a HoloLens-based holoportation tool for distributed education. We examine children’s verbal communication during gameplay on an MR platform, with the same experiment replicated on Zoom for comparison. Our analysis draws from both the gameplay sessions and post-experiment interviews. However, challenges such as the time-intensive nature of manual transcription ( Hahta et al. 2023 ) and the language barrier posed by Finnish data require extensive data processing. To address these issues, we employ large language models (LLMs) ( Vaswani et al. 2017 ) to support transcription ( Graves et al. 2013 ) , correction ( Zhang et al. 2019 ) , translation ( Johnson et al. 2017 ) , speaker identification ( Ma et al. 2017 ) , and emotion detection ( Huang and Dong 2019 ) , assessing their effectiveness and practical utility throughout the workflow. 2. Related Work HoloLens creates a Mixed Reality (MR) environment by projecting holograms through depth images captured by Intel RealSense cameras, whose RGBD systems are detailed in ( Keselman et al. 2017 ) , covering optics, noise handling, and performance in gesture recognition and 3D reconstruction. A review of 44 studies from 2016 to 2020 highlights HoloLens’s role in engineering, architecture, and medical education, particularly in enhancing visualization and workflows ( Park et al. 2021 ) . Immersive experiences like the Van Gogh exhibition have also been evaluated for their cognitive and emotional impact using sentiment and emotion recognition tools ( África et al. 2023 ) , supporting the value of emotion analysis in education, where promoting positive emotional states can enhance engagement and improve learning outcomes ( Munezero et al. 2013 ) . Achieving this requires accurate transcription and emotion detection, which recent advances in Large Language Models (LLMs) have significantly improved. Self-supervised models such as wav2vec 2.0 enable better transcription in low-resource languages like Finnish ( Baevski et al. 2020 ) , while speaker identification benefits from approaches like Contextual Beam Search, which combines LLMs and acoustic diarization to enhance accuracy through audio and lexical cues ( Weninger et al. 2023 ) . Recent work also highlights the value of involving children directly in the design of MR experiences. A participatory study with HoloLens 2 and 3D holograms showed how students co-designed social interactions, supporting early integration of educational goals in XR development ( Goagoses et al. 2024 ) . This aligns with the current study’s focus on child-centered, emotionally rich MR environments. 3. Study Setup The study compares two setups: the experimental group (A–B) , where one participant uses a HoloLens to view their partner as a hologram (room A) and the other uses a 3D camera and TV screen (room B); and the control group (B–B) , where both participants used room B, interacting via TV screens and shared digital whiteboards. Rooms are connected with custom software, with HoloLens and 2D cameras in room A, and 3D cameras and TV screens in room B. Tablets on stands allowed drawing on digital whiteboards. Participants received training to become familiar with the devices and interaction style, emphasizing natural communication over technology testing. Familiar images of animals or characters were prepared in advance. In pairs, participants take turns describing and miming gestures of assigned images while their partner guesses and draws on the whiteboard. After five minutes, roles are switched with a new image set. The experimental group experiences both holographic and real-time TV screen interactions, enhancing spatial awareness through MR, while the control group engages only via 2D TV screen setups for comparison. 4. Data Collection, Data Processing and Tools Video and audio feeds are recorded using cameras, HoloLens headsets, and microphones. Lidar cameras are used in 2D mode to prevent interference with HoloLens. For the control group, interactions are recorded via Zoom. Surveys and interviews conducted post-study focus on participants’ feelings during the experiment, their familiarity with similar experiences, and their perception of time passing. All recordings are stored securely, and video files were edited and converted to audio for natural language processing tasks. The original study materials are securely stored on the University’s Seafile service (a cloud-based file synchronization and sharing platform), with access restricted to the researchers involved in the study. These materials will be deleted after the study ends, or within a maximum of 5 years. The data will not be shared externally and will remain within Europe. Initial Data: The dataset includes 10-minute videos from 5 pairs, with 3 pairs in the control group. Videos for the experimental group are recorded separately, while a single video covers both participants in the control groups. Additionally, four 30-minute group interviews are conducted post-experiment, resulting in 17 files and 250 minutes of Finnish-language data. Speech-to-Text Processing: To automate video and audio conversion to text, we evaluate transcription tools using manually transcribed data from a previous study. Accuracy is measured with Character Error Rate (CER) ( Morris et al. 2004 ) , calculated by comparing the number of insertions, deletions, and substitutions with the ground truth. A lower CER indicates better accuracy. All files were converted to ‘.wav‘ format. Three tools were tested: • Google Cloud ( Google Cloud 2023 ) : Best performance with CER 0.24 overall, 0.31 for group discussions, and 0.17 for two-person conversations. • Amazon Transcription ( Amazon Web Services 2023 ) : CER of 0.29. • Aalto ASR 1 1 1 https://www.kielipankki.fi/tuki/aalto-asr-automaattinen-puheentunnistin/ : Found unsuitable due to poor recognition. Transcriptions are processed on European servers: Aalto ASR in Finland, Amazon in Ireland, and Google in Sweden. See Appendix A.1 for detailed results. Text Processing and Translation: Transcription outputs contain errors such as misrecognitions and missing punctuation. To improve quality and translate Finnish to English, we use a multi-step process (visualized in Appendix B.2 ):

Read the complete text