TARIPlay: A Test Framework for AR Applications based on Interactive Area Tracking in Playback Videos DOI: 10.1145/3744916.3787817 Conference: The International Conference on Software Engineering; April 2026; Rio De Janeiro, Brazil 4 Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE ’26), April 12–18, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3744916.3787817 ISBN: 979-8-4007-2025-3/2026/04 CCS: Software and its engineering Software testing and debugging CCS: Social and professional topics Software maintenance CCS: Human-centered computing Mixed / augmented reality

Sayed Amir Mousavi and Xiaoyin Wang · 2026

Read TARIPlay: A Test Framework for AR Applications based on Interactive Area Tracking in Playback Videos DOI: 10.1145/3744916.3787817 Conference: The International Conference on Software Engineering; April 2026; Rio De Janeiro, Brazil 4 Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE ’26), April 12–18, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3744916.3787817 ISBN: 979-8-4007-2025-3/2026/04 CCS: Software and its engineering Software testing and debugging CCS: Social and professional topics Software maintenance CCS: Human-centered computing Mixed / augmented reality by Sayed Amir Mousavi and Xiaoyin Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16544.txt.

Introduction

Mousavi and Wang propose TARIPlay in their 2026 study, a testing framework for augmented-reality apps. It finds usable interaction zones inside recorded playback videos. AR interfaces come from the surrounding world rather than fixed screens, so the tool tracks candidate areas over time. It keeps only stable, visible areas for an automated tester to tap. Tested on four open-source apps and nine videos, it reached 55.8% branch coverage on AR code against 41.98% for the Monkey tool in the same experiment.

Catalog record

CREATORSayed Amir Mousavi and Xiaoyin Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62998
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16544

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TARIPlay: A Test Framework for AR Applications based on Interactive Area Tracking in Playback Videos DOI: 10.1145/3744916.3787817 Conference: The International Conference on Software Engineering; April 2026; Rio De Janeiro, Brazil 4 Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE ’26), April 12–18, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3744916.3787817 ISBN: 979-8-4007-2025-3/2026/04 CCS: Software and its engineering Software testing and debugging CCS: Social and professional topics Software maintenance CCS: Human-centered computing Mixed / augmented reality Sayed Amir Mousavi and Xiaoyin Wang Abstract . As Augmented Reality (AR) becomes more and more embedded in daily life, ensuring the quality, safety, and reliability of AR applications is increasingly important. However, AR apps present unique challenges for automated testing. Unlike static GUI layouts in traditional mobile apps, AR apps acquire their interaction interface from the surrounding environment, which is volatile and non-deterministic. Recent advancements like ARCore Playback and ARKit Replay allow developers to reuse real-world scenarios by recording and playing back enriched videos, enabling more feasible automated AR testing. However, using playback videos introduces two major challenges: test inputs must be timed precisely, and interactive areas in the video are dynamic, irregular, and difficult to identify. To address these challenges, we propose TARIPlay , a framework that analyzes playback videos to detect, track, and filter proper interactive areas over time for automated testing. In particular, TARIPlay identifies viable test opportunities based on criteria like stability and visibility, then feeds this information to an automated testing engine to simulate user interactions. We perform an experiment with four open-source AR apps and nine playback videos. Evaluation results show that TARIPlay significantly outperforms the existing tool Monkey in test coverage (55.8% over 41.98% on branch coverage) of AR-related code, and can also be used to assess the quality of playback videos for testing suitability. largesymbols"00 largesymbols"01 † † cc-license: by 1. Introduction Augmented Reality (AR) overlays digital content onto real-world views in real-time for immersive user experiences. Applications such as Pokémon Go ( 31 ) , Amazon ( 1 ) , IKEA Place ( 21 ) , and Google Lens ( 13 ) demonstrate AR’s transformative potential across gaming, education, healthcare, and retail ( 34 ) . As AR apps get more and more involved in daily activities, their quality, safety, and reliability attract more and more attention. Due to AR apps’ real-time interactions with complex environmental contexts in the real world, their bugs may cause more severe consequences, even immediate safety risks. For instance, in AR navigation applications, when imprecise content placement or improper occlusion handling are performed in the code, virtual elements may occlude critical aspects of the physical environment, leading to compromised user decisions and immediate risk for traffic accidents ( 35 ) . Similarly, a flaw in an AR-supported surgery system may lead to severe harm to patients if giving inaccurate guidance ( 22 ) . These real-world implications underscore the importance of robust testing and quality assurance practices in the development of AR apps. Despite the demand of high software quality, existing testing support for AR apps is still at its early stage. Traditional automated GUI testing frameworks fall short due to their inability to handle AR apps’ interactions with real-world scenes ( 28 ) . As a result, AR developers often have to rely on manually constructed testing scenes and manual test execution ( 28 ) , which are labor-intensive and inefficient. Simulated environments such as Unity Mars ( 48 ) have been recently developed, but it not yet clear how effective virtual reality scenes may simulate real-world scenes in AR testing, because they lack the noises and imperfections of the reality (e.g., coarse surfaces, diffuse reflections). Recently, ARCore ( 11 ) and ARKit ( 3 ) both introduced recording and playback capabilities (i.e., ARCore Playback ( 12 ) and ARKit Replay ( 2 ) ) to enable scenario reuse. These new features allow mobile phone users to record a video (enriched with sensor information) of the real world and run AR applications directly on the video. With these new features, the automatic execution of test cases for AR apps finally become feasible, and datasets of playback videos (e.g., ARTBank ( 56 ) ) have been constructed to facilitate AR app testing. However, testing with playback videos is a setting much different from GUI testing ( 17 ; 38 ; 27 ) , raising two novel challenges for automating the testing process. First, in automatic GUI testing, the testing process is fully controlled by the driver and we only need to feed a sequence of GUI events in order without worrying much about the time gap between them. But playback videos are strictly time-constraint, so test opportunities only occur at certain time period of a video. For example, a detected plane may show up at the 10 th second of the video and disappear at the 21 st second of the video. So an input event must be fed within the that time period to trigger a related app behavior. Second, unlike traditional GUI with predefined UI elements (e.g., buttons, checkboxes) and layouts, interactive areas (e.g., planes, human faces, collectively called trackables ) in a playback video are dynamically detected by the underlying AR framework so both their locations and bounding boxes are rapidly changing over time. Even worse, some interactive areas in the video may be of irregular shapes (e.g., a very narrow strip), moving fast on the screen, or disappearing quickly, so they may not be proper interactive areas for testing (and human users typically do not interact with them). To illustrate the two challenges, in Figure 1 , we show the dynamics of a detected plane over 2 seconds in a playback video. In the figure, the table top plane moves and disappears quickly and may not be a good interactive area to target in testing. Therefore, without knowing the time period, location, and size information about proper interactive areas in playback videos, it would be very difficult to automate AR testing with them. VR testing tools (e.g., VRTest ( 60 ) , VRGuide ( 53 ) ) cannot be applied to testing with playback videos because they rely on the flexible control of camera movement and angle to explore the virtual scene, and their goal is to create the most efficient exploration paths of the camera. However, for playback videos, once they are created, the camera’s exploration path becomes fixed, and the goal of testing tools becomes maximizing the interaction frequency and variety on the fixed exploration path. Figure 1 . Dynamics of Planes in 2 seconds In this paper, we present TARIPlay , a T est framework for AR applications based on I nteractive area tracking in Play back videos. In particular, TARIPlay addresses the special challenge of playback-video-based AR testing (compared with virtual-scene-based AR testing) by extracting test interfaces from videos. Given a Android ARCore playback video, our framework first uses the API of underlying ARCore session to extract all interactive areas within each frame of the video. Then, it uses a novel algorithm to combine interactive areas across frames and identify interactive areas that appear over a

Read the complete text