Synchronized Realities: Towards Magic Mobile Experiences through Aligned AR Conference: CHI 2026 Workshop on Next Steps for Augmented Reality On-the-Move: Challenges & Opportunities; Aril 2026; Barcelona, Spain

Jan Henry Belz · 2026

Read Synchronized Realities: Towards Magic Mobile Experiences through Aligned AR Conference: CHI 2026 Workshop on Next Steps for Augmented Reality On-the-Move: Challenges & Opportunities; Aril 2026; Barcelona, Spain by Jan Henry Belz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15924.txt.

Introduction

Jan Henry Belz reflects on synced realities for context-aware AR in mobility settings. Matching what users see, hear and feel matters for immersion in virtual reality, but real-world elements often lie beyond user control in AR. Generative AI now enables on-demand content creation. Combined with context data, it allows cases that align smoothly with the user's setting. Belz presents synced cases, compares their shared traits and differences, and discusses the upside and risks of syncing AR with the real world.

Catalog record

CREATORJan Henry Belz
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE11734
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15924

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Synchronized Realities: Towards Magic Mobile Experiences through Aligned AR Conference: CHI 2026 Workshop on Next Steps for Augmented Reality On-the-Move: Challenges & Opportunities; Aril 2026; Barcelona, Spain Jan Henry Belz Abstract . In virtual reality environments, the alignment of perceptual modalities is crucial for immersion and presence. In the AR domain, it is difficult to create such alignments because elements in the physical world are often beyond the user’s control. However, recent advances in generative AI enable on-demand content creation, enabling highly reactive AR experiences. Combined with contextual information about the physical world, it has become possible to design experiences that seamlessly align with the user’s environment. In this reflection paper, I emphasize the importance of "synchronized" realities for context-aware AR experiences, particularly in mobility scenarios. I present several examples of existing synchronized experiences and examine their commonalities and distinctions. Finally, I discuss opportunities and pitfalls of synchronizing AR experiences with the physical world. 1. Introduction The alignment of perceptual modalities is a core requirement for immersive virtual reality (VR) scenarios ( 10 ) . In virtual reality, the entire world can be controlled by the designer or the user; however, this is not possible in augmented reality (AR) scenarios. Physical elements of the real world can only be accessed through embedded hardware, such as in smart home environments (ubiquitous computing) or, in mobility scenarios, in-cabin technology (e.g., cars or buses) ( 14 ) . Therefore, the AR experience must adapt to the physical world to provide a seamless, integrated experience ( 4 ) . Recent advances in artificial intelligence (AI) research enabled almost human-like zero-shot interpretation of user context (or at least a mimicked version of it) ( 15 ) . Natural-language interaction with multimodal perception models enables automated description of a user’s environment in "human-readable" ways, e.g., to capture the "vibe" or atmosphere of a moment. Moreover, generative AI allows the creation of customized media content that can be presented to users on demand. For example, information such as images and videos relevant to the current situation can be queried, generated, and displayed in real time. When these methods are combined meaningfully, the user’s current situation can be analyzed in depth, and an appropriate user experience can be delivered ( 8 ) . While AR generally refers to the visual overlay of information ( 1 ) , mixing and augmenting reality with contextual information can occur across multiple modalities (e.g., audio augmented reality (AAR) ( 3 ) ). This offers wide opportunities, particularly for mobile applications, such as adaptive music experiences ( 9 ) . Since mobile scenarios involve rapidly changing contextual information, mobile AR applications need to be even more reactive than in static environments. Therefore, to perfectly align the digital experience with the physical reality, The resulting "synchronization" of the digital and physical realities offers enormous potential for seamless, truly ubiquitous user experiences. Through meticulously synchronized experiences, technology can feel like magic, supporting the user in their current activity with zero friction. In this reflection paper, I want to point out the potential of "synchronized reality". I present existing work that proved the effectiveness of such synchronizations, including my own contribution to this area. In three example scenarios, I illustrate commonalities and core aspects of synchronizing AR experiences to the user context. Finally, I discuss opportunities and pitfalls for designing such experiences. 2. Example Scenarios Synchronized music A commuter leaves the office at dusk and starts walking home through a softly lit residential area. Their music player happens to land on a track whose tempo aligns with their pace: the beat sits right on the rhythm of their steps, and the arrangement “opens up” just as they turn onto a quieter street lined with warm streetlights. Nothing in the system explicitly references the sunset or the neighborhood, yet the combination of a steady pace, mellow harmony, and restrained energy makes the moment feel authored, as if the city and the soundtrack are moving in the same meter. 9 create such a meticulous adaptation of the music that the user currently listens to. By synchronizing musical elements with visual landmarks in the environment (such as tunnel exits or highway ramps), they produce an immersive and seamless integration of digital and physical components. Contextual storytelling On a weekend trip, someone drives through a city they don’t know well, listening to an audiobook they selected days earlier. Mid-chapter, the narrator starts describing the very district the listener is currently passing through, mentioning a riverside promenade, a specific bridge silhouette, and the sound of trams in the distance. As the listener looks up and sees the same bridge and hears a tram pass, the story’s setting and the physical place snap into alignment. In my previously published research project Story-Driven , I implemented this scenario in a working prototype by utilizing on-demand AI-generated stories. Narrating an audiobook in real time allows for adjusting and guiding the plot in the right direction, no matter where the user is moving ( 2 ) . Visual AR retrospection In the morning, a photo app shows a small "memory" card on the lock screen: a picture from exactly one year ago of friends sitting outside with coffee, wrapped in scarves. Today happens to be similarly cold and bright, and the user is about to meet one of those friends again. The resurfaced image is not just temporally correct; it also matches the day’s atmosphere and upcoming social context, prompting the user to recall the mood, the conversation, and even the route they took. Similarly, 5 captured a continuous stream of images from the user’s first-person perspective, which was later used to recall specific moments of the day, thereby augmenting the user’s memory using contextual data. Nowadays, wearable AR devices can achieve the same effect on demand, providing an automatic, seamlessly integrated memory aid. Several other examples exist as research prototypes: stress reduction through an ambient device ( 12 ) or breathing guide ( 6 ) , real-time adjustment to audience engagement ( 7 ) , and adaptive, immersive VR environments ( 11 ) are research areas that have been investigated, built, and evaluated by HCI researchers. While this highlights the potential of meticulous synchronization between the physical and real world, the publication dates also show how sparse the research coverage of such experiences is. 3. Discussion This set of three different scenarios has already been implemented in existing research prototypes. Considering the technical possibilities that context-aware AR combined with generative AI provides, many more scenarios are feasible. When looking at these scenarios from a broader point of view, five key commonalities arise: • Anchor: Every scenario has an initial set of variables that define the situational context. In the example of the synchronized music, this is the visual stimulus of the environment ("softly lit residential area") and the rhythm of the user’s steps. Identifying and sensing the anchor is crucial for providing a well-synchronized experience. • Precision: Depending on the type of user experience, elements may have to be adjusted more or less precisely, from multiple times per second to only once per hour. For example, in our storytelling project Story-Driven , we checked for new data every five seconds ( 2 ) ,

Read the complete text