Augmented Assembly: Object Recognition and Hand Tracking for Adaptive Assembly Instructions in Augmented Reality DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Chicago, Illinois ISBN: 978-1-4503-XXXX-X/2018/06 C

Alexander Htet Kyaw, Haotian Ma, Sasa Zivkovic, Jenny Sabin · 2026

Read Augmented Assembly: Object Recognition and Hand Tracking for Adaptive Assembly Instructions in Augmented Reality DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Chicago, Illinois ISBN: 978-1-4503-XXXX-X/2018/06 C by Alexander Htet Kyaw, Haotian Ma, Sasa Zivkovic, Jenny Sabin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11535.txt.

Introduction

Alexander Htet Kyaw, Haotian Ma, Sasa Zivkovic, and Jenny Sabin guide assembly with AR. Their 2026 preprint joins object recognition with hand tracking. Augmented reality can now overlay help on physical tasks. Assembly work is a natural fit for such guidance.

The team's workflow adapts instructions as the build goes on. It first identifies custom components in front of the user. Step-by-step instructions then appear in the headset view. The system detects deviations from the planned assembly. It updates the instructions based on the user's hands. Hands-on interaction thus drives the guidance in real time. Adaptive help could cut errors in complex custom builds.

Catalog record

CREATORAlexander Htet Kyaw, Haotian Ma, Sasa Zivkovic, Jenny Sabin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-11535

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Augmented Assembly: Object Recognition and Hand Tracking for Adaptive Assembly Instructions in Augmented Reality DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Chicago, Illinois ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Human-centered computing Interaction techniques CCS: Human-centered computing Augmented reality CCS: Computing methodologies Computer vision problems CCS: Computing methodologies Machine learning approaches Alexander Htet Kyaw, Haotian Ma, Sasa Zivkovic, Jenny Sabin Abstract . Recent advances in Augmented reality (AR) have enabled interactive systems that assist users in physical assembly tasks. In this paper, we present an AR-assisted assembly workflow that leverages object recognition and hand tracking to (1) identify custom components, (2) display step-by-step instructions, (3) detect assembly deviations, and (4) dynamically update the instruction based on users’ hands-on interactions with physical parts. Using object recognition, the system detects and localizes components in real time to create a digital twin of the workspace. For each assembly step, it overlays bounding boxes in AR to indicate both the current position and the target placement of relevant components, while hand-tracking data verifies whether the user interacts with the correct part. Rather than enforcing a fixed sequence, the system highlights potential assembly errors and interprets users deviations as opportunities for iteration and creative exploration. A case study with LEGO-blocks and custom 3D-printed-components demonstrates how the system links digital instructions to physical assembly, eliminating the need for manual searching, sorting, or labeling of parts. Figure 1. Object recognition and hand tracking in augmented reality provide visual feedback during the physical assembly process. For each step, a bounding box highlights the relevant component, a red outline indicates its target placement, and the full assembly model is displayed in front. In the left image, the system displays a red cross when the user’s hand hovers over a component outside the current assembly step, while in the right image, a green check mark confirms interaction within the assembly step. teaser 1. Introduction Assembling physical objects is an embodied and tangible process that integrates perception, coordination, and spatial reasoning through direct interaction with materials and space. While digital fabrication and design tools have advanced rapidly, the act of physical assembly still relies heavily on static instructions, paper manuals, videos, or digital models, that separate guidance from action ( Lee-Cultura and Giannakos 2020 ; Yang et al. 2020 ) . This disconnect often leads to errors, inefficiency, and fragmented attention, especially when users must interpret 2D information for 3D tasks ( Henderson and Feiner 2011 ) . Emerging research in augmented and mixed reality has begun to address these challenges by overlaying digital guidance directly within the user’s environment ( Wang et al. 2022 ; Han et al. 2025 ; Kang et al. 2018 ; Hattab and Taubin 2019 ) . Yet, many existing systems remain limited by fixed component tracking or pre-registered models, offering little adaptability to real-world variations (Wei et al., 2023) ( Fang et al. 2023 ; Lee et al. 2023 ) . At the same time, advances in computer vision and machine learning present new opportunities for making assembly processes more responsive, interactive, and context-aware ( Garcia de Paiva Pinheiro and Cuperschmid 2025 ) In this work, we explore how real-time spatial understanding of assembly components, combined with users’ hand interactions, can enhance embodied and situated making ( Ma and Kyaw 2023 ; Kyaw et al. 2025b ) . Our system integrates object recognition and hand tracking within an augmented reality workflow to identify physical assembly components, monitor real-world user interactions, and dynamically update the digital assembly instructions accordingly (Figure. 1 ). The system not only visualizes assembly deviations but also adapts the assembly instructions to accommodate user improvisation and creative intent in the physical act of making. By linking digital feedback to physical action, the system supports adaptive guidance that responds to user behavior instead of enforcing a fixed sequence. The system demonstrates: • An AR assembly workflow that integrates object recognition and hand tracking to create a digital twin and provide step-by-step instructions aligned with the user’s physical interactions and real-world assembly components. • A feedback mechanism that highlights errors and detects deviations to dynamically adapt instructions based on the physical assembly state, and allows users to explore assembly alternatives guided by structural analysis. • A case study showing how the system removes the need for manual searching, sorting, or labeling of parts during physical assembly, using LEGO blocks and custom 3D-printed components as examples. 2. Related Work AR has been used to overlay digital instructions directly onto the physical workspace to support embodied assembly and fabrication task ( Chen et al. 2024 ; Liu et al. 2023 ; Funk et al. 2017 ) . Prior work demonstrates that AR can enhance task comprehension by visualizing part placement, sequencing, and spatial orientation to reduce cognitive load during assembly ( Yuan et al. 2024 ; Alves et al. 2022 ) . Recent studies on using AR for task guidance emphasize the importance of feedback and tangible interaction, however, there remain opportunities to incorporate real-time perception for adaptive assembly instructions ( Cao et al. 2023 ; Nishihara and Okamoto 2015 ) Object Recognition in Augmented Reality and Assembly Tasks. Integrating computer vision into AR has enabled more adaptive assembly workflows. Previous research has utilized object recognition to recognize assembly states, automatically advance AR instructions ( Stanescu et al. 2023 ) , align virtual model registration ( Yan 2022 ) , and verify whether a part has reached its target placement location ( Kästner et al. 2020 ) . Outside of AR-assisted assembly, vision–language models have also been used to detect assembly states in robotic assembly. However, pretrained models typically perform well only on common assembly tools and components they were trained on ( Li et al. 2025 ) . In this research, we integrate object recognition with an automatic synthetic-data generation and training pipeline to enable learning on custom assembly components such as 3D-printed parts. Object recognition is then used to detect the real-time locations of individual parts and associate them with corresponding digital instructions by dynamically highlighting both their current and target spatial regions. Hand Tracking for Augmented Reality and Error Detection. A growing body of work has investigated gesture-based AR interfaces ( Gavgiotaki et al. 2023 ; Kyaw et al. 2024b ) . Gestural recognition has been used for feedback-based assembly instructions, but primarily as an input to update the step sequence or visual display ( Kyaw et al. 2024a ; Spencer 2023 ) . Recent systems has combine hand tracking/action recognition with object detection to verify stage completion and detect errors ( Zhang et al. 2025 ; Mokuwe et al. 2022 ) . However, this is primarily implemented as a quality-control step using fixed-camera setups and computer vision, rather than paired with augmented–reality–based visual instruction displays. In this research, we integrate hand and object tracking with AR instructions to enable real-time visual guidance and error detection. Moreover, instead of merely detecting errors, our system dynamically updates the assembly instructions based on the identified errors. Synthesis. While prior systems have used object recognition to identify assembly states, advance instructions, or align virtual models, our approach extends this by visually highlighting the real-time location of individual components

Read the complete text