Eye-in-Finger: Smart Fingers for Delicate Assembly and Disassembly of LEGO

Zhenran Tang, Ruixuan Liu, Changliu Liu · 2025

Read Eye-in-Finger: Smart Fingers for Delicate Assembly and Disassembly of LEGO by Zhenran Tang, Ruixuan Liu, Changliu Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06848.txt.

Introduction

Zhenran Tang, Ruixuan Liu, and Changliu Liu put a camera in the fingertip of a robot gripper to handle tight toy brick builds, in a paper from 2025. Fixed or wrist cameras lose sight of small parts in clutter, yet brick fits demand sub millimeter care.

Their smart finger carries a cheap, sharp camera at the tool tip, so the robot sees the fit point up close as it works. Tests on brick assembly and teardown showed live fine fixes to aim errors. The slack for aim drift grew five fold over the base setup, which eased robust work in tight stacks.

Catalog record

CREATORZhenran Tang, Ruixuan Liu, Changliu Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06848

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Eye-in-Finger: Smart Fingers for Delicate Assembly and Disassembly of LEGO Zhenran Tang, Ruixuan Liu, Changliu Liu Abstract Manipulation and insertion of small and tight-toleranced objects in robotic assembly remain a critical challenge for vision-based robotics systems due to the required precision and cluttered environment. Conventional global or wrist-mounted cameras often suffer from occlusions when either assembling or disassembling from an existing structure. To address the challenge, this paper introduces “Eye-in-Finger”, a novel tool design approach that enhances robotic manipulation by embedding low-cost, high-resolution perception directly at the tool tip. We validate our approach using LEGO assembly and disassembly tasks, which require the robot to manipulate in a cluttered environment and achieve sub-millimeter accuracy and robust error correction due to the tight tolerances. Experimental results demonstrate that our proposed system enables real-time, fine corrections to alignment error, increasing the tolerance of calibration error from 0.4mm to up to 2.0mm for the LEGO manipulation robot. I Introduction Fig. 1: (a): Top: EiF on a robot, holding a LEGO brick. Bottom: Cut-out views of EiF with key dimensions in millimeters. (b): A robot using EiF to correct misalignment and pick up LEGO bricks from a static board (top) and a moving hand (bottom). (c): Challenges with conventional approaches in LEGO manipulation that EiF addresses. Humans rely on vision for overall spatial perception but depend on tactile sensing for fine-grained, high-precision interactions [ 1 ] . For example, when threading a needle, placing a microchip on a circuit board, or performing delicate sutures in surgery, visual guidance provides an initial estimate, while tactile feedback refines positioning and detailed operations. In robotic systems, visual feedback similarly serves as a powerful tool for global perception, enabling high-level scene understanding and object localization [ 2 , 3 ] . However, for precise manipulation of small objects, traditional vision-based approaches face critical challenges related to accuracy and occlusion. Robotic manipulation of small objects presents multiple challenges, among which occlusion and precision are particularly important. In fine-detail tasks such as assembling with small components or performing surgery, the end effector is often larger than the target components, increasing the probability of occlusion when utilizing vision guidance. Moreover, vision systems often fail to provide the position estimation accuracy required for precise manipulation of small objects [ 4 ] . While tactile sensing offers high-resolution local perception, it suffers from inherent limitations, including high cost, durability concerns, and prolonged feedback latency [ 5 ] . Additionally, tactile sensors typically necessitate the use of soft materials at the point of contact, imposing constraints on mechanical design and further restricting their applicability in industrial and medical settings. To bridge this gap, we introduce Eye-in-Finger (EiF) , a vision-integrated sensing method that delivers low-cost, high-resolution perception directly from the tip of the end effector. Our approach provides precise spatial information in translation and rotation, achieving levels of accuracy traditionally reserved for contact-based tactile sensing. By embedding vision at the contact point, EiF effectively overcomes occlusion challenges and enables real-time, fine adjustments during delicate manipulation tasks. This approach allows robots to refine their actions dynamically, just as a human does by combining visual guidance with fingertip tactile feedback. In this work, we use robotic LEGO assembly and disassembly as a structured test case to evaluate the effectiveness of the EiF approach. LEGO-based manipulation demands sub-millimeter accuracy due to the tight tolerances of its interlocking mechanisms, making it an ideal benchmark for testing high-precision robotic systems. Fig. 1 (b) illustrates the precision constraint. If the tool is not precisely aligned with the brick, it will collide with the brick and likely damage the object. In addition, if bricks are not firmly pressed together, a minuscule tilt can occur as shown in Fig. 3(a) , which needs to be detected promptly. Popular setup with the use of wrist mounted camera are prone to both visual occlusion and physical obstruction, as the evolving LEGO structure itself can often block the line of sight, and the added bulk hampers reachability when assembling clutered LEGO structures (shown in Fig. 1 (c)). These challenges highlight the need for localized, end-effector-level perception to ensure precise tool-to-brick alignment and robust error correction during dynamic assembly tasks. I-A Contributions To address these challenges, this paper presents Eye-in Finger, a hardware-software co-design approach for accurate, occlusion-free LEGO position estimation in robotic assembly. Specifically, our contributions include: 1. A novel end-of-arm tool (EOAT) design with an embedded camera, integrating a vision system directly into the contacting tool tip. The design enables precise perception and guarantees a clear view of the object to manipulate. 2. A vision-based pipeline capable of precisely estimating LEGO brick poses and correct misalignment 3. Experimental validation demonstrating significantly improved accuracy and robustness over prior approaches. By integrating hardware innovations with real-time perception algorithms, this work provides a practical solution for vision-guided LEGO manipulation and offers insights into addressing occlusion and precision challenges in robotic assembly. II Related Work II-A Eye-in-hand Visual Servoing Recent studies have successfully integrated cameras in grippers that use visual servoing to grasp items [ 6 , 7 ] . Typically, these methods employ a camera mounted between the gripper to maintain an unobstructed view of the target object, thereby shrinking the footprint and reducing occlusion from the gripper itself. Their approaches have mainly targeted larger objects with high tolerances, such as harvesting tomatoes or grasping door handles. These configurations are less suitable for tasks that require a small stationary tool instead of grippers, such as LEGO assembly, where the toolhead must remain fixed. Therefore, our approach integrates the camera inside the toolhead—further constraining its size. We also emphasize fine-grained local perception, achieving the sub-millimeter accuracy necessary for manipulating LEGO bricks. II-B Tactile Feedback High-precision feedback has been achieved through vision-based tactile sensing, which demonstrates promise for accurately localizing small objects and performing insertion tasks with tight tolerance [ 8 , 9 ] . However, these techniques usually require specially manufactured sensing hardware, which is generally costly. In addition, contemporary tactile sensors require specialized soft materials at the point of contact, which imposes additional challenges for applications such as LEGO manipulation, which requires a rigid and firm contact to force snapping over the knobs on the bricks to establish stable connections. Our approach, by contrast, achieves comparable perceptual accuracy with far lower cost and do not require physical contact, thereby broadening the range of applicable tasks. (a) Knobs segmentation. (b) Masking. (c) Bounds estimation. (d) Silhouette reconstruction. (e) Compute offset. Fig. 2: Illustration of the pipeline for pose estimation from partially occluded LEGO target. II-C LEGO Manipulation Prior works [ 10 , 11 ] propose reinforcement learning based LEGO manipulation policies but are difficult to transfer to the real world for customized LEGO assembly. A recent study [ 12 ] presents a hardware-software co-design that enables industrial robots to manipulate LEGO bricks efficiently. This system leverages specialized EOAT designs and the inherent precision of industrial robots to perform LEGO assembly tasks. [ 13 ] extends the EOAT to a bimanual system and enables robots to build customized and delicate LEGO structures. However, these works operate with only force feedback, necessitating accurate calibration, and are susceptible to the accumulation of tolerance errors, particularly in complex, delicate assemblies. Our work addresses

Read the complete text