SesaHand: Enhancing 3D Hand Reconstruction via Controllable Generation with Semantic and Structural Alignment
Zhuoran Zhao, Xianghao Kong, Linlin Yang, Zheng Wei, Pan Hui, Anyi Rao · 2026
Read SesaHand: Enhancing 3D Hand Reconstruction via Controllable Generation with Semantic and Structural Alignment by Zhuoran Zhao, Xianghao Kong, Linlin Yang, Zheng Wei, Pan Hui, Anyi Rao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00443.txt.
Introduction
Zhao, Kong, Yang, Wei, Hui, and Rao improve synthetic hand images for three dimensional hand reconstruction, where game engine pictures lack texture variety, arms, and interacting objects. Their SesaHand method draws human behaviour meaning from captions through chain of thought inference, keeping human centred context while dropping stray scenery. Hierarchical fusion then aligns hand and body structure across scales, with extra attention on hand regions. These images beat prior work and also lift reconstruction accuracy.
Catalog record
| CREATOR | Zhuoran Zhao, Xianghao Kong, Linlin Yang, Zheng Wei, Pan Hui, Anyi Rao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54191 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.00443
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SesaHand: Enhancing 3D Hand Reconstruction via Controllable Generation with Semantic and Structural Alignment Zhuoran Zhao, Xianghao Kong, Linlin Yang, Zheng Wei, Pan Hui, Anyi Rao Abstract Recent studies on 3D hand reconstruction have demonstrated the effectiveness of synthetic training data to improve estimation performance. However, most methods rely on game engines to synthesize hand images, which often lack diversity in textures and environments, and fail to include crucial components like arms or interacting objects. Generative models are promising alternatives to generate diverse hand images, but still suffer from misalignment issues. In this paper, we present SesaHand, which enhances controllable hand image generation from both se mantic and s tructural a lignment perspectives for 3D hand reconstruction. Specifically, for semantic alignment, we propose a pipeline with Chain-of-Thought inference to extract human behavior semantics from image captions generated by the Vision-Language Model. This semantics suppresses human-irrelevant environmental details and ensures sufficient human-centric contexts for hand image generation. For structural alignment, we introduce hierarchical structural fusion to integrate structural information with different granularity for feature refinement to better align the hand and the overall human body in generated images. We further propose a hand structure attention enhancement method to efficiently enhance the model’s attention on hand regions. Experiments demonstrate that our method not only outperforms prior work in generation performance but also improves 3D hand reconstruction with the generated hand images. 1 Introduction 3D hand reconstruction from a single image plays a crucial role in computer vision, human-computer interaction, and AR/VR applications ( 23 ; 26 ; 1 ) . It also has a great potential in dexterous robotic manipulation for embodied intelligence ( 18 ; 38 ) . Current state-of-the-art methods ( 60 ; 40 ; 27 ) rely on large amounts of accurate ground-truth labels for training, but obtaining these labels is expensive and time-consuming. Therefore, synthetic data has gained growing interest in the computer vision community ( 54 ; 5 ; 57 ; 24 ) . Synthetic hand datasets typically feature various hand poses and are rendered by game engines to enhance realism ( 17 ; 31 ) . Several recent studies have explored training 3D hand reconstruction models with real and synthetic data, achieving better results than only training with real data ( 31 ; 69 ) . However, hand texture maps and backgrounds used to create synthetic hand datasets are limited, which affects the diversity of synthetic data. Hands are often placed in environments incompatible with the contextual semantics . Moreover, most synthetic hand datasets only show floating hands without arms and hardly feature hand-object interaction (see Fig. 1 (a)), conflicting with the human body structure and failing to reflect real human behavior. Diffusion models have shown impressive capability in generating realistic images from textual descriptions ( 51 ; 21 ) . Texts are used as semantic controls to provide images with diverse settings. Several studies further explore controllable image generation with additional control modalities ( 66 ; 44 ) , which unlocks the potential for paired data generation. Pre-trained diffusion models gain rich human prior information ( 8 ; 30 ) from massive training data, and controllable generative models ensure image-label alignment. These make diffusion models a good alternative to mitigate the limitations of existing hand image synthesis approaches. Figure 1: (a) We present a controllable hand image generation method that generates diverse hand images with semantic and structural alignment. (b) 3D hand reconstruction performance in the wild can be improved with better semantic- and structural- aligned generated images. Several works employ controllable diffusion models to generate hand data. However, they are restricted to lab environments ( 59 ) , suffer from inefficient training for hand-related feature refinement ( 46 ) , or are skeleton-based, ignoring the influence of hand shape ( 11 ) . Moreover, prior work ( 46 ) directly uses the large vision-language model (VLM) for image description generation, which poses challenges for plausible hand image generation due to the overthinking issues ( 58 ) of VLMs. The overthinking issue can introduce irrelevant information into image descriptions, creating a semantic gap from insufficient human behavior context and leading to implausible hand image generation. Moreover, since the hand is a critical part of the human body, neglecting the structural information of the human body associated with the hand can cause misalignment issues in generated images, such as floating hands and implausible human poses (see Fig. 1 (a)). With these motivations in mind, we propose SesaHand, enhancing text-conditioned controllable hand image generation from both semantic and structural alignment perspectives. For semantic alignment, we find that the human-centric context within the image caption is crucial for generating plausible hand images and can be decomposed into four components: human pose, action, hand action, and the environment through prompt influence and attention analyses. We define this human-centric context as human behavior semantics. To extract this semantics effectively, we propose a pipeline with Chain-of-Thought (CoT) inference ( 56 ; 67 ) , which employs a step-by-step thinking approach to identify and extract human behavior semantics while eliminating irrelevant details. Specifically, the CoT inference consists of three stages. First, a \mathrm{Captioner} generates an initial image caption for an input image. Second, an \mathrm{Extractor} identifies and decomposes essential components of the image caption with few-shot learning. Third, a \mathrm{Composer} composes the extracted components together into the final text prompt. The proposed CoT inference can generate image descriptions with sufficient semantic information about human behavior, facilitating the construction of improved text-image pairs to train the text-to-image model. In diffusion-based models, self-attention maps preserve geometric and structural information of images, which has been proven useful for image editing ( 34 ) and human image animation ( 8 ) . This motivates us to strengthen the alignment of the hand and human body in controllable image generation by leveraging the structural information within self-attention maps. Specifically, we integrate structural information with different granularity with both global and local representations by extracting multi-resolution self-attention maps in the encoding and middle blocks of the control module ( 66 ) and hierarchically fuse them to refine the features fed into the Stable Diffusion generative backbone ( 51 ) . This helps provide enhanced structural information useful for hand image generation. Moreover, cross-attention maps reveal the semantic correspondence between image features and text embeddings. To highlight the local hand structure in image features, we propose an efficient approach of adding bias terms to the hand-related cross-attention maps, rather than relying on a slow embedding refinement process ( 46 ) . Extensive experiments demonstrate that our method outperforms existing approaches in hand image generation. Notably, we further show that the generated hand images improve the 3D hand reconstruction performance on in-the-wild datasets (see Fig. 1 (b)), paving the path for exploring the use of synthetic data from generative models in 3D hand reconstruction task. We summarize our contributions as follows: 1. We propose a CoT inference-based pipeline that extracts human behavior semantics to construct text-image pairs for training, which mitigates overthinking issues in VLM-generated captions and improves semantic alignment in hand image generation. 2. We improve the structural alignment in hand image generation through a hierarchical structural fusion method that integrates multi-level features for hand-body alignment, along with a hand