HandVQA: Diagnosing and Improving Fine-Grained Spatial Reasoning about Hands in Vision-Language Models
MD Khalequzzaman Chowdhury Sayem, Mubarrat Tajoar Chowdhury, Yihalem Yimolal Tiruneh, Muneeb A. Khan, Muhammad Salman Ali, Binod Bhattarai, Seungryul Baek, [3pt] UNIST · 2026
Read HandVQA: Diagnosing and Improving Fine-Grained Spatial Reasoning about Hands in Vision-Language Models by MD Khalequzzaman Chowdhury Sayem, Mubarrat Tajoar Chowdhury, Yihalem Yimolal Tiruneh, Muneeb A. Khan, Muhammad Salman Ali, Binod Bhattarai, Seungryul Baek, [3pt] UNIST (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26362.txt.
Introduction
Sayem and co-authors introduce HandVQA, a large test of how well vision-language models read hand poses. Fine hand detail matters in settings like robot-assisted surgery and chip work. The bench holds over 1.6 million multiple-choice questions drawn from three 3D hand datasets. Questions probe angles, distances, and positions between hand joints. Tests of LLaVA, DeepSeek, and Qwen-VL models expose made-up finger parts and weak geometry, while light fine-tuning with LoRA gives a clear path to gains.
Catalog record
| CREATOR | MD Khalequzzaman Chowdhury Sayem, Mubarrat Tajoar Chowdhury, Yihalem Yimolal Tiruneh, Muneeb A. Khan, Muhammad Salman Ali, Binod Bhattarai, Seungryul Baek, [3pt] UNIST |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 88498 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.26362
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HandVQA: Diagnosing and Improving Fine-Grained Spatial Reasoning about Hands in Vision-Language Models MD Khalequzzaman Chowdhury Sayem, Mubarrat Tajoar Chowdhury, Yihalem Yimolal Tiruneh, Muneeb A. Khan, Muhammad Salman Ali, Binod Bhattarai, Seungryul Baek, [3pt] UNIST Abstract Understanding the fine-grained articulation of human hands is critical in high-stakes settings such as robot-assisted surgery, chip manufacturing, and AR/VR-based human–AI interaction. Despite achieving near-human performance on general vision-language benchmarks, current vision-language models (VLMs) struggle with fine-grained spatial reasoning, especially in interpreting complex and articulated hand poses. We introduce HandVQA, a large-scale diagnostic benchmark designed to evaluate VLMs’ understanding of detailed hand anatomy through visual question answering. Built upon high-quality 3D hand datasets (FreiHAND, InterHand2.6M, FPHA), our benchmark includes over 1.6M controlled multiple-choice questions that probe spatial relationships between hand joints, such as angles, distances, and relative positions. We evaluate several state-of-the-art VLMs (LLaVA, DeepSeek and Qwen-VL) in both base and fine-tuned settings, using lightweight fine-tuning via LoRA. Our findings reveal systematic limitations in current models, including hallucinated finger parts, incorrect geometric interpretations, and poor generalization. HandVQA not only exposes these critical reasoning gaps but provides a validated path to improvement. We demonstrate that the 3D-grounded spatial knowledge learned from our benchmark transfers in a zero-shot setting, significantly improving accuracy of model on novel downstream tasks like hand gesture recognition (+10.33%) and hand-object interaction (+2.63%). Project page, code, and dataset: https://kcsayem.github.io/handvqa/ . Figure 1 : Overview of HandVQA’s transfer effect. Fine-tuning a base Vision-Language Model (VLM) on HandVQA teaches it explicit 3D hand geometry and joint-level spatial reasoning. The resulting Spatial-Aware VLM exhibits zero-shot generalization to novel downstream tasks: both image-based gesture recognition and video-based hand-object interaction recognition. Spatial-Aware VLM achieves consistent accuracy gains without task-specific training. 1 Introduction Figure 2 : Overview of HandVQA Question Format. This figure illustrates the structure of our benchmark, which divides hand pose estimation into five sub-tasks: Angle, Distance, and Relative Position along X, Y, and Z axes. A hand image with annotated joint indices (top left) supports multiple-choice questions per task type, derived from 3D joint coordinates and the correct answers are shown in green . Imagine a high-stakes environment like robot-assisted surgery, where a subtle misinterpretation of a surgeon’s hand gesture whether a finger is slightly bent or fully extended can mean the difference between a successful operation and a critical error 42 ; 19 . Or consider chip manufacturing, where robotic agents mimic human motions with microscopic precision, and a misjudged fingertip angle can cause costly defects 17 ; 55 . In such scenarios, understanding the fine-grained articulation of the human hand is not just desirable, it is essential. Vision Language Models (VLMs) have rapidly become the perceptual and decision-making core of embodied agents, AR/VR platforms, and multimodal assistants due to their ability to unify visual perception and natural-language reasoning within a single foundation model 43 . As these systems increasingly rely on such unified reasoning, fine-grained hand-pose understanding becomes crucial ( 61 ; 4 ) . Hands are often the primary medium through which humans convey actions, intent, and control ( 14 ) ; misinterpreting whether a finger is curled, pointing, or relaxed can result in semantic confusion or physical failure. For VLMs to be safely deployed in these real-world contexts, robust, fine-grained hand understanding is a foundational requirement. Despite significant advancements in general visual question answering (VQA), where state-of-the-art VLMs often achieve near-human accuracy on tasks like VQAv2 ( 2 ) , they notably falter on tasks involving detailed spatial reasoning ( 67 ; 10 ; 50 ; 32 ) . Recent benchmarks highlight their particular weaknesses with basic spatial distinctions like left versus right, achieving only around 56% accuracy compared to human performance at 99% ( 28 ) . These shortcomings point to a reliance on superficial correlations rather than genuine geometric comprehension. Articulated hand poses, characterized by complex spatial relationships across 21 joints, remain a significant challenge. To systematically address this challenge, we introduce HandVQA , a diagnostic benchmark specifically designed to evaluate VLMs’ fine-grained understanding of hand pose and anatomy through targeted visual question answering. HandVQA is constructed using precise 3D annotations from widely-used datasets—FreiHAND ( 70 ) , InterHand2.6M ( 46 ) , and FPHA ( 20 ) . We disentangle the hand pose estimation task into five separate subtasks: joint angles (measured at a single joint) and distances and relative positions along the X, Y, and Z axes (measured between pairs of joints) (see Fig. 2 ). While we draw conceptual motivation from 15 , our formulation is tailored specifically to the spatial nuances of hand anatomy. Our pipeline generates controlled multiple-choice questions probing these specific joint angles, distances, and relative positions, thereby minimizing ambiguity and encouraging genuine spatial reasoning (e.g., "Is the distal interphalangeal joint of the middle finger closer to that of the ring finger or the index finger?" ). By mapping 3D hand joint relations into structured natural language, HandVQA enables a diagnostic view into how well VLMs grasp spatial pose concepts. We evaluate several open-source large VLMs, including LLaVA ( 36 ) , Qwen-VL ( 3 ) , and DeepSeek ( 41 ) . These models are fine-tuned using parameter-efficient LoRA adapters ( 22 ) and evaluated on held-out samples from the same datasets used for training. Our findings indicate significant performance gaps: base models generally perform poorly, often similar to or below random guess, particularly on distance-related questions, reflecting a fundamental lack of spatial grounding. While fine-tuning markedly improves performance, confirming that VLMs can acquire spatial awareness with sufficient data; significant limitations persist, especially regarding the intricate task of accurately interpreting joint angles. Our detailed analysis reveals key insights: VLMs often settle for simplified answers (e.g., repeatedly predicting “close” for distances or “slightly bent” for angles) indicating superficial alignment rather than true understanding. Furthermore, we observe that performance superiority in one spatial reasoning task rarely generalizes across others, highlighting that current VLM architectures might require stronger vision encoders or full-model fine-tuning to robustly capture intricate spatial relationships. The implications of our study extend beyond benchmarking. Accurate hand pose interpretation is crucial in robotics for safe human-robot interactions, in AR/VR for immersive user experiences, and in medical assistance for sterile and precise gesture-based device control ( 16 ; 4 ; 47 ; 65 ) . HandVQA addresses a critical gap in existing benchmarks, but our work demonstrates it is more than just a diagnostic tool; it is a robust training resource. We hypothesize that by mastering the 3D-grounded spatial reasoning in HandVQA, models acquire a transferable skill. We validate this by showing that models trained with HandVQA achieve significant zero-shot generalization to novel, unseen downstream tasks, including static Gesture Recognition and dynamic, video-based Hand-Object Interaction (See Fig. 1 ). HandVQA thus provides a concrete path to improving VLM spatial reasoning capabilities broadly. To summarize our contributions include: • We introduce a large-scale benchmark with over 1.6 million questions, created via an automated pipeline that generates anatomically-grounded VQA queries about 3D hand joint angles, distances, and relative positions. • We provide a Comprehensive evaluation of leading VLMs (e.g., LLaVA, DeepSeek, Qwen-VL), revealing their systemic failures in base spatial grounding (often performing at or below random guess) and highlighting persistent challenges, such as interpreting fine-grained joint angles,