Can Vision-Language Models Understand Construction Workers? An Exploratory Study
Hieu Bui, Nathaniel E. Chodosh, Arash Tavakoli · 2026
Read Can Vision-Language Models Understand Construction Workers? An Exploratory Study by Hieu Bui, Nathaniel E. Chodosh, Arash Tavakoli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10835.txt.
Introduction
Hieu Bui, Nathaniel E. Chodosh, and Arash Tavakoli ask whether vision-language models can read construction workers' behavior. Robots are joining construction workflows step by step. To work beside people safely, a robot must interpret what workers are doing. Vision-language models promise such reading without long domain-specific training.
The study is exploratory: it probes what these models get right and wrong on site imagery. Construction scenes are cluttered, dusty, and full of partial views. A model that shines on clean benchmarks may falter there. The results help robotics teams judge how far general models can be trusted around workers, and where custom training is still a must.
Catalog record
| CREATOR | Hieu Bui, Nathaniel E. Chodosh, Arash Tavakoli |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-10835
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can Vision-Language Models Understand Construction Workers? An Exploratory Study Hieu Bui, Nathaniel E. Chodosh, Arash Tavakoli Abstract As robotics become increasingly integrated into construction workflows, their ability to interpret and respond to human behavior will be essential for enabling safe and effective collaboration. Vision-Language Models (VLMs) have emerged as a promising tool for visual understanding tasks and offer the potential to recognize human behaviors without extensive domain-specific training. This capability makes them particularly appealing in the construction domain, where labeled data is scarce and monitoring worker actions and emotional states is critical for safety and productivity. In this study, we evaluate the performance of three leading VLMs-GPT-4o, Florence 2, and LLaVa-1.5-in detecting construction worker actions and emotions from static site images. Using a curated dataset of 1,000 images annotated across ten action and ten emotion categories, we assess each model’s outputs through standardized inference pipelines and multiple evaluation metrics. GPT-4o consistently achieved the highest scores across both tasks, with an average F1-score of 0.756 and accuracy of 0.799 in action recognition, and an F1-score of 0.712 and accuracy of 0.773 in emotion recognition. Florence 2 performed moderately, with F1-scores of 0.497 (action) and 0.414 (emotion), while LLaVa-1.5 showed the lowest overall performance (F1-scores of 0.466 for action and 0.461 for emotion). Confusion matrix analyses revealed that all models struggled to distinguish semantically close categories-such as “Collaborating in teams” versus “Communicating with supervisors,” or “Focused” versus “Determined”-highlighting limitations in current VLMs when applied to visually nuanced, domain-specific tasks. While the results indicate that general-purpose VLMs can offer a baseline capability for human behavior recognition in construction environments, further improvements-such as domain adaptation, temporal modeling, or multimodal sensing-may be needed for real-world reliability. This study provides an initial benchmark and practical insights for deploying human-aware AI systems in complex, safety-critical settings. I Introduction The construction industry is transforming rapidly with the introduction of robotic systems-from drones and autonomous vehicles to quadruped robots-designed to improve job site efficiency, safety, and monitoring [ 46 , 58 , 59 , 49 ] . These agents must navigate complex, unstructured environments while collaborating with human workers. However, for effective collaboration, robots must go beyond obstacle avoidance or object recognition to understand human behavior [ 70 , 40 , 20 , 59 , 58 ] . This understanding encompasses not only worker actions but also their physiological and emotional states, whether during tasks or in isolation [ 6 , 18 , 30 , 37 ] . Much like physical activities, cognitive states such as fatigue or anxiety critically influence safety, decision-making, and site coordination [ 18 , 5 ] . Consequently, recent research has increasingly focused on automating state detection using wearable sensors [ 65 , 3 , 31 , 17 ] and computer vision [ 44 , 41 , 56 ] to support real-time monitoring and adaptive decision-making. While traditional computer vision models rely on large labeled datasets and lack generalizability [ 69 , 38 ] , Vision-Language Models (VLMs) offer a robust alternative. For instance, task-specific architectures like YOLO [ 35 ] often require extensive retraining to adapt to new contexts. Conversely, VLMs such as GPT-4o integrate visual and linguistic processing to interpret complex scenes through natural language reasoning [ 69 ] . Although these models excel in general multimodal tasks, their ability to understand human behavior in safety-critical construction environments remains largely unexplored [ 6 , 12 ] . This study is an explanatory approach to investigating whether general-purpose VLMs, without fine-tuning for the construction domain, can accurately detect both the actions and emotional states of construction workers from still images, compared to labels generated by a human annotator. Our work paves the way for integrating VLMs into real-time monitoring systems that support human-robot collaboration in complex and safety-critical environments. By examining the performance of three state-of-the-art models on a curated dataset of labeled construction site images, we aim to evaluate their ability to serve as a foundation for human-aware robotic systems in real-world job sites. The rest of the paper is organized as follows: we begin by reviewing related work on worker state monitoring and vision-language modeling. We then describe our dataset, labeling process, and experimental pipeline, followed by an evaluation of model performance. Finally, we discuss the implications of our findings, limitations, and directions for future research. II Background Information Automating site monitoring is a growing priority in the architecture, engineering, and construction (AEC) industry [ 48 , 53 ] . Given the dynamic and hazardous nature of construction environments, non-intrusive tracking of worker behavior, equipment, and site progress is essential for enhancing safety, productivity, and real-time decision-making [ 27 , 11 , 55 ] . In line with the automation goals, robotics adoption in construction is gaining momentum due to their potential to enhance productivity, efficiency, and worker safety by taking on hazardous tasks [ 45 ] . As the interaction level between the robot and the human increases, the communication of needs, states, and expectations between the two entities become more necessary [ 1 ] . In line with this expectation, studies have started to propose various methods to understand and predict workers’ states in real time. Worker state can be broadly defined as the physical, cognitive, and emotional condition of a worker at any given time, encompassing aspects such as fatigue, stress, attentiveness, workload, and overall well-being [ 8 , 17 , 30 ] . Accurately assessing these states is critical in high-risk construction environments, where lapses in attention or physical exhaustion can result in accidents or decreased performance [ 45 , 49 , 1 , 58 ] . In this regard, prior studies collectively show that worker’s physiological measures, facial expression, actions, and contextual metrics can be used as an input for predicting various states such as cognitive workload, performance metrics (e.g., attention level), or stress when collaborating with or without a robotic agent [ 45 , 49 , 1 , 58 , 68 , 26 ] . On the physiological side, studies have demonstrated the utility of cardiovascular metrics, eye tracking, and brain activity measures for recognizing worker states such as fatigue [ 11 , 9 , 42 , 39 ] , stress [ 34 , 33 , 63 , 50 ] , and cognitive load [ 29 , 16 , 28 ] . While these physiological indicators are valuable, they often require the use of additional wearable sensors, which may interfere with workers’ comfort or freedom of movement on-site. To address these limitations, researchers are increasingly exploring vision-based approaches that rely solely on video data to predict worker states without intrusive sensors [ 68 , 42 ] . These studies demonstrate the potential of computer vision to assess both physical actions and cognitive conditions. For instance, Yu et al. monitored physical fatigue using a vision-based 3D motion capture system and biomechanical modeling; they demonstrated that joint-level fatigue can be assessed automatically using only RGB video, eliminating the need for wearables [ 68 ] . Similarly, Li et al. investigated mental fatigue in equipment operators using eye-tracking, showing that supervised learning models (specifically SVM) can classify fatigue levels with up to 85% accuracy [ 42 ] . Similar approaches were taken for understanding worker actions using computer