Passive Construction Site Safety Monitoring via Persona-Scaffolded Adversarial Chain-of-Thought VLM Verification

Ananth Sriram, Neel Mokaria, Rajveer Singh, [4pt] Department of Computer Science, University of Maryland, College Park, MD, USA {asriram2, nmokaria, rajveer}@umd.edu, [2pt] Authors listed in alphabetical order; all contributed equally. · 2026

Read Passive Construction Site Safety Monitoring via Persona-Scaffolded Adversarial Chain-of-Thought VLM Verification by Ananth Sriram, Neel Mokaria, Rajveer Singh, [4pt] Department of Computer Science, University of Maryland, College Park, MD, USA {asriram2, nmokaria, rajveer}@umd.edu, [2pt] Authors listed in alphabetical order; all contributed equally. (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19869.txt.

Introduction

Sriram, Mokaria and Singh present a passive end-of-shift safety monitor for construction sites, the deadliest US industry sector. Video from body worn and wall mounted cameras passes three stages. YOLO11 covers protective equipment and hazard detection, SAM 3 covers segmentation and deduplication, and Qwen3-VL with a persona scaffolded adversarial protocol covers compliance checks. Persona backstories in the stage three prompt lift precision over single pass prompting, most on hallucination prone violations. The system maps violations to OSHA standards, scores ergonomic risk from pose keypoints, and issues reports.

Catalog record

CREATORAnanth Sriram, Neel Mokaria, Rajveer Singh, [4pt] Department of Computer Science, University of Maryland, College Park, MD, USA {asriram2, nmokaria, rajveer}@umd.edu, [2pt] Authors listed in alphabetical order; all contributed equally.
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42912
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19869

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Passive Construction Site Safety Monitoring via Persona-Scaffolded Adversarial Chain-of-Thought VLM Verification Ananth Sriram, Neel Mokaria, Rajveer Singh, [4pt] Department of Computer Science, University of Maryland, College Park, MD, USA {asriram2, nmokaria, rajveer}@umd.edu, [2pt] Authors listed in alphabetical order; all contributed equally. Abstract Construction remains the deadliest industry sector in the United States, with 1,055 fatal worker injuries recorded in 2023 at a rate of 12.9 deaths per 100,000 full-time equivalent workers. The majority of these incidents are preventable, yet existing monitoring approaches are either prohibitively expensive, require real-time human operators, or address only a narrow subset of detectable violations. This paper presents a passive, end-of-shift construction safety monitoring pipeline, processing video from two independent camera modalities — POV body-worn and fixed wall-mounted cameras — through a unified three-stage architecture: (1) a fine-tuned YOLO11 detector for primary PPE and hazard detection, (2) SAM 3 for segmentation refinement, worker deduplication, and PPE-to-worker association, and (3) Qwen3-VL-8B-Instruct with a novel three-pass adversarial chain-of-thought protocol for compliance verification and hallucination control. The principal methodological contribution is the Stage 3 prompt design: we find that the specific construction of professional persona backstories, following the method-actor prompt engineering framing, drives an observed 12% precision improvement on an informal three-author review of the 12-video Ironsite video corpus, with the largest gains on hallucination-prone categories. Structural message isolation enforces observational independence between a generator pass, a discriminator pass, and a reconciliation pass governed by asymmetric rules that encode explicit priors about the relative reliability of human observation versus automated detection for different violation categories. The system maps detected violations to specific OSHA standards, performs REBA-inspired ergonomic risk scoring from pose keypoints, and produces per-worker safety reports with timestamped frame-level evidence. A quantitative evaluation harness is released alongside the codebase for future reproduction. 1 Introduction The Bureau of Labor Statistics reported a fatal work injury rate of 12.9 deaths per 100,000 full-time equivalent workers in the U.S. construction sector in 2023, making it the industry with the most fatalities of any sector [ 1 ] . Men accounted for 91.5% of fatalities. Fall protection (OSHA Standard 1926.501) alone accounts for the single largest violation category cited by OSHA year over year [ 2 ] . The operational gap we target is the lack of an affordable, passive, continuous monitoring approach that can watch multiple workers across a construction site, resolve their identities over time, and produce an actionable report without requiring real-time human review. Existing solutions tend to be expensive, require dedicated real-time operators, or cover only a narrow slice of the problem—for instance, hard hat detection alone. We explore an end-of-shift processing paradigm as one possible response. Rather than streaming video in real time, the system processes footage after a shift completes, which makes it more practical to deploy on commodity hardware while using higher-capacity models (including an 8-billion-parameter vision-language model) and multi-pass verification that would be difficult to support under real-time latency constraints. The contributions of this paper are descriptive and architectural: 1. A unified three-stage pipeline (YOLO \rightarrow SAM 3 \rightarrow VLM) that handles both PPE and posture/ergonomic analysis from two independent camera modalities. 2. A method-prompted, persona-scaffolded three-pass adversarial chain-of-thought VLM protocol — following the method-actor framing of Doyle [ 11 ] — with structurally enforced independence between passes and an asymmetric reconciliation policy, which yields an observed 12% precision improvement over single-pass prompting on the development corpus. 3. REBA-inspired ergonomic risk scoring from YOLO pose keypoints with tuned confidence gates. 4. An end-of-shift backend that maps detected events to specific OSHA standards and produces per-worker safety reports with timestamped evidence. 2 Related Work 2.1 PPE Detection in Construction Environments Automated detection of personal protective equipment (PPE) using computer vision has been an active area of research since the widespread adoption of deep convolutional neural networks. Early work by Fang et al. [ 3 ] demonstrated that Faster R-CNN could detect hard hat compliance from far-field surveillance footage with sufficient accuracy for practical deployment, establishing the feasibility of passive vision-based monitoring in outdoor construction environments. Nath et al. [ 4 ] extended this line of work to multi-class PPE detection using YOLOv3, reporting mean average precision (mAP) values above 0.85 on construction-specific datasets. More recent studies have leveraged YOLOv5 and YOLOv8 architectures on larger composite datasets, consistently demonstrating that detection accuracy degrades under variable illumination, occlusion, and worker clustering [ 12 ] . A persistent limitation across this body of work is the restriction to a single PPE category or a single camera viewpoint, without integration of ergonomic risk assessment or persistent worker identity tracking across a shift. 2.2 Ergonomic Risk Assessment via Pose Estimation Traditional ergonomic risk assessment in construction relies on manual observation instruments, principally the Rapid Entire Body Assessment (REBA) [ 5 ] and Rapid Upper Limb Assessment (RULA), which require trained observers and are inherently episodic rather than continuous. Computer vision approaches to automating these assessments have drawn primarily on skeleton-based pose estimation. Yan et al. [ 6 ] proposed wearable IMU-based systems to detect musculoskeletal disorder (MSD) risk in real time, achieving high sensitivity for trunk flexion and arm overreach events. Subsequent work demonstrated that markerless pose estimation using convolutional neural network-based keypoint models can recover joint angles sufficient for ergonomic scoring from monocular video without instrumented garments [ 13 ] . However, these systems have been validated primarily in controlled laboratory settings; their performance on outdoor construction footage with partial occlusion and variable depth remains incompletely characterized. 2.3 Worker Re-identification and Identity Persistence Maintaining persistent worker identities across a full shift is a prerequisite for accumulating violation histories and generating per-worker compliance reports, yet it has received comparatively limited attention in the construction safety literature. Detection-based tracking systems operating in construction environments must contend with visual confounds introduced by high-visibility PPE — similar vest colours and helmet profiles reduce the discriminative power of appearance-based features [ 14 ] . BoT-SORT [ 15 ] and related tracking-by-detection algorithms address short-term occlusion effectively through coupled Kalman filter motion prediction and appearance re-identification, but are not designed for the multi-hour gaps characteristic of shift-level tracking. The present work adopts a lightweight identity database combining BoT-SORT for within-session tracking with appearance embedding similarity for re-association after disappearance events, representing a pragmatic intermediate between pure tracking and full Re-ID. 2.4 Vision-Language Models for Safety Assessment The application of large vision-language models (VLMs) to safety-critical inspection tasks is nascent but growing. Generalist VLMs have shown promising zero-shot performance on visual question answering tasks involving workplace safety scenes, yet their deployment in safety-critical contexts is complicated by hallucination — the generation of plausible but factually unsupported outputs [ 16 ] . Chain-of-thought prompting, while improving task accuracy on reasoning benchmarks, has been reported to induce systematic overconfidence in VLMs by causing token probabilities to reflect consistency with the model’s own reasoning trace rather than uncertainty about visual evidence [ 17 ] . Multi-agent and self-consistency prompting approaches [ 7 ] have demonstrated that soliciting multiple independent completions and aggregating by consistency can reduce hallucination rates, which motivates the multi-pass verification protocol described in this work. To the best of our knowledge, prior published work has not applied this style of multi-pass adversarial VLM verification specifically to

Read the complete text