Experimental Study on Surveillance Video-Based Indoor Occupancy Measurement for Occupant-Centric Control
Irfan Qaisar, Kailai Sun, Qingshan Jia, Qianchuan Zhao · 2026
Read Experimental Study on Surveillance Video-Based Indoor Occupancy Measurement for Occupant-Centric Control by Irfan Qaisar, Kailai Sun, Qingshan Jia, Qianchuan Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26081.txt.
Introduction
Qaisar and co-authors test ways to count people in rooms from surveillance video for smarter building control. Good occupancy data drives heating and cooling decisions in smart buildings. They compare three pipelines on real lab footage with hand-labeled ground truth. Plain detection is least stable over time. Tracking smooths the counts. Adding a language model step cuts false empty readings further. The best pair, YOLOv8 with DeepSeek, hits 0.8824 accuracy and an F1 of 0.9320. Plugged into a control model, it points to 17.94 percent HVAC energy savings.
Catalog record
| CREATOR | Irfan Qaisar, Kailai Sun, Qingshan Jia, Qianchuan Zhao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58280 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.26081
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Experimental Study on Surveillance Video-Based Indoor Occupancy Measurement for Occupant-Centric Control Irfan Qaisar, Kailai Sun, Qingshan Jia, Qianchuan Zhao Abstract Accurate occupancy information is essential for closed-loop occupant-centric control (OCC) in smart buildings. However, existing vision-based occupancy measurement methods often struggle to provide stable and accurate measurements in real indoor environments, and their implications for downstream HVAC control remain insufficiently studied. To achieve Net Zero emissions by 2050, this paper presents an experimental study of large language models (LLMs)-enhanced vision-based indoor occupancy measurement and its impact on OCC-enabled HVAC operation. Detection-only, tracking-based, and LLM-based refinement pipelines are compared under identical conditions using real surveillance data collected from a research laboratory in China, with frame-level manual ground-truth annotations. Results show that tracking-based methods improve temporal stability over detection-only measurement, while LLM-based refinement further improves occupancy measurement performance and reduces false unoccupied prediction. The best-performing pipeline, YOLOv8+DeepSeek, achieves an accuracy of 0.8824 and an F1-score of 0.9320. This pipeline is then integrated into an HVAC supervisory model predictive control framework in OpenStudio-EnergyPlus. Experimental results demonstrate that the proposed framework can support more efficient OCC operation, achieving a substantial HVAC energy-saving potential of 17.94%. These findings provide an effective methodology and practical foundation for future research in AI-enhanced smart building operations. [type=editor,style=chinese, orcid=0000-0002-4831-977X] [type=editor,style=chinese,orcid=0000-0003-1648-3409] [type=editor,style=chinese,orcid=0000-0002-4683-7215] [type=editor,style=chinese,orcid=0000-0002-7952-5621] keywords ,Smart Buildings ,Computer Vision ,Occupant-Centric Control ,Surveillance Video ,Large Language Model, † † corresponding: Corresponding authors. † † These authors contributed equally to this work. 1 INTRODUCTION Buildings represent a major source of global energy consumption and greenhouse gas emissions, accounting for a significant portion of global energy use 16 . Achieving the net-zero target by 2050 therefore requires substantial reductions in carbon emissions from the building sector 21 . Within this context, heating, ventilation, and air-conditioning (HVAC) systems typically constitute the dominant share of operational energy demand, with their contribution commonly reported to be in the range of 40–50% in commercial and institutional buildings 7 . As modern HVAC systems increasingly operate as closed-loop cyber-physical systems, reliable occupancy information has become an essential feedback input for enabling energy-efficient and demand-responsive control strategies 56 ; 19 . Beyond energy performance, occupancy data also plays a critical role in supporting indoor environmental quality (IEQ), ensuring human health, and enabling safety-related functions such as emergency management and situational awareness 22 . To achieve Net Zero emissions by 2050, occupant-centric control (OCC) aims to use real-time information on occupancy and indoor conditions to adjust HVAC and other building systems according to actual demand 34 . OCC has gained increasing attention because conventional schedule-based operation often ignores real occupancy patterns, which can increase energy use and reduce comfort 35 . IEA EBC Annex 66 emphasizes that occupant presence and behavior strongly influence building energy use and building performance 5 . IEA EBC Annex 79 highlights a broader shift from static schedule-based operation toward adaptive, demand-responsive strategies that explicitly account for occupant needs, behavior, and interactions with building systems 6 . Previous studies show that occupant-centric HVAC control can achieve substantial energy savings while maintaining acceptable indoor conditions and comfort, with reported savings of about 10-40% in building simulation 51 ; 37 . Among the inputs required by OCC, occupancy is one of the most important because it directly affects when and how building systems should operate 18 . If occupancy is estimated inaccurately, HVAC systems may condition vacant spaces or apply setback during actual occupancy 25 . This can reduce energy efficiency and thermal comfort, especially because occupants’ thermal states and adaptive behavioral responses are sensitive to temporal variations in indoor environmental conditions 38 . The issue is especially important for model predictive control (MPC), where future control actions depend on forecasts of system states and exogenous inputs such as occupancy 55 ; 39 . Therefore, accurate and temporally stable occupancy information is essential for effective OCC implementation in practice. List of Abbreviations CO 2 Carbon Dioxide DL Deep Learning DeepSORT Deep Simple Online and Realtime Tracking HVAC Heating, Ventilation, and Air Conditioning IEQ Indoor Environmental Quality LLM Large Language Model MAE Mean Absolute Error ML Machine Learning ID-switch Identity Switch MOT Multi-Object Tracking MPC Model Predictive Control OCC Occupant-Centric Control PIR Passive Infrared RMSE Root Mean Squared Error SORT Simple Online and Realtime Tracking VLM Vision Language Models YOLO You Only Look Once Many sensors (e.g., passive infrared (PIR) sensors, CO2 sensors, temperature sensors, humidity sensors, WiFi) have been explored for occupancy measurement and prediction in buildings 10 ; 47 . However, these methods often rely on indirect indicators and struggle to provide reliable occupant counts in real indoor environments, especially under dynamic and complex usage conditions 12 ; 41 . Occupancy prediction methods based on historical data have also been widely studied for building control applications 61 . Yet their performance can degrade when occupancy behavior becomes irregular, event-driven, or highly dynamic, which limits their reliability for real-time control 59 ; 27 . These limitations motivate the need for more robust and control-relevant occupancy measurement approaches. Vision-based occupancy measurement has attracted growing attention because cameras provide direct information on indoor occupant presence and count 13 ; 58 . Prior studies have explored scene-based counting, head detection, and fusion frameworks to improve estimation performance 54 ; 43 . However, robust deployment in real environments remains challenging due to occlusion, background clutter, head-like objects, limited camera coverage, and temporal instability 4 . To improve temporal consistency, tracking methods such as SORT, DeepSORT, ByteTrack and MOTIP are widely used to associate detections across frames 40 ; 8 ; 52 ; 60 ; 15 . While these methods can reduce short-term fluctuations 24 , their effectiveness remains constrained by detection errors under occlusion, crowding, and rapid motion, which may cause identity switches and fragmented trajectories 44 . Traditional vision-based methods still have limited performance in complex indoor environments, especially when detection and tracking outputs are unstable over time 4 ; 44 . Recent large language models (LLMs) and vision-language models (VLMs) have shown strong reasoning abilities 45 , which can support higher-level consistency checking and contextual inference beyond conventional perception pipelines 1 ; 29 ; 31 . This suggests that LLMs may help improve traditional occupancy measurement methods by refining noisy perceptual outputs. However, such reasoning-based refinement has been only rarely explored for indoor occupancy measurement in buildings and remains largely under-studied in the building energy and HVAC control literature 17 . In addition, most existing studies focus on improving occupancy accuracy, while relatively little attention has been paid to how occupancy quality affects downstream building control performance 14 . More importantly, the energy-saving potential of such approaches still needs to be verified in experimental practice 28 . To bridge these gaps, this study conducts an experimental investigation of surveillance-video-based indoor occupancy measurement for occupant-centric HVAC operation. Representative perception pipelines, including detection-only, tracking-based, and LLM-enhanced approaches, are evaluated under identical conditions using a real indoor surveillance dataset with frame-level manual ground-truth annotations. Their performance is assessed in terms of accuracy and temporal stability. In addition, the best-performing pipeline is integrated into an MPC-based HVAC control framework in OpenStudio and EnergyPlus to examine its impact on building energy use and occupant comfort. The main contributions of this study are as follows: • An