Enhancing Fault Diagnosis in GWAC: A Monitoring System for Telescope Arrays

Y. Xu, G. W. Li, J. Wang, L. P. Xin, H. B. Cai, X. H. Han, X. M. Lu, L. Huang, J. Y. Wei · 2025

Read Enhancing Fault Diagnosis in GWAC: A Monitoring System for Telescope Arrays by Y. Xu, G. W. Li, J. Wang, L. P. Xin, H. B. Cai, X. H. Han, X. M. Lu, L. Huang, J. Y. Wei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12562.txt.

Introduction

Xu and colleagues described in 2025 a monitoring system built to cut fault-finding time for a large Chinese telescope array. The array links over a hundred devices, a hundred servers, and thousands of software modules in a three-second imaging cycle. Complexity drove failure rates so high that observation efficiency fell below half in 2023.

Their system adds two novel views: one tracking how states evolve, another following short-lived processes through their lifecycle. Combined with live state and key-parameter views, these give operators a complete picture. After a year of use, faults are located within minutes, roughly ten times faster than before. The authors note the design should transfer to other telescope arrays with similar complexity.

Catalog record

CREATORY. Xu, G. W. Li, J. Wang, L. P. Xin, H. B. Cai, X. H. Han, X. M. Lu, L. Huang, J. Y. Wei
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-12562

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Fault Diagnosis in GWAC: A Monitoring System for Telescope Arrays Y. Xu, G. W. Li, J. Wang, L. P. Xin, H. B. Cai, X. H. Han, X. M. Lu, L. Huang, J. Y. Wei Abstract The Ground-based Wide-Angle Cameras array (GWAC) necessitates the integration of over 100 hardware devices, more than 100 servers, and upwards of 2500 software modules, all synchronized within a 3-second imaging cycle. However, the complexity of real-time and high concurrency processing of big data have historically resulted in a substantial failure rate, with estimated observation efficiency of less than 50% in 2023. To address these challenges, we developed a monitoring system aimed at enhancing fault diagnosis efficiency. The system features two innovative monitoring views: state evolution monitoring and transient lifecycle monitoring. These, combined with instantaneous state monitoring and key parameter monitoring views, create a comprehensive and holistic monitoring strategy. This paper details the system’s architecture, data collection methods, and the design philosophy of monitoring views. After a year of practical fault diagnostics, the system has demonstrated the ability to identify and localize faults within minutes, achieving fault localization speeds nearly ten times faster than traditional methods. Additionally, the system’s design exhibits high generalizability, making them applicable to other telescope array systems. I Introduction The Ground-based Wide-Angle Cameras array (GWAC), one of the main ground-based facilities of the Chinese-French SVOM mission ( Wei et al. 2016 ) , is to construct a fully automated, self-triggering, and self-follow-up transient survey system, and has completed the construction of the real-time autonomous detection for transients. GWAC has reported a series of scientific results, such as the independent detection of one high-energy burst, GRB 201223A ( Xin et al. 2023 ) , with a time scale of only 29 seconds; the discovery of more than 200 white-light flares ( Li et al. 2024 ; Li et al. 2023 ) ; the discovery of two cold star superflares with amplitudes reaching around 10th magnitude ( Xin et al. 2024 ; Xin et al. 2021 ) ; and long-term activity research on cold star flares ( Li et al. 2023 ) . The GWAC consists of 10 mounts, 50 image sensors, 50 camera focusing decices, and over 100 supporting servers. The data processing pipeline covers multiple subsystems such as observation scheduling ( Han et al. 2021 ) , observation control, automatic focusing [Huang, 2016], automatic guiding, real-time scientific data processing, automatic follow-up ( Xu et al. 2020 ) , and scientific result management ( Xu et al. 2021 ) . The daily operations of the GWAC rely on the coordination of various software and hardware modules, involving complex processes that result in a relatively high fault rate. The two primary challenges are as follows: 1. Hardware-Software Dependency: The hardware operations depend heavily on software feedback. For example, the mount’s pointing correction relies on astrometric results from observed images, and maintaining image quality depends on image assessment results. These results are affected by varying weather conditions, introducing uncertainty in astrometric and image assessments. Since scientific data processing also depends on these feedback results, the overall feedback loop is long and lacks reliability. 2. Complex Software Architecture: The software architecture is intricate, requiring high concurrency and real-time processing. The data processing pipeline must address various tasks such as observation planning, image quality assessment, template catalog generation, hardware feedback, cross-matching with multiple catalogs, and automated follow-up of transient. Each image processed requires over 50 software modules, and with all cameras operating simultaneously, more than 2,500 module invocations occur per image cycle. The interdependencies between these modules mean that a single failure can disrupt the entire process, reducing system robustness. When the number of telescopes is small, daily malfunctions are less frequent and have a minimal impact on observation efficiency. However, as the telescope number increases, malfunctions become inevitable, often surpassing the diagnostic and repair capacity of the maintenance staff. This results in prolonged operation with faults and a significant drop in system efficiency. Therefore, GWAC urgently needs an efficient monitoring system to continuously track the system’s status, provide fault diagnosis alerts, and guide maintenance staff in quickly resolving issues, ultimately enhancing operational efficiency. Current monitoring solutions are inadequate for GWAC’s needs due to its complexity, high concurrency, and real-time requirements. For instance, the Cherenkov Telescope Array monitoring system ( Costa et al. 2022 ) primarily tracks hardware degradation to prevent major failures. LAMOST ( Hu et al. 2021 ) focuses on real-time assessments of guiding system performance, focal surface defocus, submirror performance, and active optics system performance. Similarly, radio astronomy projects like SKA ( Di Carlo et al. 2016 ) deploy monitoring software mainly for hardware health status. These systems emphasize hardware health or observation efficiency but are insufficient for GWAC. GWAC, being a complex system integrating high-concurrency data processing and precision mechanical control, requires a monitoring system that offers real-time hardware status tracking and comprehensive software pipeline monitoring. The GWAC monitoring system oversees the full running status of all telescopes and presents monitoring data through diverse views. To ensure comprehensive fault coverage, the system employs various data collection methods, including hardware status, real-time pipeline status, and key image parameters. During the collaboration of hardware and software, the system generates massive amounts of raw monitoring data, surpassing manual analysis capabilities. By integrating and abstracting the raw data, the system provides multidimensional monitoring views that simplify data interpretation and reduce reliance on the experience and skills of maintenance personnel. Additionally, it enables deeper insights into internal operations and evolutionary processes, supporting manual fault diagnosis and laying the groundwork for future automated fault detection. This paper introduces a monitoring system for GWAC, which designs a diversified monitoring data collection and visualization scheme. The structure of the paper is as follows: Section II describes the system architecture and its relationship with existing GWAC pipelines; Section 3 details the design and implementation of the system, including the construction of monitoring views, database design, and system implementation; Section 4 presents fault diagnosis cases based on the monitoring views; and the final section summarizes and provides an outlook for the proposed monitoring system. II System Architecture To enhance the efficiency of fault detection and diagnosis in the GWAC system, we developed a monitoring system integrated into the existing GWAC pipeline. As illustrated in Figure 1 , the monitoring system comprises two main components: data collection and monitoring views. Figure 1: The architecture of GWAC monitoring system. The system consists of two main components: data collection and monitoring views. Data collection is integrated into the existing GWAC pipeline to collect data and generate four monitoring views. II.1 Data Collection The data collection component, indicated by the gray dashed box in Figure 1 , aggregates data from seven subsystems in the GWAC data processing pipeline: the Observation Scheduling System, Observation Control System, Automatic Guiding System, Automatic Focusing System, Image Processing System, Automatic Follow-up System, and Scientific Results Management System. The collected data falls into three categories: key module invocation times, image and instrument parameters, and transient processing information, represented by the orange, green, and red arrows, respectively. The presence of these colored arrows next to a subsystem’s box in the figure indicates the type of data collected for that subsystem. For instance, the

Read the complete text