Test Management and Coordination During the Vera C. Rubin Observatory Commissioning and Early Operations Using Zephyr Scale

Bruno Quint, Tiago Ribeiro, Erik Dennihy, Brian Stalder, David Sanmartim, Keith Bechtol · 2026

Read Test Management and Coordination During the Vera C. Rubin Observatory Commissioning and Early Operations Using Zephyr Scale by Bruno Quint, Tiago Ribeiro, Erik Dennihy, Brian Stalder, David Sanmartim, Keith Bechtol (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31795.txt.

Introduction

Quint and five colleagues describe how the Vera C. Rubin Observatory managed hundreds of commissioning tests with a tool borrowed from software testing. Commissioning meant planning and running integration and on-sky tests across subsystems and teams spread over several countries. The observatory adopted Zephyr Scale, a test manager that lives inside Jira.

Each test becomes a Test Case holding the script needed to run it at the summit. Every day, cases are grouped into a Test Cycle that forms that day and night's plan. Complex tests are written as partly automated JSON files that the Scheduler, the observatory's real-time observing engine, can ingest directly. Benefits and limits of stretching a software tool to observatory scale are weighed.

Catalog record

CREATORBruno Quint, Tiago Ribeiro, Erik Dennihy, Brian Stalder, David Sanmartim, Keith Bechtol
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.31795

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Test Management and Coordination During the Vera C. Rubin Observatory Commissioning and Early Operations Using Zephyr Scale Bruno Quint, Tiago Ribeiro, Erik Dennihy, Brian Stalder, David Sanmartim, Keith Bechtol Abstract The commissioning of the NSF–DOE Vera C. Rubin Observatory required coordinating the planning, design, and execution of hundreds of integration and on-sky tests involving different subsystems and geographically distributed teams. To support this task, we adopted a Jira-native test management tool, Zephyr Scale. The initial use of Zephyr Scale focused solely on system verification and validation. Its use was rescoped to coordinate higher-level tests, and it is still in use in early operations. Zephyr Scale allows the creation of Test Cases, which represent individual tests. Each Test Case contains the information needed to execute a test at the summit. This includes a step-by-step script. Every day, Test Cases are grouped into a Test Cycle, which represents the test plan for all tests to be executed that day and that same night. We describe the defined workflow for test creation, review, and deployment, which bridges the gap between ideation and on-sky execution within a Test Cycle. We also outline how we write more complex tests as partially automated JSON files consumed by the Scheduler—the system’s real-time, constraint-aware observation optimization engine. This integration enables the Scheduler to ingest high-level observing scripts that communicate with subsystems via an abstraction layer to execute common observatory operations, such as slewing, tracking, and data acquisition. Finally, we summarize the benefits and limitations of using Zephyr Scale, designed initially to coordinate software testing, for large-scale observatory commissioning and early operations. a]NSF–DOE Vera C. Rubin Observatory/NSF NOIRLab, Casilla 603, La Serena, Chile b]NSF–DOE Vera C. Rubin Observatory/NSF NOIRLab, 950 N. Cherry Ave., Tucson, AZ 85719, USA c]Department of Physics, University of Wisconsin-Madison, Madison, WI 53706, USA keywords Zephyr Scale, Jira, Workflows, Operations, Coordination † † authorinfo: Further author information: (Send correspondence to B.Q.) B.C: E-mail: [email protected], Telephone: +56 51 2205 200 1 INTRODUCTION The NSF–DOE Vera C. Rubin Observatory formally completed its construction phase, together with the last part of Systems Integration, Testing and Commissioning late in 2025. The observatory operates two telescopes at Cerro Pachón, in Chile: the 8.4 m Simonyi Survey Telescope, which carries the LSST Camera (LSSTCam) [ 8 ] , and the 1.2 m Auxiliary Telescope (AuxTel), used for atmospheric characterization. Before LSSTCam was installed, the Simonyi Survey Telescope was commissioned on sky with the LSST Commissioning Camera (LSSTComCam) [ 7 ] . LSSTComCam should not be confused with LSSTCam: the two names differ only by the abbreviation for “commissioning”, so we spell them out in full throughout this paper to keep them distinct. This was a fast-paced phase of the observatory’s history with daily changes in procedures and guidelines. During peak commissioning activities, there were more than fifty individuals with different roles and expertise involved in designing, running, or analyzing the results of hundreds of tests. Now the observatory is in the early operations stage and we expect to replace the constant changes with a steady routine. However, even in this new era, coordinating tasks at this scale still requires the information to propagate to more than thirty people in different teams, and quickly. Some of the teams involved in the daily coordination are the Observing Specialists, the Rubin Summit Scientists, and the In-Kind Contributors. We refer to these groups collectively as the Observing Team. The Observing Specialists are the main drivers of the observatory and execute the individual tasks that are part of each night’s plan. Rubin Summit Scientists provide technical and scientific support to implement tests and plan the nightly operations. In-Kind Contributors bring expertise on higher-level performance and requirements. Together, Rubin Summit Scientists and In-Kind Contributors design new tests and perform analyses on image quality, stray light, and thermal and environmental performance. To support coordination at this scale, we adopted Zephyr Scale 1 1 1 https://smartbear.com/test-management/zephyr-scale/ , a Jira-native tool originally designed for software testing. During early stages of commissioning, we re-scoped this tool for verification and validation tests. It proved to be useful for organizing and tracking tests, and for executing them step-by-step. Later, it became the primary tool for tests, procedures, and nightly plans. Zephyr Scale was critical for communication and coordination during commissioning and we still use it during the early stages of operations. In Section 2 , we describe what tools, including Zephyr Scale, featured in the daily life during commissioning and early operations and how they worked together. In Section 3 , we describe the process used to convert any idea into a formal procedure, how the nightly plans converged, and the workflow on a weekly basis. Finally, we close out this work with a summary of the benefits and limitations of Zephyr Scale when used to coordinate tests during commissioning and early operations. 2 Operational Framework Coordinating and running activities at the summit involves two main aspects: the technical aspect, namely how the Observing Team interacts with the observatory; and the operational reality, namely that no two days are quite alike. We address the technical aspect first. On the technical side, we need to be able to issue low-level commands to control individual components, and run high-level scripts for more complex operations such as sending the telescope to track a target on sky and take images. Finally, we need to be able to tell the observatory to start a survey and it should run by itself. The operational reality is that most days differ from each other. Ideally speaking, we want to have a steady routine. But this is not what happens. We have technical issues. Optimization tests. Dedicated investigations. And we need to decide what happens when. On top of that, we need to ensure that any relevant information about each of these activities is properly passed to the Observing Specialists, who are the final consumers of procedures and documentation. They need to have the tools to record the actual results of tests so we can find these results later and perform analysis. This section focuses on the technical side: the Observatory Control System and the two mechanisms we use to drive more complex tests, namely JSON BLOCKs and Scheduler Configurations. The coordination side — how we plan, document, and track tests with Zephyr Scale, and how it works together with other tools such as Slack, Jira, and Confluence — is covered in Section 3 . 2.1 Vera C. Rubin Observatory Control System The Observatory Control System is a Python-based service that relies on a Service Abstraction Layer (SAL) [ 4 ] , which uses Kafka as the main communication protocol [ 6 ] . Each relevant hardware component in the observatory is represented as a Commandable SAL Component (CSC) [ 3 ] which works as a link between its low-level controller and the rest of the system. Users interact with CSCs through two main interfaces. The simplest are SAL Commands, which target a single CSC directly. For more complex tasks, we rely on SAL Scripts — special Python scripts that coordinate multiple CSCs, include status checks, and handle error conditions. Both SAL Commands and SAL Scripts are dispatched through the ScriptQueue , a dedicated CSC, using the LSST Observatory Visual Environment (LOVE)

Read the complete text