submitted to Frontiers Efficient High Performance Computing with the ALICE Event Processing Nodes GPU-based farm Federico Ronchetti ∗1,5 , Valentina Akishina 2 , Edvard Andreassen 1 , Nora Bluhme 2 , Gautam Dange 2 , Jan de Cuveland 2 , Giada Erba 1 , Hari Gaur 2 , Dirk Hutter 2 , Grigory Kozlov 4 ,

2024

Read submitted to Frontiers Efficient High Performance Computing with the ALICE Event Processing Nodes GPU-based farm Federico Ronchetti ∗1,5 , Valentina Akishina 2 , Edvard Andreassen 1 , Nora Bluhme 2 , Gautam Dange 2 , Jan de Cuveland 2 , Giada Erba 1 , Hari Gaur 2 , Dirk Hutter 2 , Grigory Kozlov 4 , (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13755.txt.

Introduction

The authors described a GPU-based computing farm in a 2024 paper for the ALICE experiment at CERN. The experiment studies collisions of atomic nuclei and faces a flood of data in the latest runs of its collider. A single software frame now handles both live processing and later reprocessing of that data.

The farm, built from nodes packed with graphics cards, does the live rebuild of collision events. Graphics cards do this work faster per watt than standard chips, which keeps energy costs down. The paper reports how the farm was built and how it performs under real loads. Such setups guide other big science projects facing data floods.

Catalog record

YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-13755

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

submitted to Frontiers Efficient High Performance Computing with the ALICE Event Processing Nodes GPU-based farm Federico Ronchetti ∗1,5 , Valentina Akishina 2 , Edvard Andreassen 1 , Nora Bluhme 2 , Gautam Dange 2 , Jan de Cuveland 2 , Giada Erba 1 , Hari Gaur 2 , Dirk Hutter 2 , Grigory Kozlov 4 , Luboš Krčál 1 , Sarah La Pointe 2 , Johannes Lehrbach 2 , Volker Lindenstruth 2,3,4 , Gvozden Neskovic 2 , Andreas Redelbach 2 , David Rohr 1 , Felix Weiglhofer 2 , Alexander Wilhelmi 2 1 European Organization for Nuclear Research (CERN), Geneva, Switzerland 2 Frankfurt Institute for Adv5anced Studies, Frankfurt am Main, Germany 3 Goethe University, Frankfurt am Main, Germany 4 GSI Helmholtz Centre, Darmstadt, Germany 5 INFN, Laboratori Nazionali di Frascati, Frascati, Italy ∗ Corresponing author Abstract Due to the increase of data volumes expected for the LHC Run 3 and Run 4, the ALICE Collaboration designed and deployed a new, energy efficient, computing model to run Online and Offline (O 2 ) data processing within a single software framework. The ALICE O 2 Event Processing Nodes (EPN) project performs online data reconstruction using GPUs (Graphic Processing Units) instead of CPUs and applies an efficient, entropy-based, online data compression to cope with Pb–Pb collision data at a 50 kHz hadronic interaction rate. Also, the O 2 EPN farm infrastructure features an energy efficient, environmentally friendly, adiabatic cooling system which allows for operational and capital cost savings. Keywords: scientific computing, sustainable computing, HTC, HPC, GPU Abstract 1 Introduction The Large Hadron Collider (LHC) accelerator at CERN returned to full operation on July 5 th , 2022 when proton–proton collisions occurred at a record center-of-mass energy of 13.6 TeV and data taking activities resumed. During the LHC shutdown (2019–2021), the ALICE detector [ 1 ] underwent a substantial upgrade [ 2 ] providing improved track reconstruction, and an increased interaction rate of up to 50 kHz for Pb–Pb collisions in continuous readout mode. These improvements enabled recording a data sample a factor of 10 larger than the combined Run 1 and Run 2 samples. 2 The upgraded ALICE detector The ALICE detector comprises a central barrel (by far the largest data producer in the system) and a forward muon arm. The central barrel relies mainly on four sub-detectors for particle tracking: 1. the new Inner Tracking System (ITS) which is a 7-layer, 12.5 gigapixel monolithic silicon tracker 2. the upgraded Time Projection Chamber (TPC) with GEM-based readout for continuous operation 3. the Transition Radiation Detector (TRD) 4. the Time-Of-Flight (TOF) The muon arm is composed of three tracking devices: 1. a newly installed Muon Forward Tracker (a silicon tracker based on the same monolithic active pixel sensors used for the new ITS) 2. a revamped Muon Chambers system 3. a Muon Identifier (previously a trigger detector adapted to run in continuous readout). 2.1 The O 2 EPN project The major ALICE hardware upgrades for Run 3 necessitated the development and implementation of a completely new computing model: the O 2 project. This project unifies online (synchronous) and offline (asynchronous) data processing into a single software framework, enabling the same code-base to be executed in both contexts with appropriate selections and parameters. Naturally, the O 2 computing model necessitated a comprehensive overhaul and significant scaling up of the experiment’s computing farms to accommodate the demands of data readout and processing. To address these challenges, the ALICE EPN farm consists of 350 high-performance servers, each equipped with four boards for a total of 2800 GPUs. With compressed data rates already reaching 1–2 PB/day, the dramatic increase in data volumes compared to Run 2 made storing raw data entirely impractical, driving the need for efficient online compression and the adoption of GPUs in place of CPUs to accelerate reconstruction tasks. GPUs, with their superior compute throughput enabled by intrinsic parallelism, offer both cost and energy savings compared to CPU-based solutions. Without GPUs, approximately eight times as many CPU-only servers of the same type, along with additional resources, would be required for the online processing of TPC data from Pb–Pb collisions at a 50 kHz interaction rate (corresponding to an instantaneous LHC luminosity of 6\times 10^{27} cm -2 s -1 ) [ 3 ] . 3 ALICE Computing for Run 3 and beyond The ALICE Collaboration has been a pioneer in leveraging GPUs for data compression and online processing since 2010, starting with the High-Level Trigger (HLT) computing farm [ 4 ] . The HLT had direct access to detector readout hardware and raw data, playing a critical role in Run 2 by enabling data compression for heavy-ion collisions. Its software framework was already advanced enough to perform online data reconstruction using GPUs. The operational experience gained with the HLT farm during Runs 1 and Run 2 was instrumental in shaping the design and development of the current O 2 software and hardware systems. 3.1 Computing infrastructure Figure 1 illustrates the ALICE data readout and processing workflow in Run 3. The detectors’ front-end electronics boards are connected to custom Field-Programmable Gate Array (FPGA) boards, capable of continuous readout and Zero Suppression (ZS), which are hosted on the First Level Processor (FLP) farm nodes located near the experimental cavern. The connecting links are based on the GBT (GigaBit Transceiver) architecture, a versatile and high-performance communication framework developed by CERN for use in high-energy physics experiments. GBT is designed to handle the stringent data transmission requirements of particle detectors, providing robust, low-latency, and high-throughput (4.8 Gb/s) links between the detector front-end and readout electronics. The TPC outputs a prohibitive raw data rate of 3.3 TB/s at the front end, so ZS in the FPGA is crucial for data rate reduction, providing manageable rates, especially for high interaction rate Pb—Pb collisions. The FLP and EPN farms handle the high data throughput using an InfiniBand (IB) network fabric and a custom software suite (detailed later in the text). The compressed, reconstructed data produced by online processing are transferred to the central CERN Information Technology (IT) data center, located a few kilometers from the ALICE experimental site, via dedicated links employing standard TCP/IP over Ethernet. A seamless and balanced exchange between the EPN farm’s IB fabric and CERN IT is facilitated by four IB-to-Ethernet gateways positioned near the EPN farm’s five core 200 Gb/s IB managed switches. The CERN data center provides approximately 150 PB of storage, managed by sophisticated storage policies (see Section 4.1 ) to ensure efficient use of buffer space during different phases of experiment data taking. 3.2 The ALICE O 2 EPN farm The EPN farm data center was built on the surface at the LHC Point 2 experimental site, was designed using a modular approach (hence it is located farther from the ALICE cavern than the FLP farm). The concept was implemented using IT containers so that the data center could easily scale to meet any potential need for additional IT equipment. The ALICE data center comprises four containers designed to house IT equipment. Currently, three of these containers are used to host the EPN farm worker nodes, infrastructure nodes, and network equipment, while the fourth serves as a utility container for the time being. The layout

Read the complete text