Technical challenges designing a prototype common readout board for LHCb future upgrades
Julien Jiro Langouët on behalf of the PCIe400 development collaboration · 2024
Read Technical challenges designing a prototype common readout board for LHCb future upgrades by Julien Jiro Langouët on behalf of the PCIe400 development collaboration (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01648.txt.
Description
Abstract The LHCb Upgrade I implemented a triggerless data acquisition system, employing a versatile readout board performing data protocol conversion and first stage of event building. It is a key component of the chain as the same hardware is used for readout of all sub-detectors, fast command and LHC clock distribution to front-end electronics.
Catalog record
| CREATOR | Julien Jiro Langouët on behalf of the PCIe400 development collaboration |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-01648
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Technical challenges designing a prototype common readout board for LHCb future upgrades Julien Jiro Langouët on behalf of the PCIe400 development collaboration Abstract The LHCb Upgrade I implemented a triggerless data acquisition system, employing a versatile readout board performing data protocol conversion and first stage of event building. It is a key component of the chain as the same hardware is used for readout of all sub-detectors, fast command and LHC clock distribution to front-end electronics. LHCb Upgrade II will increase the global throughput by five times. Also, the transition to 5D techniques, exemplified by the FastRICH detector, requires phase deterministic clock distribution to front-end electronics. The PCIe400 development with an output bandwidth of 400 Gbit / s \mathrm{Gbit}\text{/}\mathrm{s} over PCIe Gen 5 or 400GbE, and up to 48 bidirectional links at 25 Gbit / s \mathrm{Gbit}\text{/}\mathrm{s} , significantly outperforms current systems. Its FPGA embeds 4 million logic elements and 32 GB \mathrm{GB} of memory, potentially providing a twelvefold increase in processing capability compared to existing PCIe40 board developed for LHCb Upgrade I. Latest available PLL with intrinsic jitter <100 fs \mathrm{fs} pk-pk in combination with logic to measure and control clock phases that should enable reaching ultimate precision at 𝒪 ( 10 ) ps \mathcal{O}(10)~$\mathrm{ps}$ . The design process went through meticulous studies on three major topics : Thermal dissipation, Power distribution and Signal Integrity. An early power consumption resulted with a total power dissipated up to 220 W \mathrm{W} which is more than 3 times what the previous generation board had been designed for. The study compares solid and heat-pipe heat-sink base in Computational Fluid Dynamics simulations to define the parametric limits of an air cooled solution targeting our application. PCB thickness limits imposed by the PCIe form factor also puts high constraints over the stack-up choice. Indeed, power estimation showed power rails sinking up to 100 A \mathrm{A} . Furthermore, 18 of the 22 power rails on-board are dedicated to the FPGA while power regulators can not be placed optimally all around the FPGA. A systematic verification of voltage drop in all power planes allowed to fit a solution matching tolerances on all FPGA power supplies. In the end, a controlled-depth milled edge connector PCB unlocked the solution to satisfy power distribution integrity. Finally, 112 Gbit / s \mathrm{Gbit}\text{/}\mathrm{s} PAM4 striplines have 50 GHz \mathrm{GHz} bandwidth. Designing such striplines requires peculiar attention not to create impedance mismatch. S-parameter extraction using classic 2D simulator showed different results in terms of attenuation compared to new 3D simulators considering full geometry of vias. The counterpart of 3D simulators being the processing times. We study on how to reduce simulation time from 10 days to 10 hours. This work allowed to run comparison between various geometries focusing on breakout zones. 1 Module architecture and features The LHCb Upgrade I is designed to run at an instantaneous luminosity of 2\times 10^{33}cm^{-2}s^{-1} , achieved with a bunch-crossing rate of up to 40 MHz and a mean number of proton-proton interactions per bunch crossing of six, producing a data throughput of 5 \mathrm{TB}\text{/}\mathrm{s} . With the HL-LHC upgrade, the detector will operate at a maximum instantaneous luminosity of 1.5\times 10^{34}cm^{-2}s^{-1} , producing \sim 40 proton-proton interactions per bunch crossing with a data throughput of 25 \mathrm{TB}\text{/}\mathrm{s} . [ 1 ] The LHCb data acquisition system foreseen for the Upgrade II is similar to the one developed for the Upgrade I. The yield of event useful for physics is maximized by using a fully synchronous readout of each bunch crossing. This architecture is called triggerless because there is no more trigger decision sent to the front-end electronics. Instead, the data acquisition system consist of a single stage detector readout, followed by event-building on local area network and by real time reconstruction and selections of events. The system relies on a single custom made readout board, together with Commercial Off the Shelf (COTS) data-center hardware. [ 2 ] The prototype readout board developed, called PCIe400, is at the interface between the front-end electronics on detector using radiation hardened protocols and back-end electronics using commercial data-center protocols. The same hardware, with different gatewares, is used for readout of all sub-detectors, fast command and LHC clock distribution to front-end electronics. The board includes Altera’s flagship FPGA, Agilex 7 M-series, with 4 million logic elements running at up to 1 \mathrm{GHz} , and embeds 32 \mathrm{GB} of high bandwidth memory. [ 3 ] It features up to 48 bidirectional links for custom protocols between 1 and 26 \mathrm{Gbit}\text{/}\mathrm{s} each. It is connected to a PCIe Gen 5 x16 bus, giving it an effective bandwidth > 400 \mathrm{Gbit}\text{/}\mathrm{s} with the host server CPU. It also implements an optional 400G Ethernet bus over a QSFP112 form factor. Finally, it manages various protocols associated with time distribution, such as TTC-PON 1 1 1 Timing, Trigger and Control Passive Optical Network and White Rabbit. [ 4 ] [ 5 ] A synoptic and a 3D layout view of the board is given in figure 1 . Routing such board represents a challenge due to high density of components. In fact, there are 2500 components with 10\,000 connections on a 18 layers 268\times 100~$\mathrm{mm}$ PCB. In particular, the FPGA ball grid array is 4500 pins over a 56\times 66~$\mathrm{mm}$ package area which represents a 25% increase in density compared to previous generation board using Altera’s Arria 10 FPGA family thanks to a hexagonal ball grid array. Figure 1 : PCIe400 Synoptic and 3D layout view. 2 Thermal dissipation Using Altera Power and Thermal Calculator and scaled resource usage on new FPGA compared to current LHCb readout system, we define a typical 2 2 2 typical case defined as 12.5% toggle rate, 640 \mathrm{M}\mathrm{H}\mathrm{z} , 60% logic, 80% RAM, 48 links @10 \mathrm{Gbit}\text{/}\mathrm{s} and worst case 3 3 3 worst case defined as 12.5% toggle rate, 640 \mathrm{M}\mathrm{H}\mathrm{z} , 80% logic, 100% RAM, 40 links @25 \mathrm{Gbit}\text{/}\mathrm{s} + 400GbE . [ 6 ] The dissipated power estimated is between 120 to 230 \mathrm{W} varying with the junction temperature due to leakage current, as show in figure 2 . It is therefore important not to under-design the heat-sink, while an air cooling solution has been privileged for its simplicity on infrastructure. Figure 2 : FPGA power dissipation estimated for typical and worst case vs junction temperature. Figure 3 : FPGA heat-sink thermal resistance vs airflow comparing standard and heat-pipe heat-sink. Considering placement constraint on board and airflow direction along the length axis of the board, standard aluminum are compared to heat-pipe enhanced heat-sink in Computational Fluid Dynamic (CFD) simulation. As shown on figure 2 , with ambient temperature of 38~$\mathrm{\SIUnitSymbolCelsius}$ , airflow of 5~$\mathrm{m}\text{/}\mathrm{s}$ , the overall thermal resistance of the heat-sink is lowered by 20% thanks to reduced spreading resistance across the heat-sink base. In this condition, heat-sink can dissipate 135 on a standard and 165 \mathrm{W} on a heat pipe heat-sink, and maintain FPGA junction temperature at 85 \mathrm{\SIUnitSymbolCelsius} . On the side of the FPGA, the on-board opto-electronic transceiver (OBT) consume 30 \mathrm{W} over 4 modules. The placement constraint results in a heat cumulative effect which standard heat-sink provided by manufacturer can not handle. As shown in figure 4 , module