Aspen Open Jets: Unlocking LHC Data for Foundation Models in Particle Physics

Oz Amram, Luca Anzalone, Joschka Birk, Darius A. Faroughy, Anna Hallin, Gregor Kasieczka, Michael Krämer, Ian Pang, Humberto Reyes-Gonzalez, David Shih · 2024

Read Aspen Open Jets: Unlocking LHC Data for Foundation Models in Particle Physics by Oz Amram, Luca Anzalone, Joschka Birk, Darius A. Faroughy, Anna Hallin, Gregor Kasieczka, Michael Krämer, Ian Pang, Humberto Reyes-Gonzalez, David Shih (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10504.txt.

Introduction

Oz Amram and co-authors introduce, in 2024, a large public dataset of particle jets from the Large Hadron Collider. The set holds about one hundred seventy-eight million high-energy jets drawn from open data recorded by the CMS experiment in 2016. It is prepared so machine learning models can train on it directly.

The team uses the data to pre-train a foundation model for particle physics. After this training, the model did better on tasks quite different from its training data, such as making jets of other kinds. The dataset is released for public use, to help groups build shared models from real collision data.

Catalog record

CREATOROz Amram, Luca Anzalone, Joschka Birk, Darius A. Faroughy, Anna Hallin, Gregor Kasieczka, Michael Krämer, Ian Pang, Humberto Reyes-Gonzalez, David Shih
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-10504

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Aspen Open Jets: Unlocking LHC Data for Foundation Models in Particle Physics Oz Amram, Luca Anzalone, Joschka Birk, Darius A. Faroughy, Anna Hallin, Gregor Kasieczka, Michael Krämer, Ian Pang, Humberto Reyes-Gonzalez, David Shih Abstract Foundation models are deep learning models pre-trained on large amounts of data which are capable of generalizing to multiple datasets and/or downstream tasks. This work demonstrates how data collected by the CMS experiment at the Large Hadron Collider can be useful in pre-training foundation models for HEP. Specifically, we introduce the AspenOpenJets dataset, consisting of approximately 178M high p T p_{T} jets derived from CMS 2016 Open Data. We show how pre-training the OmniJet - α \alpha foundation model on AspenOpenJets improves performance on generative tasks with significant domain shift: generating boosted top and QCD jets from the simulated JetClass dataset. In addition to demonstrating the power of pre-training of a jet-based foundation model on actual proton-proton collision data, we provide the ML-ready derived AspenOpenJets dataset for further public use. I Introduction While particle physics has long used machine learning techniques and is leading the way in adopting modern deep learning methods, the development and application of powerful foundation models – which have transformed natural language processing and computer vision – is still in its early stages. For the purposes of this work, a foundation model will be a model that has been pre-trained on a large amount of data for a specific task, and that can be used for different tasks downstream [ 11 ] . The pre-training thus serves as a foundation, upon which the downstream tasks can be built. Well-known foundation models include large language models (LLMs) such as GPT-3 [ 12 ] , BERT [ 25 ] and LLaMA [ 60 ] , image models such as Imagen [ 53 ] , and mixed-modality models such as CLIP [ 51 ] and ALIGN [ 34 ] . There are several advantages to the use of foundation models. One example is the potential to boost the performance on small datasets. Having access to a model pre-trained on a larger, similar dataset provides a ‘head start’ for the downstream task, allowing the model to focus on the intricacies of the smaller dataset rather than having to rediscover the general structure of the data. In addition, a foundation model can save both human and computational resources. While pre-training may be a resource-intensive task, the downstream models would require less training, less data, and less time spent on optimization compared to what would be required if these models were built from scratch. Another interesting aspect is that a trained foundation model could enable implicit sharing of data and resources between different parts of the research community, a sharing that would not otherwise be possible. Foundation models are poised to play a central role in particle physics, especially in the context of experiments at the Large Hadron Collider (LHC). They are able to process the huge amounts of complex and diverse data generated by the LHC and have the potential to integrate multiple modalities - such as images, high-level event features or particle four-vectors - allowing more comprehensive data analyses. Current work on foundation models for particle physics has mainly focused on different event generation and classification tasks [ 38 , 50 , 29 , 10 , 28 , 43 ] , investigating pre-training strategies, encoding schemes and different architectures, and quantifying the transfer learning and generalization capabilities of these models. In this paper, we explore the potential of using large quantities of Open Data from the LHC for pre-training foundation models. The LHC experiments have made significant amounts of data publicly available [ 17 , 18 , 19 , 20 , 3 ] . Foundation models can leverage the LHC open data for training and fine-tuning, potentially integrating different data modalities and data from different experiments. Furthermore, training foundation models on real data will result in a more accurate representation of particle physics processes than training on simulations. Finally, foundation models trained on a large variety of actual LHC data will help to promote transparency and collaboration in particle physics research. Here we present the AspenOpenJets (AOJ) dataset and study the transfer learning of foundation models obtained with this dataset. The AOJ dataset has been constructed from the CMS 2016 ‘JetHT’ open data [ 19 , 20 ] and consists of 178\,\rm M jets, making it the largest ML-ready dataset for LHC applications. Jets are important objects at the LHC, consisting of collimated sprays of particles. Previous open data studies of CMS jets have explored the substructure and energy flow of the jets [ 42 , 61 , 39 ] and the usefulness of open data for quark/gluon tagging [ 26 ] , but this is the first study to prepare and make public a form of the CMS open data that allows easy use of the dataset for phenomenological studies, especially for training machine learning models. As an initial proof-of-concept study, we pre-train the OmniJet - \alpha [ 10 ] foundation model on the AOJ dataset, which is expected to mostly contain QCD jets, and then fine-tune it to generate jets of different types from the JetClass [ 49 ] dataset of simulated jets. We do two such fine-tunings: first for JetClass QCD jets, which occupy a different kinematic region compared to the jets in AOJ, and then for top jets, which differ significantly in several features. We show that pre-training on AOJ indeed benefits the downstream task, making it easier for the model to generate these types of jets with fewer training examples compared to training from scratch. The utility of CMS open data for pre-training has been previously explored in [ 38 ] , using 1M single-lepton events at the object level which were collected by CMS in 2015. Here we enlarge the scope and complexity by several orders of magnitude to encompass 178M jet events at the full constituent level, that is, including not only the high-level features of the jets themselves, but also the features of all the particles inside the jet. Furthermore, the authors of [ 38 ] used event selections to shape the open data into a similar topology to the intended target simulated data. In our work, no such shaping is needed. This paper is organized as follows. Section II describes the CMS open data and the method used to derive AOJ. It also includes a brief description of JetClass, which will be used for the downstream task. Our application example is described in section III , detailing the method and results. Finally, we present our conclusions in section IV . Links to the codebase and the AOJ dataset itself can be found after the conclusions. II Data Figure 1: Comparison of AspenOpenJets (AOJ) and QCD JetClass distributions for particle and jet level observables. Here d_{0} and d_{z} represent the transverse and longitudinal impact parameter respectively, with \sigma_{d_{0}} and \sigma_{d_{z}} being their uncertainties, and \tau_{21} and \tau_{32} are N-subjettiness variables. The last panel shows the average number of jet constituents for each particle species, accompanied by the corresponding 1\sigma standard deviation bands. II.1 CMS open data The CMS experiment [ 15 ] is a large general-purpose detector at

Read the complete text