Investigating 1-Bit Quantization in Transformer-Based Top Tagging

Saurabh Rai, Prisha, Jitendra Kumar · 2025

Read Investigating 1-Bit Quantization in Transformer-Based Top Tagging by Saurabh Rai, Prisha, Jitendra Kumar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07431.txt.

Introduction

Saurabh Rai, Prisha, and Jitendra Kumar shrink a transformer model for spotting top quarks down to one-bit weights, in a paper from 2025. Detectors at colliders must sort particle sprays in real time on tight hardware. Full models are too heavy for those chips. Their model, BitParT, keeps the transformer shape but stores each weight as a single bit.

The design borrows tricks from one-bit language models and fits them to physics data. On a public top quark data set, the tiny model tags nearly as well as its full-size parent. Size and compute needs fall by a wide margin. That opens the door to running modern models inside trigger systems. It also speeds offline rebuilding of huge data sets. The work is a proof that extreme shrinking can keep its edge.

Catalog record

CREATORSaurabh Rai, Prisha, Jitendra Kumar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07431

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Investigating 1-Bit Quantization in Transformer-Based Top Tagging Thanks: The authors acknowledge financial support from the I.I.T. Jodhpur, India, under Project No. I/RIG/JTK/20240067, and from the DST-SERB, India, under Grant No. EEQ/2023/000959. Saurabh Rai, Prisha, Jitendra Kumar Abstract The increasing scale of deep learning models in high-energy physics (HEP) has posed challenges to their deployment on low-power, latency-sensitive platforms, such as FPGAs and ASICs used in trigger systems, as well as in offline data reconstruction and processing pipelines. In this work, we introduce BitParT, a 1-bit Transformer-based architecture designed specifically for the top-quark tagging method. Building upon recent advances in ultra-low-bit large language models (LLMs), we extended these ideas to the HEP domain by developing a binary-weight variant (BitParT) of the Particle Transformer (ParT) model. Our findings indicate a potential for substantial reduction in model size and computational complexity, while maintaining high tagging performance. We benchmark BitParT on the public Top Quark Tagging Reference Dataset and show that it achieves competitive performance relative to its full-precision counterpart. This work demonstrates the design of extreme quantized models for physics applications, paving the way for real-time inference in collider experiments with minimal and optimized resource usage. I Introduction Particle physics research aims to explore the universe at the most fundamental level. The Standard Model (SM) of particle physics is a remarkably successful framework for describing the fundamental building blocks of nature and the forces that govern their interactions. Yet, many open questions remain, from the matter-antimatter asymmetry to the mystery of dark matter. High energy accelerators are used as powerful tools to mimic the conditions of the early universe by colliding particles at ultra-relativistic velocities. For example, large-scale experiments like CMS and ATLAS have been built at the Large Hadron Collider (LHC), where protons are accelerated nearly the speed of light and collide at unprecedented energies in the multi-TeV range. These collisions occur at an extremely high rate, and the outcome of each collision recorded by the detectors is defined as an “event”, capturing key information such as particle trajectories, momentum, and energies at the detector level. A collection of recorded events is referred to as a dataset and is used for physics measurements. In the SM, quarks and leptons are considered as fundamental constituents of matter. The visible matter in the universe is mostly made up of the lightest quarks, up and down, along with electrons, all bound together in atoms. The strange quark, while also considered light, usually appears only in unstable hadrons. The heavier quarks, charm and bottom, decay rapidly and are found in unstable hadrons as well. The top quark mass is 172.76 \pm 0.3 GeV/c 2 , making it the heaviest. It decays almost instantly due to its very short mean lifetime of 5\times 10^{-25} seconds, and is unique as it does not form hadrons at all. Because the top quark decays before hadronization, it allows probing the bare quark properties and performing precision Quantum Chromodynamics (QCD) studies. It also couples strongly to the Higgs field, influencing electroweak symmetry breaking and providing a sensitive probe for physics beyond the Standard Model (BSM). The decay products of the top quark include quarks and gluons that rapidly hadronize, forming collimated streams of particles known as jets. This is where “ top tagging ” becomes important, as it enables the identification of jets originating from top quark decays, distinguishing them from those arising from other processes. It leverages jet reconstruction algorithms to analyze the substructure of the jets and infer the presence of a top quark based on characteristic decay patterns. The conventional top tagging algorithm used at the Tevatron relied on b‑tagging, where top quarks were produced near threshold, so their decay products, especially the bottom quark jets, were distinctly separated and could be distinguished using secondary vertex or soft-lepton b-tagging, combined with conventional jet reconstruction and kinematic selection [ 1 , 2 , 3 ] . At the LHC, however, top quarks are often boosted, so advanced methods such as k_{T} and anti‑ k_{T} algorithms with large‑R jets, subjet b‑tagging, substructure variables, and grooming techniques are used [ 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 ] . These methods exploit the internal structure and kinematic patterns of the top quark decay products. Although these methods perform well for top tagging, they face performance challenges under extreme pile-up and suffer from large systematic uncertainties. Machine learning (ML) techniques bring several major advances to the field of top tagging. Instead of relying only on jet substructure variables, ML models can work directly with low-level jet information, exploiting complex and nonlinear correlations between jet observables [ 13 , 14 , 15 , 12 ] . Consequently, a range of approaches have been designed for ML-based top tagging, starting from boosted decision trees (BDTs) and convolutional neural networks (CNNs) [ 16 ] and fully connected dense neural networks [ 17 , 18 , 19 ] , to more advanced graph neural networks (GNNs) [ 20 , 21 , 22 ] , point cloud networks [ 23 , 24 ] , and recently transformer models [ 24 , 25 , 26 , 27 , 28 , 29 , 30 ] . These ML methods not only served as an efficient choice but also pushed tagging performance forward. Despite all these advancements, the enormous size and complexity of the LHC data still impose enormous computing demands, both in offline data reconstruction and at the trigger level. The upcoming High-Luminosity LHC (HL-LHC) will add additional challenges due to higher collision rates, larger data volumes, and processing requirements. Deep learning architectures for LHC data processing often require large and complex models to meet the demands of high data volume and complexity. They also rely on extensive training datasets and require fast, high-throughput inference. All of this makes them challenging to scale and deploy efficiently on available computing [ 13 , 31 ] . Similar requirements are also common in computer science and the machine learning domain when deploying DL models on mobile, embedded, or resource-limited platforms. To address this, the broader community has developed a range of model compression techniques, including quantization [ 32 , 33 , 34 ] , pruning [ 35 ] , and low-rank approximations [ 36 ] , which not only reduce memory but also compute costs while preserving task performance. A notable recent advancement in this direction is the BitNet family of architectures [ 37 , 38 , 39 , 40 ] , replacing full-precision floating-point multiplications with low-bit integer operations that can be efficiently implemented using bitwise computation. The BitNet approach to extreme low-bit quantization aims to maximize computational and energy efficiency while maintaining competitive performance, offering a promising direction for scalable, low-power deep learning model integration into HEP experiment pipelines. Motivated by these recent developments, we explore a 1-bit quantized variant of the Particle Transformer ( ParT ) architecture, which we refer to as the BitParT model, as a case study for top quark tagging. Our work, based on the BitNet approach for extreme quantization, addresses a similar direction as a recent study in the HEP community [ 41 ] , which demonstrated the viability of low-bit quantization across various

Read the complete text