HALO: H ardware- a ware quantization with lo w critical-path-delay weights for LLM acceleration
Rohan Juneja, Shivam Aggarwal, Safeen Huda, Tulika Mitra, Li-Shiuan Peh · 2025
Read HALO: H ardware- a ware quantization with lo w critical-path-delay weights for LLM acceleration by Rohan Juneja, Shivam Aggarwal, Safeen Huda, Tulika Mitra, Li-Shiuan Peh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-19662.txt.
Introduction
Rohan Juneja and four co-authors present HALO, a 2025 arXiv paper on shrinking language models with the chip in mind. Standard shrinking picks bit counts blind to how real circuits behave. That leaves speed and energy gains on the table, since some math units finish early while others lag. HALO instead asks which weights sit on the slow path through the chip's math units.
The method favors weights whose paths run fast, which lets the chip clock higher with a few voltage and speed tweaks. It also calms switching inside the math units, saving energy. On common AI chips, HALO more than tripled speed on average and cut energy use by about half, the team reports, with little loss of answer quality.
Catalog record
| CREATOR | Rohan Juneja, Shivam Aggarwal, Safeen Huda, Tulika Mitra, Li-Shiuan Peh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-19662
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HALO: H ardware- a ware quantization with lo w critical-path-delay weights for LLM acceleration Rohan Juneja, Shivam Aggarwal, Safeen Huda, Tulika Mitra, Li-Shiuan Peh Abstract Quantization is critical for efficiently deploying large language models (LLMs). Yet conventional methods remain hardware-agnostic, limited to bit-width constraints, and do not account for intrinsic circuit characteristics such as the timing behaviors and energy profiles of Multiply-Accumulate (MAC) units. This disconnect from circuit-level behavior limits the ability to exploit available timing margins and energy-saving opportunities, reducing the overall efficiency of deployment on modern accelerators. To address these limitations, we propose HALO , a versatile framework for Hardware-Aware Post-Training Quantization (PTQ). Unlike traditional methods, HALO explicitly incorporates detailed hardware characteristics, including critical-path timing and power consumption, into its quantization approach. HALO strategically selects weights with low critical-path-delays enabling higher operational frequencies and dynamic frequency scaling without disrupting the architecture’s dataflow. Remarkably, HALO achieves these improvements with only a few dynamic voltage and frequency scaling (DVFS) adjustments, ensuring simplicity and practicality in deployment. Additionally, by reducing switching activity within the MAC units, HALO effectively lowers energy consumption. Evaluations on accelerators such as Tensor Processing Units (TPUs) and Graphics Processing Units (GPUs) demonstrate that HALO significantly enhances inference efficiency, achieving average performance improvements of 270% and energy savings of 51% over baseline quantization methods, all with minimal impact on accuracy. 1 National University of Singapore, 2 OpenAI {rohan, shivam, tulika, peh}@comp.nus.edu.sg, [email protected] Code — https://github.com/ecolab-nus/HALO 1. Introduction Transformer-based large language models (LLMs) have grown exponentially, increasing 100-fold every two years, far outpacing the 3.1× improvements in hardware. This widening gap has made inference increasingly costly, as seen with models like LLaMA (65 billion parameters) and GPT-4 (1.76 trillion parameters), which require vast computational resources. Quantization, which reduces model size and cost by lowering bit-width, is crucial for efficiency. However, implementing quantization is challenging because of the diverse, fragmented landscape of hardware accelerators, each optimized for specific quantization techniques and data types. Existing quantization techniques fail to account for these diverse hardware, highlighting the need for adaptable strategies that can optimize model efficiency across different accelerator architectures by leveraging circuit-level characteristics. Figure 1: HALO quantization framework, using architectural details to yield Pareto-optimal trade-offs for diverse deployments. Most existing quantization techniques ( Aggarwal et al. 2024 ) overly focus on reducing bit-width without considering hardware-specific factors, often treating critical components like Multiply-Accumulate (MAC) units as a black box. In ML accelerators like Google’s TPUs, MAC units used for matrix multiplications central to LLM inference, occupy 77-80% of the chip area and account for 50-89% of total power consumption ( Elbtity et al. 2024 ) . Our key observation is that MAC units exhibit significant variability in performance and power characteristics, depending on specific weight patterns after quantization. Critical-path-delay in a hardware circuit refers to the time taken by the longest chain of logic operations, which determines the maximum speed at which the overall system can operate. In digital circuits, the path taken by a computation depends on the weight values being processed. Through detailed circuit analysis, we discover that certain weight configurations enable shorter critical-paths, allowing the MAC units to operate at higher frequencies, whereas others result in longer critical-paths that constrain the system’s overall operating speed, as illustrated in Fig. 4 and Fig. 5 . This variability highlights an unexplored opportunity to enhance inference performance and energy efficiency through more sophisticated, hardware-aware weight quantization strategies. Yet, current techniques remain oblivious to this crucial quantization-hardware interplay. Hardware accelerators for LLMs, such as GPUs, use Dynamic Voltage and Frequency Scaling (DVFS) to balance performance and power. NVIDIA GA100 GPUs, for instance, support up to 181 DVFS configurations ( Ali et al. 2023 ) . Quantization directly influences the critical-path-delays in MAC units, which in turn determines how well DVFS can be utilized to optimize both performance and energy efficiency. Our key insight is that by carefully selecting quantization levels that correspond to favorable critical-path-delays, we can enable higher operating frequencies while maintaining model accuracy. We present HALO , Hardware-Aware LOw critical-path delay quantization framework, illustrated in Fig. 1 , which unifies post-training quantization and DVFS into a single hardware-driven optimization strategy. At a high level, HALO first groups weights by their critical‑path-delay and then assigns each group the best voltage‑frequency setting to speed up “fast” tiles and save energy on “slow” ones. HALO accepts detailed hardware descriptions, such as MAC unit frequency and power profiles, supported DVFS configurations, and tile sizes, along with user-defined design goals for accuracy, performance, and energy efficiency. It outputs a set of Pareto-optimal quantized models, each paired with a corresponding DVFS schedule tailored for efficient deployment on target accelerators such as GPUs and TPUs. Below, we outline our main contributions. • We perform comprehensive evaluation of MAC circuit timing and energy variations, identifying weights that are critical for preserving model accuracy through gradient-based sensitivity profiling. • We introduce HALO, a hardware-aware quantization method utilizing timing, energy, and weight sensitivity to boost LLM efficiency. • We propose efficient integration techniques for HALO into existing hardware architecture such as TPUs and GPUs, enhancing performance while preserving accuracy. Our integrated approach delivers significant benefits for LLM inference, achieving on average 270% performance gains and 51% energy savings. These improvements are driven by HALO’s ability to co-optimize quantization levels with DVFS operating points, effectively bridging the gap between model compression and hardware adaptation. 2. Related Works Quantization Methods. Post-training quantization (PTQ) methods such as GPTQ ( Frantar et al. 2023 ) and AWQ ( Lin et al. 2024 ) aim to reduce LLM weights to 3 or 4 bits while preserving key precision. Other approaches like outlier-aware quantization ( Lee et al. 2024 ) and SqueezeLLM ( Kim et al. 2024 ) differentiate between sensitive and non-sensitive weights to guide quantization. SmoothQuant ( Xiao et al. 2023 ) shifts quantization complexity to weights, enabling both weights and activations to be quantized to 8 bits. However, these methods operate at the algorithmic level and do not account for underlying hardware. HALO is the first framework to integrate circuit-level insights, considering timing and DVFS constraints, into the quantization process to improve efficiency without compromising LLM accuracy. Quantization Accelerators. GOBO ( Zadeh et al. 2020 ) and OlAccel ( Park et al. 2018 ) use mixed precision to preserve accuracy, while BitFusion ( Sharma et al. 2018 ) adapts precision based on workload needs. GOBO depends on full-precision compute units, and OlAccel encodes outliers using coordinate lists, both requiring custom accelerators. HALO, in contrast, is an orthogonal software-level technique that enhances efficiency without demanding hardware changes, making it suitable for deployment on current AI infrastructure (such as GPUs and TPUs), as well as future accelerators. 3. Background Figure 2: Impact of MAC unit on systolic array efficiency. The Multiply-Accumulate (MAC) unit is a fundamental component in AI accelerators, playing a significant role in both power consumption and area utilization. As illustrated in Fig. 2 (a), each MAC unit features three input ports and two output ports. The unit operates by multiplying the weight w with the activation a to produce the product wa . This product is then added to the third input y_{n-1} , resulting in the updated partial sum y_{n} .