PAHQ: Accelerating Automated Circuit Discovery through Mixed-Precision Inference Optimization

Xinhai Wang, Shu Yang, Liangyu Wang, Lin Zhang, Huanyi Xie, Lijie Hu, Di Wang · 2025

Read PAHQ: Accelerating Automated Circuit Discovery through Mixed-Precision Inference Optimization by Xinhai Wang, Shu Yang, Liangyu Wang, Lin Zhang, Huanyi Xie, Lijie Hu, Di Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23264.txt.

Introduction

Xinhai Wang and six co-authors present PAHQ, a method to speed up automated circuit discovery in language models. Circuit discovery finds the sparse subnetworks responsible for a task, a cornerstone of mechanistic interpretability. The standard ACDC method is too slow and memory-hungry for large models, and faster versions rely on linear shortcuts that hurt faithfulness.

PAHQ instead optimizes mixed-precision inference to cut cost while keeping the analysis faithful. The authors aim for circuit maps of large models that researchers can actually compute. Trustworthy circuits help explain how models produce their answers internally.

Catalog record

CREATORXinhai Wang, Shu Yang, Liangyu Wang, Lin Zhang, Huanyi Xie, Lijie Hu, Di Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-23264

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PAHQ: Accelerating Automated Circuit Discovery through Mixed-Precision Inference Optimization Xinhai Wang, Shu Yang, Liangyu Wang, Lin Zhang, Huanyi Xie, Lijie Hu, Di Wang Abstract Circuit discovery, which involves identifying sparse and task-relevant subnetworks in pre-trained language models, is a cornerstone of mechanistic interpretability. Automated Circuit Discovery (ACDC) has emerged as a pivotal methodology in circuit discovery, but its application to large language models is severely limited by computational inefficiency and prohibitively high memory requirements. Although several accelerated approaches have been proposed, they primarily rely on linear approximations to ACDC, which significantly compromises analytical faithfulness. Our proposed method for accelerating automated circuit discovery, Per Attention Head Quantization (PAHQ), takes a fundamentally different approach by optimizing the efficiency of each individual patching operation. PAHQ leverages a fundamental alignment between activation patching and mixed-precision quantization (MPQ): interpretability analysis through patching essentially performs targeted ablation studies. Therefore, we can maintain high precision exclusively for investigated components while safely reducing precision elsewhere in the network. PAHQ-accelerated ACDC reduces runtime by up to 80% and memory consumption by up to 30% compared to unaccelerated ACDC while maintaining faithfulness. Importantly, our method readily integrates with existing edge-based circuit discovery techniques by modifying the attention computation mechanism. This training-free approach provides a practical and novel pathway for accelerating mechanistic interpretability methods. Our code is available at https://github.com/626619403/PAHQ. 1 Introduction As LLMs scale up Vaswani et al. 2017 ; Minaee et al. 2025 , understanding their internal mechanisms becomes crucial for effective utilization and supervision Chen et al. 2021 ; Li et al. 2022 ; Manzari et al. 2023 . A main method for understanding model behavior is to decompose a model into circuits—interpretable computational subgraphs responsible for specific model behaviors Wang et al. 2022 ; Bereska and Gavves 2024 . The Automatic Circuit Discovery (ACDC) method proposed by Conmy et al. 2023 served as the first scalable, end-to-end approach for identifying behavior-related circuits in models. While ACDC is effective, it relies on activation patching (replacing activations from one forward pass with those from another) using an expensive greedy search that ablates each edge, which limits its scalability Bhaskar et al. 2024 . One alternative is Edge Attribution Patching (EAP) Syed et al. 2023 , which replaces activation patching with gradient-based linear approximations. While EAP is significantly faster, these first-order approximations often compromise faithfulness to the full model. Given these computational and memory constraints, along with the need to maintain model performance, quantization Han et al. 2016 ; Jacob et al. 2017 ; Krishnamoorthi 2018 naturally emerges as a promising approach to address these challenges. The quantization operation is to convert the high-precision floating-point parameters of the model into lower-precision representations to reduce the memory footprint and speed up inference. However, in Section 2 , we found that when precision is reduced from 32 bits to 8 bits, direct quantization (also called Round-to-Nearest Quantization, RTN-Q) significantly increases the minimum quantization step size, causing activation values below this threshold to be truncated to zero and ultimately rendering the automated circuit discovery method ineffective. Furthermore, RTN-Q reduces activation precision when the model aggregates these values (e.g., in residual streams).. When two numbers are added, the larger number dominates the smaller one, masking the activation differences of the selected edge so that the algorithm cannot detect the influence of the selected edge. This represents the most challenging obstacle in applying quantization to accelerate automated circuit discovery methods. Fortunately, all these problems stem from the precision degradation of selected edges, and we can use mixed-precision quantization (MPQ) Saxena et al. 2025 ; Huang et al. 2025 ; Lee et al. 2024a to mitigate these effects. Mixed-precision quantization enables selective maintenance of high numerical precision exclusively for specific components under investigation while safely reducing precision elsewhere in the network, avoiding the issues caused by direct quantization. However, most conventional mixed-precision quantization methods require substantial pre-training time to identify important components. Moreover, due to GPU memory limitations, mixed-precision quantization methods typically stores high-precision weights in CPU memory, while only the critical weight subset needed for current computation and all low-precision weights are retained on GPU. The dynamic nature of important weights introduces frequent CPU-GPU weight migration in automated circuit discovery, resulting in significant I/O latency. Our key insight is that there exists a natural alignment between activation patching Conmy et al. 2023 and mixed-precision quantization techniques: activation patching fundamentally conducts targeted ablation by examining how specific neural activations affect model outputs, which perfectly aligns with mixed-precision quantization strategies. Since we already know which components are important when using circuit discovery methods, we can directly leverage the selected edges to eliminate the pre-training process. This characteristic also enables us to predict the next high-precision weights to be accessed, allowing us to effectively mask data transfer overhead through a computation-communication overlap strategy that interleaves low-precision computation with high-precision weight communication, while maintaining optimized memory usage to address the CPU-GPU communication bottleneck. Motivated by these insights, we develop PAHQ (Per-Attention Head Quantization), a training-free quantization method for automated circuit discovery. We observe that automated circuit discovery algorithms typically evaluate only one edge at a time. We leverage this property by maintaining high precision exclusively for the critical component involved in that edge evaluation, while converting all other components to lower precision. To address the memory scheduling and precision compatibility challenges, we implement a comprehensive CUDA stream scheduler for attention heads that effectively masks high-precision weight loading latency through concurrent low-precision computation. Across three circuit discovery tasks, ACDC accelerated by PAHQ demonstrated over 40% improvement in AUC-ROC for classifying circuit component edges compared to 8-bit RTN quantization. When applied to GPT-2, ACDC accelerated by PAHQ reduces computation time by approximately 80% while achieving memory compression ratios exceeding 30% compared to unaccelerated ACDC. 2 Direct Quantization Does Not Work on ACDC To better illustrate our proposed approach, we provide necessary preliminaries about automated circuit discovery and quantization in the appendix A . In this section, we present preliminary studies on directly applying quantization to ACDC to determine whether it can improve ACDC’s efficiency. Figure 1: ROC curves. RTN-quantization degrades performance. We directly applied quantization from 32 bits to 8 bits to all weights and activations of the model Radford et al. 2019 in the ACDC Conmy et al. 2023 method, which is called Round-to-Nearest Quantization (RTN-Q), on the Indirect Object Identification (IOI) task Wang et al. 2022 and tested its performance. Figure 1 shows that the AUC-ROC was significantly reduced compared to the original ACDC, indicating that direct quantization is not suitable for it. As mentioned in Appendix A , since the replacement operation only changes the activation value of the selected edge, the difference in the model’s output is primarily related to the difference in activation values of the selected edge, rather than other components. Specifically, this degradation can be attributed to two main factors: Numerical Underflow and Mantissa Loss. Numerical Underflow. When the quantization precision of weights and activation values is reduced from 32 bits to 8 bits (or less), the minimum representable precision increases substantially. For FP8_E4M3 1 1 1 FP8_E4M3 is an 8-bit floating point storage format, with 4 bits for the exponent, 3 bits for the mantissa, and one bit for the sign. , the

Read the complete text