An efficient leakage power optimization framework based on reinforcement learning with graph neural network Authors: Peng Cao, Yuhan Dong, Zhanhua Zhang, Wenjie Ding, Jiahao Wang Published: 2024 ABSTRACT AbstractThreshold voltage (Vth) assignment is convenient for leakage optimization due to the exponential relation between leakage power and Vth by swapping logic cells without routing effort. However, it poses great challenge in large scale circuit design as an NP-hard problem. Machine learning-based approaches have been proposed to solve this problem, aiming to achieve well tradeoff between leakage power reduction and runtime speed up without new induced timing violation. In this paper, a leakage power optimization framework based on reinforcement learning (RL) with graph neural network (GNN) is first-ever proposed to formulate Vth assignment as a RL process by learning timing and physical characteristics of each circuit instance with GNN. Multiple instances are selected in a non-overlapped manner for each RL action iteration to speed up convergence and decouple timing interdependence along circuit path, where the corresponding reward is carefully defined to tradeoff between leakage reduction and potential timing violation. The proposed framework was validated by the Opencores and IWLS 2005 benchmark circuits with TSMC 28 nm technology. Experimental results demonstrate that our work outperforms prior non-analytical and GNN-based methods with better leakage power optimization by additional 5% to 17% reduction, which is highly consistent with the commercial tool. When transferring the trained RL-based framework to unseen circuits, it achieves the roughly identical leakage optimization results as seen circuit and speed up the runtime by 5.7× to 8.5× compared with commercial tool.Keywords: Threshold voltage, Leakage power, Reinforcement learning, Graph neural networkSubject terms: Engineering, Electrical and electronic engineering Introduction With the continuous development of integrated circuit, the proportion of leakage power in the total consumption is increasing as process node scales (up to 40% in 7 nm technology node1), which means that the optimization of leakage power consumption has become an essential step in advanced technology. Therefore, threshold voltage (Vth) assignment and gate size adjustment are often combined to reduce leakage power, in which Vth assignment is more individually used for its convenience in swapping gates with the same footprint and no extra routing effort. Moreover, leakage power will change exponentially as threshold voltage changes instead of the linear relation with gate size2. Unfortunately, the available threshold voltages of an instance are discrete which makes leakage power optimization problem to be an NP-hard problem3. Exiting Vth assignment algorithms in commonly used electronic design automation (EDA) tools are usually based on various pseudo-linear heuristics or analytical methods driven by static timing analysis (STA) which can easily guide into globally sub-optimal solutions4. In recent years, besides the extensive research on the related hardware implementations5,6, neural networks have been utilized in academia to overcome the shortcomings of traditional EDA algorithms, trying to explore a excellent enough Vth assignment scheme in the design space7,8. Among these, graph neural network (GNN) has gained prominence due to their ability to effectively learn from graph-structured data9, which aligns well with circuit design tasks10–12. However, supervised learning methods often require large amounts of labeled data, and when this data is insufficient or not representative, models can suffer from over-fitting. This over-fitting makes it difficult to generalize beyond the training set and limits the model’s ability to explore the broader design space or find globally optimal solutions for new, unseen cases13. As a machine learning method, reinforcement learning (RL) studies agents interacting with exterior environments. In general, RL is an experience-driven paradigm of maximizing the total reward by taking actions based on the environment. RL agent receives feedback from the environment in the form of rewards or penalties based on its actions and uses this feedback to improve its future actions. Over time, through exploration and exploitation of the environment, the agent seeks to learn an optimal policy13. In some well-defined environment with discrete or continuous action space, it has shown huge success and been proven to be even more effective than human performance, such as Go14. In the realm of EDA, some works12,15–17 have applied RL to different applications and achieved impressive results. Inspired by this, since Vth assignment for leakage power optimization can be well treated as a Markov decision process (MDP), RL have huge potential to achieve better performance than prior approaches. Combining the strengths of GNN and RL, as demonstrated in recent research4,17,18, offers a powerful approach that capitalizes on GNN’s ability to model complex circuit structures and RL’s capacity for adaptive decision-making in dynamic environments. In this work, we propose a RL-based leakage power optimization framework to perform Vth assignment by leveraging GNN to learn timing and physical attributes from instances of circuit netlist. The proposed framework outperforms prior learning based methods in terms of leakage reduction and achieves consistent optimization results with commercial tools as well as significant speedup when being transferred to untrained circuits. The main contributions of our work are summarized as follows. To the best of our knowledge, we are the first to apply RL to assign Vth for leakage power optimization, in which the embeddings from GNNs are taken as states and the Vth assignments are considered as actions. The reward function is well defined to balance leakage power saving and potential risk of new timing violations. To the best of our knowledge, we are the first to apply RL to assign Vth for leakage power optimization, in which the embeddings from GNNs are taken as states and the Vth assignments are considered as actions. The reward function is well defined to balance leakage power saving and potential risk of new timing violations. In order to speed up convergence of RL agent, multiple instances are selected in each action iteration to get their rewards in parallel, which are processed in a non-overlapped manner to decouple the timing interdependence along circuit paths. In order to speed up convergence of RL agent, multiple instances are selected in each action iteration to get their rewards in parallel, which are processed in a non-overlapped manner to decouple the timing interdependence along circuit paths. The trained leakage power optimization framework has been validated to be highly consistent with commercial tool for both seen and unseen circuits with much less runtime. The trained leakage power optimization framework has been validated to be highly consistent with commercial tool for both seen and unseen circuits with much less runtime. In the following, the academic works related to our work are briefly summarized, following by the problem formulation, graph representation learning, reinforcement learning, and violation removal method of our proposed framework in detail. Eventually, experimental results demonstrate the advantage of the proposed framework by comparing with competitive works and commercial tool. Related works Vth assignment for power optimization have been widely studied in recent years. Some traditional non-analytical methods such as greedy or sensitivity driven heuristic19, dynamic programming algorithm20 and critical path iterations algorithm2 are used to optimize leakage power. The main idea in those methods is assigning a cost function to all instances and finding the feasible Vth assignment according to the function. Moreover, power optimization problems are also formulated into Lagrangian relaxation (LR) problems21,22 or linear programming problems23 so as to minimize leakage power under timing constraints analytically. Also, the interest in the application of machine learning (ML) algorithms for EDA has also increased. Bao24 and Patanjali et al.25 leverage linear regression and support vector machine (SVM) to enhance the power optimization quality respectively. Lu et al.10 and Mallappa et al.7 use different GNN models to predict optimized Vth and Wang et al.8,11 proposed directed graph to further improve the prediction accuracy. However, previous non-analytical or analytical Vth assignment solutions are extremely time-consuming and ML-based approaches may suffer from poor performance when being transferred to unseen designs. RL is a promising machine learning method that has been demonstrated to achieve super-human performance in some high-dimensional control problems with the help of deep learning13. Google applied RL to place marcos and achieved gratifying design turn-around time (TAT) and performance, power and area (PPA) improvement26. Other works used RL to solve gate sizing4, floorplan16 and global routing12,17,27 problems and optimize EDA tools parameters15,28. The success of the above works prove that reinforcement learning algorithm has great potential in EDA field and can achieve surprising solutions to EDA problems. Problem formulation For leakage power optimization problem, the Vth assignment can be formulated as an MDP while the trade-off between leakage power and timing slack can be measured as a reward to indicate the concentration of a given action. Considering this, the RL-based solution can be perfectly applied to solve this problem. What’s more, GNNs were adopted to capture more topology information to enhance the model’s transferable capabilities. Specifically, we take each instance’s embedding obtained from GNNs as a state and swapping a new Vth means an action. The reward is set considering the trade-off between leakage power and timing slack, which can be reflected by total negative slack (TNS). The following shows in detail how to map these variables to the specific concepts of a RL process. State(s) A state s stands for the representation of instance in a design, encapsulating its timing and physical features, along with characteristics of neighboring cells. The state space has a dimension of 192 in this work, achieved by concatenating the output embeddings from three GNN layers, each having a dimension of 64. State(s) A state s stands for the representation of instance in a design, encapsulating its timing and physical features, along with characteristics of neighboring cells. The state space has a dimension of 192 in this work, achieved by concatenating the output embeddings from three GNN layers, each having a dimension of 64. Action(a) An action a represents a new Vth assigned to an instance in the current state s. In this work, the action space has a dimension of 4, corresponding to the four types of threshold voltages (ULVT, LVT, RVT, and HVT) available for selection for each instance. Action(a) An action a represents a new Vth assigned to an instance in the current state s. In this work, the action space has a dimension of 4, corresponding to the four types of threshold voltages (ULVT, LVT, RVT, and HVT) available for selection for each instance. Reward(r) A reward r refers to the results given by the environment while assigning the current action a to the current state s, which represents the trade-off between leakage power and TNS. For each iteration, the goal of RL agent is to maximize the total reward of all selected instances. Reward(r) A reward r refers to the results given by the environment while assigning the current action a to the current state s, which represents the trade-off between leakage power and TNS. For each iteration, the goal of RL agent is to maximize the total reward of all selected instances. Trajectory (τ) A trajectory τ means a complete iterative process from time step t=1 (the first instance) to time step t=T (the final instance). For each instance, there is a corresponding state(st), action(at), and reward(rt) triple as (st, at, rt) stored in experience replay buffer. Trajectory (τ) A trajectory τ means a complete iterative process from time step t=1 (the first instance) to time step t=T (the final instance). For each instance, there is a corresponding state(st), action(at), and reward(rt) triple as (st, at, rt) stored in experience replay buffer. Proposed framework Overview A RL-based leakage power optimization framework is proposed in this work to perform Vth assignment, whose learning process is illustrated in detail in Fig. 1. The circuit netlist in Fig. 1a is represented as graph G=(V,E) where circuit instances are defined as graph vertexes V while the connections between them are edges E. By performing graph encoding in Fig. 1b for specific instance in graph G with its neighbours, the graph embedding is generated in Fig. 1c as the input state of the RL agent, where multiple instances are selected in a non-overlapped manner to decouple the timing interdependence of instances along circuit path. During the procedure of Vth assignment, a RL agent is utilized in Fig. 1d to capture this dependency between each instance. In each iteration of RL agent, every selected instance is assigned with new Vth in Fig. 1e to maximize the total reward. For the potential timing violations induced by RL actions with Vth assignment in Fig. 1e, lower Vth is reassigned to the instances in violated paths to remove violation in Fig. 1f with the consideration of minimizing total leakage increase in Fig. 1g. Detailed illustration of the proposed framework. Given a initial netlist, we firstly use GNNs to get the instances’ embeddings. For the instances selected according to Algorithm 1, the actions are determined by RL agent to assign new Vth and the rewards are obtained by timing and power. Finally, to fix timing violations induced by over aggressive predictions, the Vth of some instances are lowed with minumum leakage increase penalty. The details of GNN encoding and RL agent are explained as follows. Graph representation learning Feature selection In order to guarantee power optimization quality of RL agent, we extract the following node features from timing reports and SPEF files for GNN, which are listed in Table 1. These features are carefully chosen based on domain expertise and related to timing and physical attributes. It should be noticed that the information extracted from instances themselves are not sufficient to improve the final optimization performance because the final Vth of a target instance not only depends on these features from themselves, but also that from its neighboring instances. Therefore, the information of one instance’s neighbors including fanins, fanouts and siblings are also added to the feature set. Table 1Initial cell features.FeaturesDescriptionFeatures of target cell in slackWorst slack of all input pins out slackWorst slack of all output pins in slewMax slew of all input pins out slewMax slew of all output pins in capMean capacitance of all input pins cell delayCell propagation timeFeatures of its neighboring cells fanin slackWorst slack of all output pins of fanin cells fanout slackWorst slack of all input pins of fanout cells sibling slackMean slack of all sibling cells fanin capMean capacitance of all fanin cells fanout capMean capacitance of all fanout cells sibling capMean capacitance of all sibling cells Initial cell features. Local graph encoding To get more meaningful graph embedding, we leverage GraphSAGE29 to aggregate neighbors’ information and to encode features for all instances. As a variant of GNNs, GraphSAGE can get node feature information efficiently in an inductive way and can generate node embeddings for unseen data, which would be helpful for improving the transfer capabilities of our framework when assigning new Vth for instances in unseen circuits. Given a typical design, we first transform it into a netlist G, and for each instance v∈G, we can get its representation vector hvk by aggregating its previous representation at level k as: where Wkagg and Wkproj represent the aggregation matrices and projection matrices respectively. At the k-1 layer, a fixed number of neighbors are randomly sampled to obtain the set Nk(v) of neighbors, and the k-1 layer embedding representation of all neighbors huk-1 are averaged to obtain the overall embedding representation of neighbors hNk(v)k-1 by meanpool operation in (1). Then the embeddings of the current instance hvk-1 and the neighbors hNk(v)k-1 are concatenated by concat operation in (2), which is multiplied with Wkproj and activated by sigmoid operation to obtain the embedding representation hvk of the current instance v at layer k. As the number of layers k increases, the current instance can aggregate a wider range of neighbor information. Besides, inspired by other classical deep learning methods which combines shallow and deep layers30, we concatenate all layers’ output embedding in the final st because each embedding of the layers includes information from different depths. The shallower one includes more local information which include one level of neighbors, while the deeper one contains more global information which include neighbors from more levels. Owing to this, our framework can make more accurate assignments and converge more quickly. Reinforcement learning agent Instances selection During each action iteration, new Vth is assigned to instance(s) in netlist and the corresponding reward is calculated according to the fluctuations of leakage and TNS by timing and power analysis. Ideally, in order to fully consider the impact of Vth assignment for each instance, all instances should be selected one by one during each action iteration, which is undoubtedly unpractical due to tremendous timing and power analysis effort. Instead, selecting several instances each time may suffer from the timing interdependence issue, since the timing slack of one instance may be involved by the new Vth assignment of its neighbouring instances. To tackle this dilemma, the instances are selected in a non-overlapping manner for each action iteration in this work, where only the instances far away from each other in the netlist graph may be selected simultaneously to perform Vth assignment. In order to reduce the impact of neighbourhood to an negligible level, instances with n-hop away in the circuit graph are selected where n is set to be three in this work. Note that the node directly adjacent to the current node is called the 1-hop neighbor of the current node and n-hop indicates that the shortest distance between two nodes is n. Algorithm 1 presents the detailed implementation of the instance selection process. As shown in Algorithm 1, based on a initial instance set V′ by being randomly selected from the complete set of all instances W, the instances which are over n-hop away from any of V′ are identified to be not overlapped with V′ and added into it. The purpose is mainly to dismiss the impact of one instance’s neighbors to itself. Finally, we order the selected instances descendingly according to their worst slacks, where those with larger slacks would be assigned to lower Vth with higher priority. Instance selection in each iteration. Reward function The goal in the leakage optimization process by Vth assignment is to minimize leakage power and maximize TNS, which is guided by the custom defined reward function. Noting that increasing one instance’s Vth will bring lower leakage power and higher cell delay which may cause timing violation, the rewards function is defined as (3) to balance the impact to leakage and TNS during new Vth assignment. where Pt-1 and TNSt-1 mean the leakage power and TNS at the state st-1 while Pt and TNSt denote the leakage power and TNS at the state st after Vth assignment. Specifically, if TNS does not decrease with the Vth assignment, it means that no new timing violations are induced so that the reward Rtst,at is defined as the normalized increase or decrease of leakage power with its original value, Pt-1. Otherwise, it means that new timing violations occur and the reward is defined to be − 1 as penalty. DQN networks In this work, we use the Deep Q network (DQN)31 as a RL agent to perform Vth assignment, which consists of an Online network and a Target network and maintains an experience replay buffer. Note that the structures of both neural networks are exactly the same. Online network Selects action at of the agent based on the current state st. This network is constantly learning and updating to improve the estimate of Q value as much as possible. At each time step, the agent selects an action based on the Q value provided by the Online network, and then updates its parameters of the network based on the action and feedback from the environment. Online network Selects action at of the agent based on the current state st. This network is constantly learning and updating to improve the estimate of Q value as much as possible. At each time step, the agent selects an action based on the Q value provided by the Online network, and then updates its parameters of the network based on the action and feedback from the environment. Target network Used to calculate the maximum Q value of the next state st+1. The parameters of this network are not updated at every time step, but are copied from the parameters of the Online network at intervals. This can make the learning process more stable and avoid the shock problems caused by the rapid update of Online network. Target network Used to calculate the maximum Q value of the next state st+1. The parameters of this network are not updated at every time step, but are copied from the parameters of the Online network at intervals. This can make the learning process more stable and avoid the shock problems caused by the rapid update of Online network. Experience replay In order to break the correlation between data and improve the efficiency of learning, DQN stores the agent’s experience (st,at,rt,st+1) in a buffer, and then randomly draws samples from it for learning. Experience replay In order to break the correlation between data and improve the efficiency of learning, DQN stores the agent’s experience (st,at,rt,st+1) in a buffer, and then randomly draws samples from it for learning. Algorithm 2 presents the training process of our RL agent based on DQN. Firstly, we initialize experience replay buffer D, Online network parameters θ and Target network parameters θ- in line 1 and then the two networks are trained jointly in line 2–19 with N episodes. Specifically, during each episode, we select instances from the design according to Algorithm 1, use GNN on the selected instances to get the corresponding embeddings s, and count the number of instances T in line 3–5. Then in line 6–8, the Online network executes ε-greedy actions on all selected instances and predicts the Q values of all actions, from which we select the actions corresponding to the maximum Q values. The actions at are executed and the timing and power analysis is performed to get the reward rt and the next state st+1 in line 9. After storing all (st,at,rt,st+1) in the experience replay buffer in line 10, the Online network parameters are updated by Temporal Difference error (TD-error) in line 11–15. Firstly, a batch of B transitions (st,at,rt,st+1) are randomly selected from the experience replay buffer D as training samples in line 11. Then the target Q value, yt, of the action at performed by the current state st is calculated in line 12–14, where the Q values of all actions taken from the next state st+1 are predicted by the Target network while the maximum of these Q values is selected to calculate yt. The difference between the target Q value, yt, and the predicted Q value, Q(st,at|θ), are used to calculate the mean square error loss and gradient descent is adopted to update the Online network parameters θ in line 15. The Online network parameters are periodically copied to the Target network in line 16. At any episode, the RL training process would be terminated if the reduction of leakage power of the design is less than ΔPth in the previous Nth iterations as shown in line 17–19. RL training process. Violation removal Once Algorithm 2 terminates, the circuit instances are assigned by new Vth as the output of RL agent. However, it should be admitted that over aggressive Vth assignment may lead to timing violations at some endpoints, which should be fixed by redesigning lower Vth for instances in violated paths while minimizing leakage power recovery. Algorithm 3 introduces the procedure for removing timing violations after Vth assignment in detail. In line 1–3, a cost function cv is defined each instance to evaluate the leakage recovery between the leakage for the instance in the optimized netlist design by RL agent, P(Vthh), and that for the reassigned instance with one order lower than Vthh, P(Vthl). Specifically, by sorting all violated paths in ascending order with their timing slacks in line 4, the instances in the critical path, p, are considered to be reassigned to lower Vth with high priority to remove timing violations at the cost of leakage recovery in line 5–16. For all instances in path p, the instance with the minimal cost is selected as the candidate, g∗, in line 8, whose Vth is lowered by one order in line 9, e.g. from HVT to RVT. After that, the cost of the selected instance g∗ is updated. When the lowered Vth for g∗ has been the lowest, i.e. ULVT, the cost of g∗, cg∗, is set to be infinity in line 11. Otherwise, cg∗ is updated to be the difference between the leakage for the lowered Vth and one order lower of that in line 12. This process is repeated in line 6–15 for all instances in a path and in line 5–16 for all violated path to remove all timing violations in optimized circuit with minimal leakage recovery. Violation removal. Experimental results and discussion Experiment setup In this work, the proposed leakage optimization framework was implemented in Python3 with Pytorch 1.1 on a Linux machine equipped with NVIDIA Tesla V100 GPU and 32 GB memory. It was validated under TSMC 28 nm process by benchmark circuits from IWLS 2005 and Opencores, whose attributes are summarized in Table 2 in terms of the numbers of flip-flops and combinational cells. All these circuits were synthesized, placed and routed with Synopsys Design Compiler and IC Compiler, in which four types of threshold voltage, including ULVT, LVT, RVT and HVT, are provided for each circuit instance. For fair comparison, all instances in routed netlist are initialized to be the lowest Vth, i.e. ULVT, with the tightest timing constraint. The timing and power analysis was carried out with Synopsys PrimeTime to capture the initial node features for GNN and evaluate the rewards of each action for RL. Table 2Benchmark statistics.Designs# FF# CellCategorymc_top11265721pci32358213,512openGFX147711,498usbf_top175210,824aes58411,118wb_dma80837,339eth_top1057635,347Seen (Training)des #1/#2/#3190/190/1901530/1609/1518tv80 #1/#2/#3346/346/3463921/3689/3605Unseen (Transfer) Benchmark statistics. The main parameters of our framework are summarized as follows. The input dimension for the GNN is set at 12, for 12 timing and physical features are selected for each instance. The GNN consists of three layers, each with a dimension of 64. ReLU is employed as the activation function, and batch normalization is applied after each layer to enhance training speed and improve convergence. The RL agent is structured with three linear layers to derive actions. The input state dimension is 192, representing the combined dimensionality of the three GNN layers. The hidden layer dimensions are 64, while the output layer dimension is 4, corresponding to the four types of threshold voltages. In order to demonstrate the leakage optimization quality and runtime of the proposed framework, it was validated by two modes with different benchmark circuits as shown in Table 2, namely training mode and transfer mode. In training mode, the proposed framework is trained with seen circuits from scratch to evaluate the leakage optimization quality as well as timing slack. In transfer mode, the unseen circuits were synthesised and implemented with different timing constraints to obtain different netlists, which were validated by the trained framework for leakage optimization to demonstrate the generalization performance. Validation in training mode Tables 3 and 4 demonstrate the leakage optimization results for this work and competitive methods including a non-analytical method2, a GAT-based method7 and a GNN-LSTM-based method8, where the leakage power and timing before and after leakage optimization are listed. The commercial tool PrimeTime is utilized to perform leakage optimization as baseline. Additionally, all designs are free of timing violations prior to leakage optimization, as well as after optimization using both PrimeTime and the non-analytical method. Table 3Leakage power comparison on seen circuits.DesignsBefore opt.Prime-timeNon-analytical2GAT7GAT7 w. removalGNN-LSTM8GNN-LSTM8 w. removalOurs w/o. removalOursmc_top0.4340.0180.0200.0180.0180.0180.0180.0180.018pci321.2830.2320.3160.2330.2400.2310.2320.2320.232openGFX1.4600.2260.2650.2840.2990.2290.2360.2200.222usbf_top1.0450.1070.1130.0910.1450.0970.1110.1080.111aes0.7670.1430.1550.1400.1510.1490.1560.1470.148wb_dma2.3290.1640.1350.1640.2330.1590.1820.1540.158eth_top2.0240.0210.0240.0260.0260.0240.0250.0210.021Norm.18.93×1.00×1.13×1.05×1.20×0.95×1.05×0.99×1.00×The unit for leakage is mW. Leakage power comparison on seen circuits. The unit for leakage is mW. Table 4Timing comparison on seen circuits.DesignsGAT7GAT7 w. removalGNN-LSTM8GNN-LSTM8 w. removalOurs w/o. removalOurs#vioTNS#vioTNS#vio.TNS#vio.TNS#vio.TNS#vio.TNSmc_top000000000000pci3240− 0.28002− 0.02000000openGFX265− 0.32005− 0.210012− 0.0900usbf_top768− 1.9800456− 1.990031− 0.1600aes89− 0.5400110− 1.37004− 0.1300wb_dma887− 1.8500512− 2.970047− 0.7800eth_top11− 0.04009− 0.02002000Ave. TNS–− 0.72–0–− 0.94–0–− 0.16–0#vio stands for the number of violated paths and the unit for TNS is ns. Timing comparison on seen circuits. #vio stands for the number of violated paths and the unit for TNS is ns. It can be seen from Tables 3 and 4 that owing to Vth assignment by RL agent with timing removal technique, this work achieves tool-quality leakage optimization results with no induced timing violation and consistent leakage power with PrimeTime, outperforming prior non-analytical method2, GAT-based method7 and GNN-LSTM method8. Before applying the violation removal process, as demonstrated in Table 3, our algorithm effectively reduces leakage power to 0.99× that of the leakage optimized by PrimeTime. Table 4 further indicates that only a small number of paths still exhibit violations requiring correction, with an average TNS of − 0.16ns across all circuit designs, allowing our proposed violation removal method to effectively resolve all timing violations within an acceptable runtime. For the non-analytical method2 which introduces a primal-dual approximation algorithm, in spite of no timing violations are induced, the leakage power is 13% higher than ours. The GAT-based method7 employs a GAT model to predict per-cell recoverable slack, and translate these slack values to equivalent power savings. Its optimized leakage is 5% higher than that of this work, with − 0.72ns TNS on average. The GNN-LSTM method8, which incorporates circuit information through a GNN-BiLSTM-based framework, achieves better power optimization than both PrimeTime and our work as indicated in Table 3. However, it exhibits an average TNS of − 0.94ns, suggesting over aggressive leakage optimization learnt from training data that results in timing violations. When removing all timing violation with Algorithm 3, the leakage for the GAT-based and the GNN-LSTM-based method would respectively rise to 20% and 5% more than our optimized leakage result. Table 5 compares runtime of different methods and provides a time breakdown of each stage of our framework. It can be seen that our work achieves an average speedup of 6.9× on seen circuits compared to PrimeTime, while also outperforming both the non-analytical2 and GNN-LSTM method8 in terms of computational efficiency. Although the runtime is longer than that of the GAT-based method7, its power and timing performance are significantly improved. Within our framework, GNN is the most time-consuming component, accounting for 45.4% of the total runtime on average, as it involves both extracting circuit features and constructing as well as processing the graph structure. The RL module is relatively efficient at 23.5%, indicating effective optimization in decision-making. The violation removal algorithm, at 31.1%, underscores its critical role in ensuring timing correctness. Table 5Runtime comparison of different methods on seen circuits and breakdown of each stage of our framework.DesignsPrimeTimeNon-analytical2GAT7GNN-LSTM8OursGNNlRLVio. removalTotalmc_top53.38.64.05.72.2 (44.9%)1.3 (26.1%)1.4 (29.0%)4.9pci3274.022.37.110.43.2 (41.8%)2.5 (32.8%)2.0 (25.4%)7.7openGFX71.531.36.99.13.6 (46.8%)1.7 (22.8%)2.3 (30.4%)7.6usbf_top60.529.16.58.93.3 (42.1%)2.1 (27.3%)2.4 (30.6%)7.8aes69.233.96.99.94.0 (48.1%)1.8 (21.8%)2.5 (30.1%)8.3wb_dma129.646.120.324.911.6 (45.9%)4.8 (19.1%)8.8 (35.0%)25.2eth_top106.139.117.325.810.0 (47.9%)3.1 (14.6%)7.8 (37.5%)20.9% of runtime––––45.4%23.5%31.1%100%speedup1.0×2.7×8.2×6.0×–––6.9×The unit for runtime is s. Runtime comparison of different methods on seen circuits and breakdown of each stage of our framework. The unit for runtime is s. Validation in transfer mode Inspired by related RL-based works which transfer knowledge to unseen designs16,26, the proposed framework was validated on unseen circuits, tv80 and des, which was implemented by multiple timing constraints and trained with all seen circuits in Table 2. Figure 2 compares our framework in transfer mode with PrimeTime in terms of leakage power and runtime, where timing violations are all fixed in the optimized circuits for both solutions. The runtime for PrimeTime is consumed by the command fix_eco_leakage invoked in IC Compiler while that for our framework consists of the time for generating Vth assignment solution with trained RL agent and timing violation removal. It can be found that even for unseen circuits, our framework achieves tool-quality optimization results with trivial leakage power increase by 1.7–4.0% as shown in Fig. 2a while the runtime is dramatically decreased by 5.7×–8.5× compared by PrimeTime as shown in Fig. 2b, indicating significant efficiency improvement in this work. Transfer trained RL agent to unseen designs tv80 and des under different timing constraint and compare with PrimeTime. (a) Leakage power and (b) runtime. Conclusion We propose a GNN-powered RL framework that performs automatic Vth assignment for leakage power optimization without any human intervention. Experiments results demonstrate that the proposed framework provides commercial-quality power optimization result, outperforming prior non-analytical and GNN-based methods with better leakage power optimization by additional 5% to 17% reduction. When applied to unseen circuits, our framework is able to achieve power optimization results comparable to commercial tools without introducing additional timing violations, while accelerating runtime by 5.7× to 8.5×. Our method shows promise in the field of EDA by illustrating how reinforcement learning can address some of the challenges associated with large-scale circuit optimization. In terms of future applications, our framework could be extended to address multi-objective optimization tasks in circuit design, optimizing power, performance, and area simultaneously. It has the potential to assist in decision-making for various design tasks such as placement, pin assignment, routing, and timing optimization, enhancing the overall design process. Acknowledgements This work was supported in part by the National Natural Science Foundation of China under Grant 62174031 and in part by the Fundamental Research Funds for the Central Universities. Author contributions Peng Cao presented the theoretical analysis and guidance and led the writing of the manuscript. Yuhan Dong, Zhanhua Zhang and Wenjie Ding conducted the experiment. Jiahao Wang participated in the analysis of the experimental results. All authors reviewed the manuscript. Data availability The data that support the findings of this study are available from the corresponding author, upon reasonable request. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The data that support the findings of this study are available from the corresponding author, upon reasonable request.