Inverse Design in Distributed Circuits Using Single-Step Reinforcement Learning Jiayu Li, Masood Mortazavi, Ning Yan, Yihong Ma, Reza Zafarani Abstract The goal of inverse design in distributed circuits is to generate near-optimal designs that meet a desirable transfer function specification. Existing design exploration methods use some combination of strategies involving artificial grids, differentiable evaluation procedures, and specific template topologies. However, real-world design practices often require non-differentiable evaluation procedures, varying topologies, and near-continuous placement spaces. In this paper, we propose DCIDA, a design exploration framework that learns a near-optimal design sampling policy for a target transfer function. DCIDA decides all design factors in a compound single-step action by sampling from a set of jointly-trained conditional distributions generated by the policy. Utilizing an injective interdependent “map”, DCIDA transforms raw sampled design “actions” into uniquely equivalent physical representations, enabling the framework to learn the conditional dependencies among joint “raw” design decisions. Our experiments demonstrate DCIDA’s Transformer-based policy network achieves significant reductions in design error compared to state-of-the-art approaches, with significantly better fit in cases involving more complex transfer functions. † † affiliationnotice: Equal contribution 1 Introduction As 5G and 6G communication technologies and quantum computing advance towards higher frequencies, it becomes important for distributed circuit designs on resonators to achieve desirable performance ( He et al. 2020 ; de Ory et al. 2024 ) . These designs often include components such as square resonators ( Hong & Lancaster 1996 ) shown in Figure 1 , ring resonators ( Hong 2011 ) , or bar resonators ( Levy et al. 2002 ) . The specification of these circuits is expressed as a transfer function, which describes the frequency behavior of the distributed circuit. Designing a distributed circuit to achieve a desired transfer function remains a slow and laborious process. To accelerate the process, Circuit-GNN ( Zhang et al. 2019 ) trains a graph neural network (GNN) as a pre-trained model to predict the transfer function s_{21}(\{\omega_{i}\}^{m}_{i=1}) of a given design, which represents the ratio of output-to-input signals for frequencies \{\omega_{i}\}^{m}_{i=1} . By using the differentiability of this pre-trained model, Circuit-GNN solves the inverse design problem for s_{21} , but it relies on specific candidate topologies. In fact, experts often lack a predetermined preference for topology types. Additionally, predictions of s_{21} from the pre-trained model are less accurate than those generated by electromagnetic (EM) simulators. This gives rise to a key challenge: How can we generate distributed circuits that meet a desired s_{21} without assuming targeted topologies while enabling the use of simulation? (a) A 4-resonator circuit (b) Physical representation Figure 1 : 4-resonator circuit and its physical representation. For our design space exploration (DSE) problem, we can adopt reinforcement learning (RL)—a DSE approach deployed in various domains ( Lu et al. 2021 ; Budak et al. 2022 ; Lu et al. 2023 ; Ghraieb et al. 2021 ) . Prior research ( Jiang et al. 2021 ; Mirhoseini et al. 2020 ; Wu et al. 2023 ; Feng et al. 2021 ) has applied RL to DSE problems utilizing traditional multi-step RL to sample and identify near-optimal design decisions. The multi-step approach encodes partial design states and employs non-analytic gridding and masking techniques to satisfy conditionality on action probabilities. However, partial designs cannot provide performance feedback, the artificially imposed grids reduce fidelity, and output masking eliminate feedback opportunity. Single-step \theta -Resonance ( Mortazavi et al. 2022 ) over-came these limitations for definite-horizon fixed-dimensional design explorations. It avoided artificial gridding pitfalls that often grip multi-step RL DSE but relied on heuristic penalty assignment to the unavoidable unconditioned anomalous designs, and it only explored categorical design spaces. To address the above limitations, we propose D istributed C ircuit I nverse D esign A utomation (DCIDA), an inverse design framework tailored to generate near-optimal distributed circuits. DCIDA employs a trainable policy network that transforms a constant input (representing the blank design slate) to a set of conditional distributions defined on design dimensions. The corresponding joint distribution can be used to sample a series of discrete and continuous actions as a compound action in a single step. We introduce implied boundaries and deterministic interdependent mapping functions that map raw design actions to the physical representations of the distributed resonators. Our experiments show DCIDA outperforms Circuit-GNN and \theta -Resonance by generating distributed circuits that achieve significant reduction in error. DCIDA obtains the results without targeted topologies or the number of resonators. In summary, our main contributions include: • We propose DCIDA—an inverse design framework trained by single-step RL to generate near-optimal distributed circuits that meet desired transfer function s_{21} more tightly than Circuit-GNN. • We introduce design boundaries to reduce the DSE complexity. Within the boundaries, we formulate deterministic mapping functions that accurately translate raw compound actions into physical circuit designs. • Our experiments show that DCIDA excels in generating distributed circuits that more tightly meet desired s_{21} , significantly outperforming Circuit-GNN and \theta -Resonance. 2 Related Work Automating lumped circuit design. Lumped circuits consist of lumped components such as resistors, capacitors and transistors. Research in this domain focuses on determining optimal device parameters and circuit topologies to meet desired specifications. Some approaches employ optimization-based methods, including Bayesian Optimization ( Lyu et al. 2018 ) , Geometric Programming ( Colleran et al. 2003 ) , and Genetic Algorithms ( McConaghy et al. 2011 ; Lourenço & Horta 2012 ) , while some studies propose learning-based methods including supervised learning methods ( Dong et al. 2023 ) and RL methods ( Settaluri et al. 2020 ; Wang et al. 2020 ; Cao et al. 2022 ) . However, the methods for lumped circuit design are not applicable to high-frequency circuits with distributed design. Automating high-frequency circuit design. In the distributed circuit designs ( Cao et al. 2009 ; Feng et al. 2017 ) , one state-of-the-art method is Circuit-GNN ( Zhang et al. 2019 ) , which first trains a graph neural network as a model to predict the transfer function s_{21} . Next, in inverse design exploration, Circuit-GNN produces circuits to meet desired transfer functions, using corresponding templates specifying topology and the number of resonators (corresponding to the target transfer functions in the test set). Recent work RLDFCDO ( Gao et al. 2024 ) picks one circuit from all candidate circuits and tunes the selected circuit using RL with proximal policy optimization (PPO) algorithm ( Schulman et al. 2017 ) . The methods are different from ours. They mainly aim to predict the EM properties of distributed circuits or rely on candidate template (i.e., topology types) to solve the inverse design problem. 3 Preliminary Figure 2 : Overview of DCIDA. A policy network (b) decodes constant input tensor {\bm{I}} (a) into conditional distributions (c). From these distributions, a batch of compound action {\mathcal{A}} (d) are sampled. These “raw” samples are mapped to their uniquely corresponding physical representations of circuits (f) through injective mapping functions (e). We examine the circuits (f) with an EM simulator, or neural estimator, (g) in order to obtain the corresponding transfer function for each design representation. These transfer functions’ L_{1} distances from target transfer function yield the rewards (h) used to update \theta according to 4.2.3 . The properties of a square resonator include length a\in\mathbb{R} , its center p=\{(x,y)|x\in\mathbb{R},y\in\mathbb{R}\} on a plane, and the direction of an open slit. The slit can face one of four directions: up, down, left, or right. We can use a one-hot vector u\in\{0,1\}^{4} to represents the direction of a slit, where a single entry is 1 and others are 0 . The position of a slit relates to a vector s\in\mathbb{R}^{4} , where the non-zero entry corresponds to the slit direction, and the remaining entries are zero. A distributed circuit with N resonators is defined by the tuple ({\bm{p}},{\bm{a}},{\bm{u}},{\bm{s}}) , where {\bm{p}}\in\mathbb{R}^{N\times 2} , {\bm{a}}\in\mathbb{R}^{N} , {\bm{u}}\in\{0,1\}^{N\times 4} and {\bm{s}}\in\mathbb{R}^{N\times 4} . An EM simulator evaluates the distributed circuit by its transfer function \hat{s}_{21}=\hat{{\mathcal{Y}}}({\bm{p}},{\bm{a}},{\bm{u}},{\bm{s}}) where \hat{{\mathcal{Y}}}\in\mathbb{C}^{m} and \mathbb{C} is a complex number. We compute the error \epsilon_{\text{db}} between \hat{{\mathcal{Y}}} and the target transfer function {\mathcal{Y}}\in\mathbb{C}^{m} in db domain \epsilon_{\text{db}}=\frac{20}{m}\sum^{m}_{i=1}|\log(|{\mathcal{Y}}_{i}|)-\log(|\hat{{\mathcal{Y}}}_{i}|)|. (1) For clarity in the paper, we focus on the distributed circuit design using square resonators as the blocks. However, our approach is general and can be used with other resonators. 4 Inverse Design of Distributed Circuit 4.1 Problem Formulation As illustrated in Figure 2 , the DCIDA framework {\mathcal{H}} has trainable parameters \theta and uses a constant variable {\bm{I}} as an input. Given the number of resonators N and a desired transfer function s_{21} without candidate templates, DCIDA learns to decode {\bm{I}} to generate a near-optimal distributed circuit design of form ({\bm{p}},{\bm{a}},{\bm{u}},{\bm{s}}) . The objective is to minimize the error \epsilon_{\text{db}} between the generated circuit’s transfer function \hat{s}_{21} and the desirable transfer function s_{21} . As noted earlier, we treat the generation of distribute circuits as a DSE problem. Since no intermediate error for partial designs can be computed, we formulate our DSE problem (with a definite fixed number of decisions) as a Single-Step Markov Decision Process (SSMDP), s_{0}\rightarrow{\mathcal{A}}\rightarrow s_{D} , in terms of compound action {\mathcal{A}}=\{a_{1},a_{2},...,a_{D}\} . Here, s_{0}\in{\mathcal{S}} and s_{D}\in{\mathcal{S}} represent the initial blank slate and the complete design (i.e., s_{D}=({\bm{p}},{\bm{a}},{\bm{u}},{\bm{s}}) ), respectively. Parameters \theta determining the sampling policy network \pi are tuned to minimize a statistical loss defined by our RL algorithm described in section 4.2.3 . \small\theta^{+}\sim\underset{\theta}{\text{min}}\hskip 9.24994pt{\mathcal{L}}(\theta)\hskip 9.24994pt\rightarrow\hskip 9.24994pt\pi_{\theta^{+}}\sim\pi_{\theta^{*}}\in\Pi^{*}({\mathcal{S}},{\mathcal{A}},R), (2) Here, the SSMDP is given by ({\mathcal{S}},{\mathcal{A}},R) , \Pi^{*} is the set of potential optimal policies in response to the SSMDP dynamics with the optimal \theta^{*} , and {\mathcal{L}} corresponds to the objective function related to penalty \epsilon_{\text{db}} and defined by our RL algorithm. Parameters \theta^{+} defines a near optimal policy. 4.2 Problem Solution 4.2.1 Learning Joint Distributions and Inter-Dependencies To solve the SSMDP problem, we introduce a RL framework using a policy network with parameters \theta to generate a sampling policy \pi_{\theta}=\pi_{\theta}(a_{1},...,a_{D}|{\bm{I}}) where {\bm{I}}\in\mathbb{R}^{d} is the initial state s_{0} with ones in tensor form. The s_{D} is terminal state achieved by a compound action {\mathcal{A}} sampled from \pi_{\theta} . Policy Network. Given N number of resonators, the policy network with {\bm{I}} produces a policy demonstrated by a jointly mixed actions distribution, which can be represented in the conditional form of probability density functions {\mathcal{F}} in Eq.( 3 ). \small\begin{split}\pi_{\theta}({\mathcal{A}}|{\bm{I}})&=\prod_{i=1}^{D}f_{i}(a_{i}|a_{i+1}\cdot\cdot\cdot a_{D};{\bm{I}},\theta),\end{split} (3) where we sample the i -th design action a_{i} from f_{i}\in{\mathcal{F}} . {\mathcal{F}} include beta distributions for sampling bounded continuous actions and categorical distributions for sampling discrete actions. We experiment with MLP and Transformer as policy networks in section 5 . The number of resonators N decides the number of actions D shown in section 4.2.2 . Figure 3 : The cascade of injective interdependent functions determines the centers of resonators within the area {\mathcal{P}}\subseteq{\mathcal{B}} . 4.2.2 Mapping Actions into Designs We design boundaries and formulate a series functions to map actions into physical representations ({\bm{p}},{\bm{a}},{\bm{u}},{\bm{s}}) within the boundaries in the circuit space {\mathcal{M}} . Slit position (u,s) and resonator length a . A discrete action a_{u}\in\{0,1,2,3\} specifies a slit’s direction, where 0 , 1 , 2 , and 3 correspond to up, left, down, and right, respectively. We use the action a_{u}\in\{0,1,2,3\} to decide a slit’s direction u\in\{0,1\}^{4} as follows, \small u(i)=\left\{\begin{array}[]{rcl}1,\hskip 9.24994pti=a_{u}\\ 0,\hskip 9.24994pti\neq a_{u}\end{array}\right. (4) After determining a slit’s direction, a continuous action a_{s}\in[0,1] decides the slit’s position s relative to the center of the corresponding edge: \small s(i)=\begin{cases}\frac{1}{8}\tanh(2a_{s}-1),&i=a_{u}\\ 0,&i\neq a_{u}\end{cases} (5) Since all resonators have the same length, a resonator’s length, ranging from L to 2L , is given by a=L\times(a_{l}+1) , where an action a_{l}\in[0,1] is the corresponding action. Resonator position p . Starting the DSE on a blank slate leads to an infinitely large exploration space, making precise placement of resonators inefficient. To address the issue, we propose the Theorem 4.1 to restrict exploration space within predefined boundaries. Proofs of the Theorems are shown on the Appendix A . Theorem 4.1 . In the circuit space {\mathcal{M}} , given the resonator length a , the maximum gap g=ag_{\text{max}} (with a predefined g_{\text{max}} ), and the number of resonators N , we define an area {\mathcal{P}}=\{(x,y)\mid x\in[0,B-a],y\in[\frac{a-B}{2},\frac{B-a}{2}]\} , where B=aN+g(N-1) , as the region bounding the centers of square resonators in a distributed circuit. The distributed circuit is further confined to the area {\mathcal{B}}=\{(x,y)\mid x\in[-\frac{a}{2},\frac{2B-a}{2}],y\in[-\frac{B}{2},\frac{B}{2}]\} , where {\mathcal{P}}\subseteq{\mathcal{B}} . The offsets about positions of resonators. To precisely determine the position of each resonator within the boundary {\mathcal{B}} , we set the default center for the leftmost (first) resonator as (0,0) . Subsequently, starting from the first resonator, we sequentially add remaining resonators from left to right based on the position of the last placed resonator. Using the actions a_{f}\in\{0,1,2\} , a_{us}\in[0,1] and a_{ug}\in[0,1] to adjust the offsets, we compute the shift factor as follows: \small f=\begin{cases}0,&a_{f}=0\\ 0.2,&a_{f}=1\\ 0.5,&a_{f}=2\end{cases} (6) The uniform shift factor is u_{s}=a_{us} , and the uniform gap factor is u_{g}=a_{ug} . We define the deviation function for the shift as d_{s}(l,r,u_{s})=u_{s}(r-l)+l , where l=0 and r=af . The deviation function for the gap is d_{g}(l,r,u_{g})=e^{u_{g}(\log{\frac{r}{l}})+l} , where l=ag_{\text{min}} and r=ag_{\text{max}} . Here, g_{\text{min}} and g_{\text{max}} are the predefined minimum and maximum ratio of the gap. The interdependent functions. We propose interdependent functions to accurately locate the position of each resonator by using the deviation functions. Starting with the default center of the first resonator, we define Definition 4.2 and Definition 4.3 to construct the functions. The functions map actions a^{(i)}_{x}\in[0,1] and a^{(i)}_{y}\in[0,1] for a resonator i to the point p=(x_{i},y_{i}) where p\in{\mathcal{P}} . (x_{i},y_{i}) is a center of the resonator i . An action a^{(i)}_{d}\in\{0,1,2\} corresponds to a relative position (i.e., up, down and right respectively) of the current resonator relative to the previous one. Since mapping the rightmost resonator differs from that of the other resonators, we denote the rightmost resonator as resonator n and remaining resonators (except the leftmost one) as resonator i , where 1\leq i