On the Comparison of Reinforcement Learning and Adaptive Control for Linear Systems under Packet Loss and Uncertainty Moh. Kamalul Wafi Abstract This paper presents a comparative study between Adaptive Quantized Control (AQC) and Deep Deterministic Policy Gradient (DDPG) reinforcement learning for uncertain linear systems with input quantization over communication channels subject to packet loss. The considered setting also includes dynamic switching from a nominal unstable system to a more unstable one during operation. The AQC is designed for unknown system dynamics using acknowledgment messages to compensate for packet losses, whereas the DDPG controller is trained using the nominal system model without acknowledgment messages. Numerical results show that the DDPG controller achieves faster transient responses and improved damping within its training environment. However, under model uncertainty, packet loss, and dynamic switching, the AQC consistently demonstrates superior robustness owing to its rigorous Lyapunov stability guarantees. These results highlight the trade-off between data-driven performance and model-based robustness, and provide insight into the applicability of reinforcement learning and adaptive control for networked uncertain systems. 1 Introduction Modern systems’ complexity has presented several challenges for classical and modern control techniques. However, there has been a recent trend toward utilizing learning-based control methods, particularly in high-dimensional spaces [ 10 ] . Over the past decade, deep learning has significantly impacted both the theoretical and practical aspects of machine learning by enabling its application to non-Euclidean spaces, such as graphs with interdependencies. This revolution has expanded the scope and capabilities of machine learning beyond traditional Euclidean spaces [ 23 ] . The integration of deep learning and reinforcement learning, known as deep reinforcement learning (DRL), has been the subject of extensive control-related studies [ 21 , 1 , 7 , 17 ] . The ideas behind DRL are similar to those of deep Q-network (DQN) as seen in [ 9 ] . The DQN is limited to discrete and finite actions whereas DRL, on the other hand, allows for continuous and infinite control actions, as described in [ 8 ] . In this work, we analyze the behavior of an unstable system under two different types of control: the deep deterministic policy gradient (DDPG) reinforcement learning, which is learning-oriented and the quantized control which is adaptive-oriented. Additionally, the term “quantization” in this context refers to the limitation of communication within a specific bandwidth in networked systems. The concept of using quantization for stabilization of linear systems with finite control signals and measurements was introduced in the reference [ 2 ] . The state information is quantized in a coarse manner, with the level of precision becoming finer as it approaches the origin in a logarithmic manner. This can also be alternatively explained using the more widely accepted sector-bounded quantizer [ 3 ] . Beyond that, dealing with the uncertain system, the adaptive control with input quantizer is studied by [ 4 ] which is also implemented into systems with packet loss \eta [ 12 ] . This means there exists a probability \bar{p} of control signal not being sent to the plant, with some variations of the non-linear uncertain system [ 5 ] . Similar communication and uncertainty issues also arise in networked estimation and process control applications, including hydraulic multi-tank systems, where reliable state estimation and feedback control are essential for maintaining system performance [ 18 , 22 ] . Note that with stabilization guaranteed [ 2 , 3 , 4 , 12 ] , it is interesting to see how the learning approach performs under some limitations [ 14 ] . The DDPG reinforcement learning is a combined deterministic-actor a=\mu_{\theta}(s) instead of the stochastic \pi_{\theta}(a|s)=\mathbb{P}[a|s;\theta] and the Q-value critic Q(s,a) which both applies the feed-forward neural network (FFNN) [ 8 , 13 ] . The stability is guaranteed as presented in [ 24 ] over uncertain systems with sector-bounded for the non-linear activation function. However, we design the dynamic changes in the simulation to see how robust the trained DRL control performs under specific trained system A considering the dynamic change beyond the environment A_{w} . The two dynamics encompass systems ranging from modestly unstable to highly unstable, as defined later. The remainder of this paper is organized as follows. Section 2 reviews the adaptive quantized control framework with acknowledgment messages and packet loss. Section 3 presents the actor–critic DDPG reinforcement learning controller and its quantized implementation. Numerical comparisons between the two approaches are reported in Section 4 , followed by concluding remarks in Section 5 . This article is an extended and refined version of our preliminary conference paper [ 19 ] . Figure 1: Adaptive quantized control ( K,v ) method with time-varying quantizer q(k,u(k)) . The solid arrows between blocks mean the connected information whereas the dashed arrow show there exists a possibility not being connected. The dashed arrows behind blocks deduces the time-varying. 2 Adaptive Quantized Control In this section, we briefly review the adaptive quantized control (AQC) framework proposed in [ 4 ] and extended to systems subject to packet loss in [ 12 ] . Consider the discrete-time uncertain linear system shown in Fig. 1 , where the quantized control signal is transmitted through a communication channel subject to packet loss. The plant is described by \displaystyle x(k+1)=Ax(k)+\eta(k)Bv(k), (1) where k\in\mathbb{N}_{0} is the discrete-time index and x(0)=x_{0} is the initial condition. The vector x(k)\in\mathbb{R}^{n} denotes the system state, v(k)\in\mathbb{R}^{m} is the quantized control input, A\in\mathbb{R}^{n\times n} is the unknown state matrix, and B\in\mathbb{R}^{n\times m} is the known input matrix. The binary random variable \eta(k)\in\{0,1\} models packet transmission, where \eta(k)=1 indicates that the control packet is successfully received by the plant, whereas \eta(k)=0 indicates packet loss. We assume that \{\eta(k)\}_{k\geq 0} are independent and satisfy \mathbb{P}\{\eta(k)=0\}\leq\bar{p},\qquad\text{for every }k, where \bar{p}\in[0,1) is an upper bound on the packet-loss probability. The quantized control input is generated from the nominal control law \displaystyle v(k)=q(k,u(k)), (2) where u(k)=H(k)x(k), and H(k)\in\mathbb{R}^{m\times n} denotes the time-varying feedback gain. The quantizer q(\cdot,\cdot) acts componentwise according to the logarithmic quantization rule \displaystyle q_{i}\triangleq\begin{cases}\varphi_{i}(k,j),&\textrm{if }u_{i}\in(\varphi_{i}(k,j+1),\varphi_{i}(k,j)],\\ -\varphi_{i}(k,j),&\textrm{if }u_{i}\in[-\varphi_{i}(k,j),-\varphi_{i}(k,j+1)),\\ 0,&\textrm{if }u_{i}=0,\end{cases}\qquad j\in\mathbb{I},\;i=1,\ldots,m, (3) where \varphi_{i}(k,j)=a_{i}(k)\rho_{i}^{j}(k),\qquad i=1,\ldots,m, with a_{i}(k)>0 and 0<\rho_{i}(k)<1 . Here, a_{i}(k) determines the quantization scale, whereas \rho_{i}(k) specifies the quantizer coarseness. The functions q_{i}(\cdot,\cdot) and u_{i}(\cdot) denote the i -th components of q(\cdot,\cdot) and u(\cdot) , respectively. For negative indices, the quantization levels are defined by \varphi_{i}(k,-j)\triangleq\frac{a_{i}(k)}{\rho_{i}^{j}(k)},\qquad i=1,\ldots,m. The logarithmic quantizer in ( 3 ) admits an equivalent representation as a time-varying sector-bounded memoryless nonlinearity [ 3 ] , \displaystyle\begin{aligned} \mathcal{Q}\triangleq&\Big\{q:\mathbb{N}_{0}\times\mathbb{R}^{m}\rightarrow\mathbb{R}^{m}:\;q(\cdot,0)=0,\\ &\left[q(k,u)-M_{1}(k)u\right]^{\top}\left[q(k,u)-M_{2}(k)u\right]\leq 0,\\ &u\in\mathbb{R}^{m},\;k\in\mathbb{N}_{0}\Big\}.\end{aligned} (4) Here, M_{1}\in\mathbb{R}^{m\times m} and M_{2}\in\mathbb{R}^{m\times m} denote diagonal matrices given by \displaystyle M_{1}(k) \displaystyle\triangleq\diag\!\left(M_{1_{1}}(k),\ldots,M_{1_{m}}(k)\right)\succ 0, \displaystyle M_{2}(k) \displaystyle\triangleq\diag\!\left(M_{2_{1}}(k),\ldots,M_{2_{m}}(k)\right)\succ 0, which satisfy M_{2}(k)-M_{1}(k)\succ 0. The sector bounds satisfy \rho_{i}(k)=\frac{M_{1_{i}}(k)}{M_{2_{i}}(k)},\qquad i=1,\ldots,m, and are illustrated in Fig. 2 . Equivalently, ( 4 ) can be written componentwise as \displaystyle M_{1_{i}}(k)u_{i}^{2}\leq q_{i}(k,u_{i})u_{i}\leq M_{2_{i}}(k)u_{i}^{2}, (5) for all u_{i}\in\mathbb{R} and k\in\mathbb{N}_{0} . Furthermore, \displaystyle\rho_{i}(k)=\frac{M_{1_{i}}(k)}{M_{2_{i}}(k)}=\frac{1-\beta\delta_{i}(k)}{1+\beta\delta_{i}(k)},\qquad i=1,\ldots,m, (6) where \delta(k)\triangleq\frac{1}{\beta}\left[M_{2}(k)+M_{1}(k)\right]^{-1}\left[M_{2}(k)-M_{1}(k)\right],\qquad\beta\neq 0. Hence, \delta(k) completely determines the quantizer coarseness. Defining \displaystyle\Delta(k) \displaystyle\triangleq\diag\!\left(\delta_{1}(k),\ldots,\delta_{m}(k)\right) \displaystyle=\frac{1}{\beta}\left[M_{2}(k)+M_{1}(k)\right]^{-1}\left[M_{2}(k)-M_{1}(k)\right], (7) the diagonal matrix \Delta(k) will be used in the adaptive controller design presented next. For completeness, we briefly summarize the adaptive quantized control framework and refer the reader to [ 4 , 12 ] for the complete controller design and stability analysis. Following [ 4 ] , the quantizer is decomposed into linear and nonlinear components as \displaystyle q(k,u)=\frac{1}{2}\left[M_{1}(k)+M_{2}(k)\right]u+q_{s}(k,u), (8) where q_{s}:\mathbb{N}_{0}\times\mathbb{R}^{m}\rightarrow\mathbb{R}^{m} denotes the nonlinear component satisfying \displaystyle\begin{aligned} \mathcal{Q}_{s}\triangleq&\Big\{q_{s}:\mathbb{N}_{0}\times\mathbb{R}^{m}\rightarrow\mathbb{R}^{m}:\;q_{s}(\cdot,0)=0,\\ &q_{s}^{\top}(k,u)q_{s}(k,u)-\frac{1}{4}u^{\top}\left[M_{2}(k)-M_{1}(k)\right]^{2}u\leq 0,\\ &u\in\mathbb{R}^{m},\;k\in\mathbb{N}_{0}\Big\}.\end{aligned} (9) For the subsequent stability analysis, we adopt the notion of Lyapunov stability in probability from [ 11 ] . Figure 2: Left: Logarithmic quantizer q for the scalar case ( m=1 ). Right: Example of the corresponding sector bounds with M_{1}(k)\equiv 1 and M_{2}(k)\in\{1+\hat{a}\mu_{i}^{j}:j\in\mathbb{I}\} . Definition 1 . Consider the stochastic discrete-time system \displaystyle x_{k+1}=f(x_{k},y_{k+1}),\qquad k\in\mathbb{N}_{0}, (10) where x_{k}\in\mathbb{R}^{n} and \{y_{k}:k\in\mathbb{N}_{0}\} is an \mathbb{R}^{d} -valued stochastic process defined on the probability space (\Omega,\mathbb{F},\mathbb{P}) . Here, \Omega denotes the sample space, \mathbb{F} the associated \sigma -algebra, and \mathbb{P}:\mathbb{F}\rightarrow[0,1] the probability measure. The stochastic process y_{k} is measurable and maps \Omega into the state space \Omega_{0}\subseteq\mathbb{R}^{d} . Define \mathbb{F}_{k}=\sigma(y_{1},\ldots,y_{k}),\qquad k\geq 1, with \mathbb{F}_{0}=\{\emptyset,\Omega\}, so that \{\mathbb{F}_{k}\}_{k\geq 0} forms an increasing filtration. The origin of ( 10 ) is said to be (i) stable in probability if \lim_{x_{0}\to 0}\mathbb{P}\left[\sup_{k\in\mathbb{N}}\|x_{k}\|>\epsilon\right]=0,\qquad\textrm{for every }\epsilon>0 (ii) asymptotically stable in probability if it is stable in probability and \lim_{x_{0}\to 0}\mathbb{P}\left[\lim_{k\to\infty}\|x_{k}\|=0\right]=1; (iii) exponentially stable in probability if there exists a constant \gamma>1 , independent of \omega , such that \lim_{x_{0}\to 0}\mathbb{P}\left[\lim_{k\to\infty}\|\gamma^{k}x_{k}\|=0\right]=1. For a set \mathcal{Q}\subseteq\mathbb{R}^{n} , the origin of ( 10 ) is said to be (i) locally (globally) almost surely stable in \mathcal{Q} if, for every initial condition x_{0}\in\mathcal{Q} (respectively, x_{0}\in\mathbb{R}^{n} ), the sample paths remain in \mathcal{Q} (respectively, \mathbb{R}^{n} ) for all k\geq 0 and converge to the origin almost surely; (ii) locally (globally) exponentially stable in \mathcal{Q} if it is locally (globally) almost surely stable and the convergence is exponential. The following lemma, adapted from [ 11 , 6 ] , recalls a sufficient condition for asymptotic convergence and Lyapunov stability in probability. Lemma 1 . Consider the stochastic system ( 10 ). Let \{x_{k}\}_{k\geq 0} be a Markov chain, and let V:\mathbb{R}^{n}\rightarrow\mathbb{R} be a positive definite Lyapunov function. For some \lambda>0 , define the set \mathcal{Q}_{\lambda}\triangleq\{x_{k}:0\leq V(x_{k})<\lambda\}. Suppose that \displaystyle\mathbb{E}[V(x_{k+1})]-V(x_{k})=-\vartheta(x_{k})\leq 0,\qquad\forall k, (11) where x_{k}\in\mathcal{Q}_{\lambda} and \vartheta(\cdot) is continuous. Then the following statements hold. (i) If x_{0}\in\mathcal{Q}_{\lambda} , then the sample paths remain in \mathcal{Q}_{\lambda} with probability at least 1-V(x_{0})/\lambda, the sequence \{V(x_{k})\} converges to a finite limit, and \lim_{k\to\infty}\vartheta(x_{k})=0 almost surely. (ii) Suppose that the conditions in (i) hold. If, for every \gamma>0 , there exists \delta>0 such that \vartheta(x_{k})\geq\delta, whenever \|x_{k}\|>\gamma and \vartheta(0)=0 , then the origin of ( 10 ) is globally almost surely stable. The adaptive quantized control law relies on acknowledgment messages that inform the controller whether the transmitted control packet is successfully received, i.e., the realization of \eta(k) . Following [ 12 ] , the time-varying nonlinear term q_{s}(k,u) is incorporated into the controller update. For completeness, we summarize the controller design procedure below before stating the corresponding stability result. 1. Select a positive definite matrix R\in\mathbb{R}^{n\times n} and a scalar \gamma\in(0,1) . 2. Solve the Riccati equation for P\geq I_{n} : \displaystyle P=\tilde{A}^{\top}P\tilde{A}+R-\tilde{A}^{\top}PB\left(B^{\top}PB\right)^{-1}B^{\top}P\tilde{A}. 3. Define A_{s}\triangleq\tilde{A}+BK_{g}^{\prime\prime}, where \tilde{A}\triangleq A+BK_{g}^{\prime},\qquad K_{g}^{\prime\prime}\triangleq-(B^{\top}PB)^{-1}B^{\top}P\tilde{A}. If (A,B) is stabilizable, then A_{s} is Hurwitz. Here, \tilde{A} is assumed to be known and unstable. 4. Select Q\in\mathbb{R}^{m\times m} and \varepsilon>0 such that 00, (14) for every k\in\mathbb{N}_{0} , together with the quantizer ( 2 ) and the gain update law \displaystyle K(k+1)=K(k) \displaystyle-\frac{\eta(k)}{1+x^{\top}(k)Px(k)}QB^{\dagger}\left[x(k+1)\right. \displaystyle\left.-A_{s}x(k)-Bq_{s}(k,u(k))\right]x^{\top}(k), (15) guarantees Lyapunov stability. In particular, the equilibrium (x(k),K(k))\equiv(0,K_{g}), where K_{g}\triangleq-(B^{\top}PB)^{-1}B^{\top}PA,\qquad K_{g}=K_{g}^{\prime}+K_{g}^{\prime\prime}, is asymptotically stable, and \lim_{k\to\infty}x(k)=0, for all x_{0}\in\mathbb{R}^{n}. Condition ( 14 ) must hold for every time step. Consequently, the sector bounds M_{1}(k) and M_{2}(k) are also required to vary with time. A simple choice is to fix M_{1}(k)\equiv I_{m}, and define M_{2}(k)\in\{1+\hat{a}\mu_{i}^{j}:j\in\mathbb{I}\}, where \hat{a}>0 and \mu_{i}>0 for every i . Since M_{2}(k)-M_{1}(k)\succ 0 , it follows that M_{2}(k)>I_{m} . Assume further that M_{2}(k) is uniformly bounded by a constant M_{\phi} , that is, M_{1}(k)\bar{p} ; otherwise, set \eta(k)\leftarrow 0 7: if k\geq t_{d} then \triangleright t_{d} denotes the switching time 8: (A,B)\leftarrow(A_{w},B_{w}) 9: end if 10: Choose \gamma\in(0,1) and compute \Delta(k) from ( 2 ) 11: if \delta\geq 1 then \triangleright enforce 0<\delta<1 12: M_{2}(k)\leftarrow M_{\phi} , \rho\leftarrow 1/\phi 13: else 14: for j_{i} , i=1,\ldots,n do \triangleright M_{1}\frac{1}{\beta}(M_{2}+M_{1})^{-1}(M_{2}-M_{1}) then 16: j\leftarrow j_{i} and stop 17: end if 18: end for 19: u(k)\leftarrow ( 13 ), v(k)\leftarrow ( 3 ) 20: end if 21: x(k+1)\leftarrow Ax(k)+\eta(k)Bv(k) 22: q_{s}(k,u(k))\leftarrow ( 8 ) 23: K(k+1)\leftarrow ( 1 ) 24: end for 3 Deep Deterministic Policy Gradient Reinforcement Learning We briefly review the deep deterministic policy gradient (DDPG) algorithm [ 8 , 13 ] , which serves as the reinforcement learning benchmark throughout this paper. Unlike stochastic policy methods, DDPG employs a deterministic policy u=\mu_{\theta}(x), which maps the state directly to the control action, instead of the stochastic policy \pi_{\theta}(u|x)=\mathbb{P}[u|x;\theta]. The state and action spaces are denoted by x\in\mathcal{S} and u\in\mathcal{A} , respectively. The learning problem is formulated as a Markov decision process (MDP) defined by the tuple (\mathcal{S},\mathcal{A},p,r) , where p_{1}(x_{1}) denotes the initial-state distribution and p(x_{t+1}|x_{t},u_{t})\coloneqq p(x_{t+1}|x_{1},u_{1},\ldots,x_{t},u_{t}) is the stationary state-transition probability. DDPG is an off-policy actor–critic algorithm in which the critic approximates the action-value function, while the actor learns the policy by maximizing the expected return. The objective is to maximize \displaystyle J(\mu_{\theta})=\int_{\mathcal{S}}\rho^{\mu}(x)\int_{\mathcal{A}}\mu_{\theta}(x,u)r(x,u)\,du\,dx, (16) where \rho^{\mu}(x) denotes the discounted state distribution induced by the policy \mu_{\theta} . The corresponding discounted return is \displaystyle r_{t}^{\gamma}=\sum\nolimits_{k=t}^{\infty}\gamma^{k-t}r(x_{k},u_{k}), (17) where \gamma\in(0,1) is the discount factor. An optimal policy \mu_{\theta}^{\ast} satisfies J(\mu_{\theta}^{\ast})\geq J(\mu_{\theta}) for every admissible policy \mu_{\theta} . Figure 3: Actor–critic DDPG controller with the time-varying quantizer q(k,u(k)) . The arrows have the same interpretation as those in Fig. 1 . DDPG is closely related to the deep Q-network (DQN), with the main difference being that DDPG is designed for continuous action spaces. The state–action value function Q^{\mu}(x,u) represents the expected return under the policy \mu_{\theta} . The corresponding optimal value function is denoted by Q^{\ast}(x,u) , from which the optimal policy is obtained as u^{\ast}(x)=\arg\max_{u}Q^{\ast}(x,u). Unlike DQN, which searches over a finite action set, DDPG assumes that the action space is continuous and that Q^{\ast}(x,u) is differentiable with respect to the action variable. Consequently, the maximization \max_{u}Q(x,u) is approximated by Q(x,\mu_{\theta}(x)), allowing the policy to be updated using gradient-based optimization. The corresponding Bellman equation is \displaystyle Q^{\ast}(x,u)=\mathbb{E}_{x^{\prime}\sim\mathbb{P}}\left[r(x,u)+\gamma\max_{u^{\prime}}Q^{\ast}(x^{\prime},u^{\prime})\right], (18) where x^{\prime}\sim\mathbb{P} denotes the next state sampled from the transition probability \mathbb{P}(\cdot|x,u) . In practice, the action–value function is approximated by a neural network Q_{\phi}(x,u) with parameters \phi . The network is trained using a replay buffer \Phi=\{(x,u,r,x^{\prime},d)\}, where d indicates whether x^{\prime} is a terminal state. The target value and the corresponding mean squared Bellman error (MSBE) are given by \displaystyle\begin{aligned} \xi_{\Phi}&=r+\gamma(1-d)\max_{u^{\prime}}Q^{\ast}(x^{\prime},u^{\prime}),\\ L(\phi,\Phi)&=\mathbb{E}_{x,u,r,x^{\prime},d\sim\Phi}\!\left[\left(Q_{\phi}(x,u)-\xi_{\Phi}\right)^{2}\right].\end{aligned} (19) The reward is designed as a function of the tracking error e\triangleq x . Introducing a small positive constant \psi to avoid division by zero and a threshold T_{h} , the reward function is defined as \displaystyle r(k)=\begin{cases}-100,&|x|>T_{h},\\[2.84526pt] \dfrac{1}{e+\psi},&\text{otherwise}.\end{cases} (20) Recall the uncertain linear system ( 1 ) together with the DDPG controller shown in Fig. 3 . The state x(k)\in\mathbb{R}^{n_{p}} is also taken as the measured output, i.e., y(k)\coloneqq x(k), while u(k)\in\mathbb{R}^{n_{u}} denotes the control input. The DDPG controller consists of two feed-forward neural networks (FFNNs): an actor network that maps the measured state to the control input, and a critic network that estimates the corresponding action-value function. As illustrated in Fig. 4 , the actor consists of \ell_{a} hidden layers, whereas the critic receives both the state and action as inputs. The state and action paths comprise \ell_{co} and \ell_{ca} layers, respectively, followed by \ell_{cs} common layers. Consequently, the total number of critic layers is \ell_{c}\triangleq\max(\ell_{co},\ell_{ca})+\ell_{cs}. The actor network is described by \displaystyle\phi_{0}(k) \displaystyle=x(k), (21a) \displaystyle\phi_{i}(k) \displaystyle=\Delta_{i}\!\left(W_{i}\phi_{i-1}(k)+b_{i}\right)\triangleq\Delta_{i}(v_{i}(k)),\qquad i=1,\ldots,\ell_{a}, (21b) \displaystyle u^{(n_{u})}(k) \displaystyle=W_{\ell_{a}+1}\phi_{\ell_{a}}(k)+b_{\ell_{a}+1}\triangleq v_{\ell_{a}+1}^{(n_{u})}. (21c) Figure 4: Architecture of the actor–critic DDPG controller. For the i -th hidden layer, W_{i}\in\mathbb{R}^{m_{i}\times m_{i-1}} and b_{i}\in\mathbb{R}^{m_{i}} denote the weight matrix and bias vector, respectively, where m_{0}=n_{p} . The corresponding pre-activation vector is given by \displaystyle v_{i}^{(j)}(k)=\sum_{t=1}^{m_{i-1}}W_{i}^{(j,t)}\phi_{i-1}^{(t)}(k)+b_{i}^{(j)},\qquad j=1,\ldots,m_{i}. (22) The activation function \Delta_{i} is applied elementwise according to \Delta_{i}(v_{i})=\bigl[\lambda(v_{1}),\ldots,\lambda(v_{m_{i}})\bigr]^{\top}, where \lambda(\cdot) denotes either the hyperbolic tangent or the ReLU activation function. The output layer has dimension n_{u} , matching the dimension of the control input. For subsequent analysis, define v_{q}=\begin{bmatrix}v_{1}^{\top}&\cdots&v_{\ell_{a}}^{\top}\end{bmatrix}^{\top},\qquad\phi_{q}=\begin{bmatrix}\phi_{1}^{\top}&\cdots&\phi_{\ell_{a}}^{\top}\end{bmatrix}^{\top}, where n_{q}=\sum\nolimits_{i=1}^{\ell_{a}}n_{i}. The vectors v_{q} and \phi_{q} collect the pre-activation and post-activation variables of all hidden layers, respectively, and the resulting control signal u(k) is applied to the plant and subsequently supplied to the critic network. The actor network admits the compact input–output representation \displaystyle\begin{bmatrix}u(k)\\ v_{q}(k)\end{bmatrix}=N\begin{bmatrix}x(k)\\ \phi_{q}(k)\\ 1\end{bmatrix},\qquad\phi_{q}(k)=\Delta(v_{q}(k)), (23) where N is constructed from the weight matrices and bias vectors of the actor network, namely W_{q}\in\mathbb{R}^{n_{q}+1} and b_{q}\in\mathbb{R}^{n_{q}+1} . Specifically, \displaystyle N\triangleq\left[\begin{array}[]{c|cccc|c}0&0&0&\cdots&W_{\ell_{a}+1}&b_{\ell_{a}+1}\\ \hline\cr W_{1}&0&\cdots&0&0&b_{1}\\ 0&W_{2}&\cdots&0&0&b_{2}\\ \vdots&\vdots&\ddots&\vdots&\vdots&\vdots\\ 0&0&\cdots&W_{\ell_{a}}&0&b_{\ell_{a}}\end{array}\right]. The representation ( 23 ) separates the linear and nonlinear components of the FFNN, whose stability properties are studied in [ 25 ] . The critic network is constructed in the same manner as the actor network and is given by \displaystyle\begin{aligned} \varphi_{0}(t)&=\begin{bmatrix}u(t)\\ x(t)\end{bmatrix},\\[2.84526pt] \varphi_{i}^{u}(t)&=\Delta_{i}^{u}\left(W_{i}^{u}\varphi_{i-1}^{u}(t)+b_{i}^{u}\right),\qquad i=1,\ldots,\ell_{ca},\\[2.84526pt] \varphi_{i}^{x}(t)&=\Delta_{i}^{x}\left(W_{i}^{x}\varphi_{i-1}^{x}(t)+b_{i}^{x}\right),\qquad i=1,\ldots,\ell_{co},\\[2.84526pt] \varphi_{\ell_{0}}(t)&=\varphi_{\ell_{ca}}^{u}(t)+\varphi_{\ell_{co}}^{x}(t),\qquad\ell_{0}\coloneqq\max(\ell_{ca},\ell_{co}),\\[2.84526pt] \varphi_{j}(t)&=\Delta_{j}\left(W_{j}\varphi_{j-1}(t)+b_{j}\right),\qquad j=\ell_{0}+1,\ldots,\ell_{0}+\ell_{cs},\\[2.84526pt] r_{c}(t)&=W_{\ell_{0}+\ell_{cs}+1}\varphi_{\ell_{0}+\ell_{cs}}(t)+b_{\ell_{0}+\ell_{cs}+1}\coloneqq v_{\ell_{0}+\ell_{cs}+1}\in\mathbb{R}^{r_{c}}.\end{aligned} (29) where the notation follows directly from the actor network in ( 21 )–( 3 ). To distinguish the reinforcement learning control input from the adaptive quantized controller, we denote the actor output by u_{r}(k)\coloneqq u(k). Suppose that (x^{\ast},u_{r}^{\ast}) is an equilibrium of the closed-loop system. Propagating the equilibrium state through the actor and critic networks yields the corresponding equilibrium values (v^{\ast},\phi^{\ast}) of the activation inputs and outputs. Consequently, (x^{\ast},u_{r}^{\ast},v^{\ast},\phi^{\ast}) is an equilibrium of the interconnected system consisting of ( 1 ), ( 21 ), and ( 29 ), provided that x^{\ast}=Ax^{\ast}+Bq(k,u_{r}^{\ast}). Algorithm 2 summarizes the proposed reinforcement learning controller with the time-varying quantizer. During training, the controller is learned using the nominal system matrix A . The learned policy is subsequently evaluated under plant variations by replacing A with A_{w} and introducing packet loss. This setting provides a direct comparison with the adaptive quantized controller, which is designed for uncertain systems through the stabilizing gain K_{g} satisfying A_{s}=A+BK_{g}. Algorithm 2 Reinforcement Learning with Quantizer 1: A,B,x_{0},k_{\rm on},R\coloneqq I_{2}\rightarrow P\geq I_{2} , FFNN parameters \psi , learning rates \tau_{a},\tau_{c} , discount factor \gamma , sampling time t_{s} , number of episodes \mathcal{E} , and terminal time \mathcal{T} 2: for k=1:t_{\max} do 3: Observe the state x(k) 4: Compute the actor output u_{r}(k) 5: Observe the reward r(k) 6: if k\geq t_{d} then \triangleright t_{d} denotes the switching time 7: (A,B)\leftarrow(A_{w},B_{w}) 8: end if 9: Choose \gamma\in(0,1) 10: Compute \Delta(k) from ( 2 ) with K(k)x(k) replaced by u_{r}(k) 11: Compute the quantized input v(k)=q(k,u_{r}(k)) using ( 3 ) 12: Update the state: x(k+1)=Ax(k)+\eta(k)Bv(k) 13: end for 4 Numerical Results and Findings In this section, we compare the performance of the adaptive quantized control (AQC) and deep deterministic policy gradient (DDPG) controllers under four simulation scenarios: (i) AQC without packet loss; (ii) AQC with packet loss satisfying \bar{p}_{1}=0.15 , \bar{p}_{2}=0.30 , and \bar{p}_{3}>0.5 ; (iii) DDPG with the input quantizer; (iv) DDPG with the input quantizer under the same packet-loss conditions as in (ii). For each scenario, the system is switched from the nominal dynamics A to the perturbed dynamics A_{w} at t_{d}\in\{10,15,20,25,30\}\ {\rm s}, while the controller is activated at k_{\rm on}=1\ {\rm s}. Consider the uncertain second-order system subject to packet loss, \displaystyle\begin{aligned} z(k+2)+\beta_{\zeta}z(k+1)+\alpha_{\zeta}z(k)&=\eta(k)b_{\zeta}v(k),\\ z(0)&=z_{0},\qquad z(1)=z_{1},\end{aligned} (30) where k\in\mathbb{N}_{0} . The parameters \alpha_{\zeta},\beta_{\zeta}\in\mathbb{R} , \zeta\in\{1,2\} , are unknown system coefficients, while b_{\zeta}\in\mathbb{R} is a known input gain. The control input is denoted by v(k)\in\mathbb{R} and is generated through the quantizer described in Section 2 . Packet transmission is governed by the binary random variable \eta(k) satisfying \mathbb{P}\{\eta(k)=0\}\leq\bar{p}_{i}, where \bar{p}_{i} denotes the packet-loss bound. Defining the state variables x_{1}(k)=z(k) and x_{2}(k)=z(k+1) . The nominal and perturbed system matrices are A=\begin{bmatrix}0&1\\ -\alpha_{1}&-\beta_{1}\end{bmatrix},\qquad A_{w}=\begin{bmatrix}0&1\\ -\alpha_{2}&-\beta_{2}\end{bmatrix}, with x=[x_{1},x_{2}]^{\top} , x(0)=x_{0} , and B=[0,b_{\zeta}]^{\top},\forall\zeta . For the AQC design, we choose R=I_{2}, which yields the Riccati solution P=\diag(1,2)>I_{2}. The system parameters are selected as \alpha_{1}=1.5,\qquad\alpha_{2}=2,\qquad\beta_{1}=\beta_{2}=-0.5,\qquad b_{\zeta}=0.4. Since only the first state is quantized, we fix M_{1}(k)\equiv 1, and choose M_{2}(k)\in\left\{1+3(1.3)^{j}:j\in\mathbb{I}\right\}. The index j determines the time-varying quantization level, satisfying 1=M_{1}