Personalized Multi-Agent Average Reward TD-Learning via Joint Linear Approximation Leo Muxing Wang, Pengkun Yang, Lili Su Abstract We study personalized multi-agent average reward TD learning, in which a collection of agents interacts with different environments and jointly learns their respective value functions. We focus on the setting where there exists a shared linear representation and the agents’ optimal weights collectively lie in an unknown linear subspace. Inspired by the recent success of personalized federated learning (PFL), we study the convergence of cooperative single-timescale TD learning in which agents iteratively estimate the common subspace and local heads. We showed that this decomposition can filter out conflicting signals, effectively mitigating the negative impacts of “misaligned” signals, and achieve linear speedup. The main technical challenges lie in the heterogeneity, the Markovian sampling, and their intricate interplay in shaping error evolutions. Specifically, not only are the error dynamics of multiple variables are closely interconnected, but there is also no direct contraction for the principal angle distance between the optimal subspace and the estimated subspace. We hope our analytical techniques can be useful to inspire research on deeper exploration into leveraging common structures. Experiments are provided to show the benefits of learning via a shared structure to the more general control problem. 1 Introduction In many real-world applications, ranging from simple assistive robotics to autonomous vehicles, autonomous agents operate in different local environments. For example, consider the simple robot vacuums. Their interacting environments are mostly shaped by the household conditions, which can vary significantly in floor plans, obstacle types (e.g., moving humans and furniture), and spatial configurations. Similarly, autonomous vehicles deployed across different regions of a metropolitan area may encounter substantial variability in road conditions and traffic patterns. Such environmental heterogeneity can lead to misaligned learning signals across agents, which may significantly hinder convergence and impair generalization of joint learning. Nevertheless, applying standard single-agent reinforcement learning (RL) solutions may lead to overall intensive, yet possibly redundant, computation and sample collection. This is because the cumulative knowledge obtained by one agent may still be partially useful to others when some common structure (though unknown) exists. These two competing intuitions raise a natural question: Question: Do the benefits of collaboration outweigh the drawbacks – or is it the other way around? Existing studies of multi-agent RL (MARL) with heterogeneous environments are mostly restricted to training a common policy or value-/Q-function ( 18 ; 37 ; 41 ; 47 ) , and the optimality is often described with respect to some imaginary environment constructed as a weighted average of the agents’ state-transition kernels ( 18 ) . Though this approach has practical value – particularly when environmental heterogeneity is mild – in many applications (such as on-device recommender systems ( 22 ) and smart healthcare ( 23 ) ), heterogeneity is often moderate to severe, which can significantly degrade the performance of a common policy. In a parallel line of research on multi-agent Markov games or competitive MARL, learning personalized (or localized) policies or value-/Q-functions has recently garnered considerable attention ( 14 ; 46 ; 27 ; 48 ; 9 ) . Specifically, in their setting, agents interact with one another within a shared environment, and the rewards received by individual agents are determined by the joint actions of all agents. It is easy to see that in such a setup, the more agents there are, the more complicated an agent’s decision problem becomes. In fact, this line of work often suffers from the so-called curse of multi-agents—the sample complexity increases linearly or even exponentially ( 29 ; 17 ; 46 ) with respect to the number of agents. In this paper, we take a step toward addressing the above question by studying cooperative temporal-difference (TD) learning, where a collection of agents collaboratively learn their heterogeneous value functions through local TD updates. Inspired by the recent success of personalized federated learning (PFL) and multi-task learning, we focus on the specific underlying common structure that the agents’ optimal weights under a shared linear representation collectively lie in a low-dimensional linear subspace ( 35 ; 8 ; 33 ; 12 ; 11 ; 34 ; 24 ) . Our contributions can be summarized as • We propose and analyze the convergence of a cooperative average-reward TD method in which agents iteratively estimate the common subspace and agent-specific heads. We show that the overall reward estimate errors decay at a rate of \widetilde{{\mathcal{O}}}(1/T) , where \widetilde{{\mathcal{O}}} hides polylog factors of T, with a faster stepsize. Our dynamics of common subspace {\mathbf{B}} and local heads \omega^{k} are single timescale, i.e., their respective stepsizes are on the same order with respect to T . We show that, when T is sufficiently large, under additional conditions, the joint estimation errors of the subspace and the local heads converge to 0 at a rate \widetilde{\mathcal{O}}(\frac{1}{\sqrt{TLK}}) , where L is the number of local steps and K is the number of agents. • Our analysis is highly nontrivial, which may be insightful for a broader audience dealing with coupled heterogeneous dynamics. Technically, the Markovian sample of TD error makes it difficult to obtain a direct contraction for the principal angle distance between the optimal subspace and the estimated subspace, but only an indirect subtraction in terms of the local weights error. To address this challenge, we show that the local weights error can be lower-bounded by the principal angle distance times a constant that depends on the diversity of the optimal local weights. • We empirically validate that the proposed personalized framework achieves superior performance compared to other frameworks, including single-agent, universal-policy, and two-timescale approaches, with respect to convergence speed, stability, and generalization. During the preparation of this paper, ( 42 ) studied discounted TD learning with two timescales and established a linear convergence speedup. Key differences are as follows: We study average-reward TD and we study the single timescale dynamics of the local heads and shared common spaces. However, under their notation, the validity of the error contraction in Eq. (54) appears to rely on the assumptions D_{1}=o(\beta_{t}) and D_{2}=o(\alpha_{t+1}) . Since both terms involve stepsize ratios such as \frac{\beta_{t}}{\alpha_{t}} and \frac{\beta_{t}}{\alpha_{t+1}} , these conditions implicitly require a sufficiently strong separation between the two stepsize sequences. While such assumptions are standard in two-timescale analyses, they are not automatically guaranteed for commonly used polynomial stepsizes and may require additional verification. In contrast, our analysis establishes error contraction under a single-timescale setting and directly controls the coupled error terms through a unified Lyapunov argument, without relying on such asymptotic separation conditions. 2 Related Work MARL under heterogeneous environments. There is an emerging interest in mathematically understanding the role of environmental heterogeneity in the performance of the federated versions of classic reinforcement learning algorithms ( 18 ; 37 ; 41 ; 47 ) , yet most existing studies focus on training a common policy or approximating a common function. ( 18 ) studied federated Q-learning and policy gradient methods, assuming known transition kernels. ( 38 ) studied federated Q-learning with unknown transition kernels, and found that, in the presence of environmental heterogeneity, the eventual convergence rate may depend solely on the number of communication rounds – multiple updates within each round cannot accelerate convergence. ( 37 ) proposed FedTD(0) with linear function approximation, and linear convergence speedup in a low-heterogeneity regime. ( 41 ) used the KL-divergence to penalize the deviation of local update from the global policy, and proved that under the setting of heterogeneous environments, the local update is beneficial for global convergence using their method. Yet, no convergence rate is provided. ( 45 ) proposed FedSARSA and proved that the algorithm can converge to a near-optimal solution. Neither ( 41 ) nor ( 45 ) characterized sample complexity. Personalization in heterogeneous environments. Training personalized policies in cooperative multi-agent systems in the presence of environment heterogeneity remains largely underexplored. ( 44 ) studied the training of personalized policies in the context where each agent has a private reward function but shares a common transition kernel for the environment. The analysis focused solely on the convergence of the global policy, defined as the average of the local policies. ( 18 ) proposed a heuristic deep FedRL method, where the policy network can be decomposed into a shared subnetwork and environment embedding layer. ( 42 ) studied personalized TD learning with two timescales and established a linear convergence speedup. 3 Problem Setup and Preliminaries Multi-agent Reinforcement Learning. We consider a multi-agent system that consists of one parameter server and K agents. The agents are modelled as Markov Decision Processes (MDP) \{{\mathcal{M}}^{k}\mid{\mathcal{M}}^{k}=\text{ for }k=1,2,...,K\} , where {\mathcal{S}} is the (possibly infinite) state space, {\mathcal{A}} is a finite action space, P^{k} ’s are the transition kernels, and R is the reward function: {\mathcal{S}}\times{\mathcal{A}}\rightarrow[-U_{r},U_{r}] . The environments P^{k} with which the agents interact are heterogeneous. The parameter server can orchestrate the learning at the agents through iteratively aggregating the updates at the agents. We study average-reward reinforcement learning ( 36 ) , which provides a principled framework for maximizing the mean reward. Even in the single-agent setting, unlike discounted and finite-horizon MDPs, the sample complexity of average-reward settings remains largely unexplored ( 16 ) . In this paper, we consider the more challenging multi-agent setting. In particular, we study personalized multi-agent average-reward TD learning. Fix a policy \pi . We assume that, for each agent k\in[K] , the limiting state distribution induced by the transition kernel P^{k} under policy \pi exists; we denote this distribution by \mu^{k} . The time-average expected reward at agent k is defined as: \displaystyle J^{k}(\pi) \displaystyle:=\lim_{T\rightarrow\infty}\frac{1}{T}{\mathbb{E}}\left[\sum_{t=0}^{T-1}R(s^{k}_{t},a^{k}_{t})\right] (1) \displaystyle=\underset{(s,a)\sim\mu^{k}\otimes\pi}{\mathbb{E}}[R(s,a)], where (s^{k}_{t},a^{k}_{t}) is the state-action pair of agent k at iteration t , for t=0,1,\cdots . The first expectation in Eq. ( 1 ) is taken over the state-action trajectories generated under the Markov chain, while the second is taken over their limiting distribution. Here, the notation \mu^{k}\otimes\pi denotes sampling s\sim\mu^{k} and a\sim\pi(\cdot\mid s) . The value function is defined as \displaystyle V^{k,\pi}(s)={\mathbb{E}}\left[\sum_{t=0}^{\infty}R(s^{k}_{t},a^{k}_{t})-J^{k}(\pi)\mid s_{0}=s\right]. (2) For ease of exposition, we drop the index and function argument in \pi . That is, for each k , J^{k}(\pi) is written as J^{k} , and V^{k,\pi}(s) as V^{k}(s) for short. Single-agent TD Learning. Let {\mathcal{V}} be a function space that V^{k} belongs to. Let \phi:{\mathcal{S}}\to L^{2}({\mathbb{N}}) be a feature map of {\mathcal{V}} . Denote the coefficients of the value V^{k} with respect to \phi as z^{k,*} , where z^{k,*}\in L^{2}({\mathbb{N}}) . For simplicity, we assume \phi is known and has finite dimension d , i.e., \phi:{\mathcal{S}}\to{\mathbb{R}}^{d} . Consequently, V^{k}(s)=\left\langle\phi(s),z^{k,*}\right\rangle,~~~~\forall~s\in{\mathcal{S}},\text{and }k\in[K]. We consider the following linear function approximation: \widehat{V}(s;z^{k})=\phi(s)^{\top}z^{k}. To drive the approximation \widehat{V}(s;z^{k}) to V^{k} , single-agent TD learning (particularly TD(0)) update is applied at an agent to learn its value function: \displaystyle z^{k}_{t+1} \displaystyle=z^{k}_{t}+\beta\bigg(R(a_{t}^{k},s_{t}^{k})-\eta_{t}^{k} (3) \displaystyle+\left(\phi(s^{k}_{t+1})-\phi(s^{k}_{t})\right)^{\top}z^{k}_{t}\bigg)\phi(s_{t}^{k}), \displaystyle\eta_{t+1}^{k} \displaystyle=\eta_{t}^{k}+\gamma\left(R(a_{t}^{k},s_{t}^{k})-\eta_{t}^{k}\right), where \beta and \gamma are stepsizes, and \eta_{t}^{k} is the reward estimator. In literature, \delta_{t}^{k}:=R(a_{t}^{k},s_{t}^{k})-\eta_{t}^{k}+\left(\phi(s^{k}_{t+1})-\phi(s^{k}_{t})\right)^{\top}z^{k}_{t} is often referred to as TD(0) error. It is well-known that under some standard technical assumptions \lim_{t\to\infty}z^{k}_{t}\to z^{k,*} ( 36 ; 30 ) . Multi-agent TD Learning: Commonality and Heterogeneity. In multi-agent TD learning, the K agents aim to collaboratively learn their respective value functions under the given \pi based on their collective state-action trajectories. Learning with shared representation is widely recognized as an effective way to separate commonalities from heterogeneity across various heterogeneous data ( 4 ; 1 ; 2 ; 20 ; 24 ) . Inspired by the literature of personalized FL and multi-task learning, following ( 35 ; 8 ; 33 ; 12 ; 11 ; 34 ; 24 ) , we assume that \{z^{k,*}:~k=1,\cdots,K\} belongs to and fully span an r -dimensional subspace of {\mathbb{R}}^{d} . That is, there exists an orthonormal matrix \mathbf{B}^{*}\in{\mathbb{R}}^{d\times r} such that \displaystyle z^{k,*}=\mathbf{B}^{*}\omega^{k,*},~~~\text{where}~~\omega^{k,*}\in{\mathbb{R}}^{r}. (4) To see the generality of this formulation: r=1 when z^{k,*}=z^{k^{\prime},*} for all k,k^{\prime} (i.e., homogeneous transitional kernels P^{k} ); r=K when z^{k,*}\perp z^{k^{\prime},*} for all k^{\prime}\not=k – which is possible when K\leq d . 4 Algorithm: PMAAR-TD We propose a personalized multi-agent average-reward TD learning algorithm ( PMAAR-TD ) based on joint linear approximation, in which agents iteratively estimate the underlying truth subspace {\mathbf{B}}^{*} and their respective local “heads” \omega^{k,*} . Specifically, \{z^{k}={\mathbf{B}}\omega^{k}\}_{k\in[K]} are decomposed into the product of the subspace estimate {\mathbf{B}}\in{\mathbb{R}}^{d\times r} , which is common over all agents, and an agent-specific head \omega\in{\mathbb{R}}^{r} . A formal description can be found in Algorithm 1 . At a high level, in each iteration, each agent locally performs the TD( L ) update, and uses the obtained TD( L ) error \delta_{t,L}^{k} – formally defined in Eq. ( 6 ) – and rewards to update estimates of its local head \omega^{k} , common subspace {\mathbf{B}} , and local reward \eta^{k} . The updates of \omega^{k} and {\mathbf{B}} can be interpreted as performing a perturbed stochastic gradient on the following joint linear approximation problem: \displaystyle\min_{{\mathbf{B}},\omega^{1},\cdots,\omega^{K}}~\frac{1}{2}\sum_{k=1}^{K}\mathbb{E}_{\mu^{k}}[(\widehat{V}^{k}(s;\omega^{k},\mathbf{B})-V^{k}(s))^{2}], (5) \displaystyle\text{where}~~\widehat{V}^{k}(s;\omega^{k},\mathbf{B})=\phi(s)^{\top}\mathbf{B}{\omega^{k}}. A natural stochastic gradient descent of Eq. ( 5 ) updates of {\mathbf{B}} and \omega^{k} for all k are: \displaystyle\omega_{t+1}^{k}=\omega_{t}^{k}+\beta\left[V^{k}(s_{t,0}^{k})-\phi(s_{t,0}^{k})^{\top}{\mathbf{B}_{t}}{\omega_{t}^{k}}\right]{\mathbf{B}_{t}}^{\top}\phi(s_{t,0}^{k}), \displaystyle\mathbf{B}_{t+1}^{k}=\mathbf{B}_{t}^{k}+\zeta\left[V^{k}(s_{t,0}^{k})-\phi(s_{t,0}^{k})^{\top}{\mathbf{B}_{t}}{\omega_{t}^{k}}\right]\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}, where \beta and \zeta are stepsizes of \omega^{k} and {\mathbf{B}} , respectively. However, V^{k} is unknown for each k . Following the semi-gradient method TD( L ), we use \displaystyle R(s_{t,0}^{k},a_{t,0}^{k})-J^{k}+R(s_{t,1}^{k},a_{t,1}^{k})-J^{k}+... \displaystyle\qquad+R(s_{t,L-1}^{k},a_{t,L-1}^{k})-J^{k}+\widehat{V}^{k}(s_{t,L}^{k};\omega_{t}^{k},\mathbf{B}_{t}) \displaystyle= \displaystyle\sum_{\ell=0}^{L-1}R(s_{t,\ell}^{k},a_{t,\ell}^{k})-J^{k}+\phi(s_{t,L}^{k})^{\top}{\mathbf{B}_{t}}{\omega_{t}^{k}} to estimate V^{k}(s_{t,0}^{k}) . The L -step TD error at each agent is \displaystyle\delta_{t,L}^{k}=\sum_{\ell=0}^{L-1}r_{t,\ell}^{k}-\eta_{t}^{k}+(\phi(s_{t,L}^{k})-\phi(s_{t,0}^{k}))^{\top}{\mathbf{B}_{t}}{\omega_{t}^{k}}, (6) where r_{t,\ell}^{k}=R(s_{t,\ell}^{k},a_{t,\ell}^{k}) . Thus, \displaystyle\widetilde{\omega}_{t+1}^{k}=\omega_{t}^{k}+\beta\delta_{t,L}^{k}{\mathbf{B}_{t}}^{\top}\phi(s_{t,0}^{k}), (7) \displaystyle\mathbf{B}_{t+1}^{k}=\mathbf{B}_{t}+\zeta{\mathbf{B}}_{t,\perp}{\mathbf{B}}_{t,\perp}^{\top}\delta_{t,L}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}. (8) Algorithm 1 PMAAR-TD 1: Input: Initial heads \{\omega_{0}^{k}\}_{k=1}^{K} ; initial subspace {\mathbf{B}}_{0}\in{\mathbb{R}}^{d\times r} , which is orthonormal; three stepsizes: \beta for local heads, \zeta for subspace, and \gamma for reward estimator; total iteration T ; projection upper bound U_{\omega} such that \|\omega^{k,*}\|\leq U_{\omega} for k\in[K] . 2: for k=1,2,\dots,K do 3: \eta_{0}^{k}=0 4: Draw s_{0,0}^{k} arbitrarily from {\mathcal{S}} 5: end for 6: for t=0,1,2,\dots,T-1 do 7: for k=1,2,\dots,K do 8: \delta_{t,L}^{k}=\sum_{\ell=0}^{L-1}(r_{t,\ell}^{k}-\eta_{t}^{k})+(\phi(s_{t,L}^{k})-\phi(s_{t,0}^{k}))^{\top}\mathbf{B}_{t}\omega_{t}^{k} . 9: \omega^{k}_{t+1}=\Pi_{U_{\omega}}\left(\omega^{k}_{t}+\beta\frac{1}{L}\delta_{t,L}^{k}\mathbf{B}_{t}^{\top}\phi(s_{t,0}^{k})\right) . 10: \mathbf{B}_{t+1}^{k}=\mathbf{B}_{t}+\zeta\mathbf{B}_{t,\perp}\mathbf{B}_{t,\perp}^{\top}\frac{1}{L}\delta_{t,L}^{k}\phi(s_{t,0}^{k})(\omega^{k}_{t})^{\top} . 11: \eta^{k}_{t+1}=\eta^{k}_{t}+\gamma\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{t,\ell}^{k}-\eta^{k}_{t}\right) . 12: s_{t+1,0}^{k}=s_{t,L}^{k} . 13: end for 14: \bar{\mathbf{B}}_{t+1}=\frac{1}{K}\sum_{k=1}^{K}\mathbf{B}_{t+1}^{k} . 15: \mathbf{B}_{t+1},\mathbf{R}_{t+1}=\texttt{QR}(\bar{\mathbf{B}}_{t+1}) . 16: Server broadcasts \mathbf{B}_{t+1} to all agents. 17: end for Nevertheless, Eq. ( 7 ) and Eq. ( 8 ) are not the final updates to obtain \omega_{t+1}^{k} and {\mathbf{B}}_{t+1} . Several key algorithmic components are incorporated to provide fine-grained control of the perturbation and to enforce contraction of the principal-angle distance between {\mathbf{B}}_{t} and {\mathbf{B}}^{*} – addressing major technical challenges. (A) Projection operation on local heads \prod_{U_{\omega}} : This operation project \omega onto a convex ball with radius U_{\omega} . Unlike existing fine-grained analyses, such as in ( 19 ) , the dynamics of {\mathbf{B}} and \omega^{k} (one for each agent), together with \eta_{t}^{k} , are highly coupled due to the personalization. Without relying on any absolute bound on \omega^{k} , characterizing the error dynamics quickly becomes intractable. Despite this technical convenience, it introduces “not so small” perturbation that interplays with the errors of other dynamics nontrivially that can be harmful to the convergence and potential speedup. (B) Subspace projected innovation in Eq. ( 8 ): Instead of let {\mathbf{B}}_{t+1}^{k} towards the local innovation \delta_{t,L}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top} , we move {\mathbf{B}} in the direction of the “residual” innovation that lies outside of {\mathbf{B}}_{t} . This projection significantly mitigates some otherwise amplified perturbations arising from multiple cross terms. For example, under this projection, the perturbation in the QR decomposition is only second order; see Lemma C.1 . (C) QR: Under this decomposition, {\mathbf{B}}_{t} is orthonormal for each t , which is a key structural property that contributes to the contraction of the principal-angle distance between {\mathbf{B}}_{t} and {\mathbf{B}}^{*} ( 8 ) . It is worth noting that ( 42 ) does not use the above algorithmic details to control the perturbation. This is because that the challenge of single-timescale (of {\mathbf{B}} and \omega^{k} ) convergence analysis for this problem is that the estimation errors of the critic and the subspace misalignment are strongly coupled, whereas in double-loop or two-timescale analysis, these error terms are usually naturally decoupled. 5 Convergence Analysis We present some technical assumptions adopted in our convergence analysis. Assumptions 1 , 2 , and 3 are widely adopted in the RL literature. Assumption 4 quantifies how well spread the underlying truth z^{k,*} is in covering the r -dimensional subspace of {\mathbf{B}}^{*} . Assumption 1 (Exploration) . The matrix A_{L}^{k}=\mathbb{E}[\phi(s^{(0)})(\phi(s^{(L)})-\phi(s^{(0)}))^{\top}], where the expectation is taken with respect to s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell+1)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1] , is negative definite and its largest eigenvalue is upper bounded by -L\lambda . This assumption is widely adopted in analyzing TD learning with linear function approximation ( 6 ; 3 ; 5 ; 25 ; 26 ; 40 ; 49 ) . Intuitively, A_{L}^{k} captures the exploration of the policy \pi under the transition kernel P^{k} within L steps. For the tabular setting, Assumption 1 holds when the policy \pi explores all state-action pairs with L steps. Assumption 2 (Uniform ergodicity) . Let {\mathbb{P}}_{0:\tau}^{k}(\cdot|s_{0}^{k}=s) denote the state distribution after \tau steps given s_{0}^{k}=s for agent k . There exist m>0 and \rho\in(0,1) , such that at all k\in[K] , d_{TV}(\mathbb{P}_{0:\tau}^{k}(\cdot|s_{0}^{k}=s),\mu^{k}(\cdot))\leq m\rho^{\tau},~\forall\tau\geq 0,s\in{\mathcal{S}}, (9) where d_{TV} is the total variation distance. In our analysis, with a little abuse of notation, we also use {\mathbb{P}}_{0:\tau}^{k}(\cdot,\cdot|s_{0}^{k}=s) , which denotes the state-action pair at time \tau given s_{0}^{k}=s . Recall that \mu^{k} is the limiting distribution induced by policy \pi and the transition kernel P^{k},\forall k . Uniform ergodicity is often assumed to characterize the noise induced by Markovian sampling ( 6 ; 25 ; 49 ; 40 ) . Furthermore, we assume bounded features. Assumption 3 . \|\phi(s)\|\leq 1 for each s\in{\mathcal{S}} . This assumption is rather standard in RL literature ( 31 ; 30 ; 6 ; 25 ; 49 ; 40 ) . The existence of a shared common subspace helps to reduce the local learning problem from a d -dimensional problem to an r -dimensional problem. In PFL, when r\ll d , this benefits are well characgterized and are indeed substantial in derived bounds ( 8 ; 24 ) . Under Assumption 3 , however, the dependence on d is masked, potentially obscuring these gains in the resulting bounds. Adopting more fine-grained assumptions could therefore lead to significantly sharper results. Developing such refinements, however, would require a fundamental rethinking of the problem and is left for future work. Let \displaystyle{\mathbf{Z}}^{*}\in{\mathbb{R}}^{d\times K},~~~\text{with }z^{k,*}~~\text{as the $k$-th column}. (10) By Eq. ( 4 ), we know that the rank of {\mathbf{Z}}^{*}{\mathbf{Z}}^{*\top} is r . We require that each of the r dimensions of {\mathbf{B}}^{*} is well-covered by \{z^{k,*}\}_{k=1}^{K} , formally stated in Assumption 4 . Assumption 4 . \frac{1}{K}\lambda^{+}_{\min}({\mathbf{Z}}^{*}{\mathbf{Z}}^{*\top})\geq\alpha for some \alpha>0 . In our analysis, we also use \lambda_{\min}^{+}=\lambda_{\min}^{+}({\mathbf{Z}}^{*}{\mathbf{Z}}^{*\top}) to denote the smallest nonzero eigenvalue of {\mathbf{Z}}^{*}{\mathbf{Z}}^{*\top} . 5.1 Main Convergence Results In Algorithm 1 , three groups of variables are updated: the common subspace estimate {\mathbf{B}}_{t} , the local heads \omega_{t}^{k} , and estimates of local time-averaged rewards (defined in Eq. ( 1 )). In this subsection, we characterize their evolutions over time. For ease of exposition, we introduce a set of notation: \displaystyle x_{t}^{k} \displaystyle={\mathbf{B}}_{t}\omega_{t}^{k}-z^{k,*},~m_{t}={\mathbf{B}}^{*\top}_{\perp}{\mathbf{B}}_{t},~y_{t}^{k}=\eta_{t}^{k}-J^{k}, (11) \displaystyle\qquad\qquad\widetilde{\omega}_{t}^{k}=\omega^{k}_{t}+\beta\delta_{t,L}^{k}\mathbf{B}_{t}^{\top}\phi(s_{t,0}^{k}), (12) \displaystyle\widetilde{x}_{t}^{k}=\bar{\mathbf{B}}_{t}\widetilde{\omega}_{t}^{k}-z^{k,*},\quad\quad\bar{m}_{t}={\mathbf{B}}^{*\top}_{\perp}\bar{\mathbf{B}}_{t}. (13) Fix \tau . For any t\geq\tau , define M_{t}={\mathbb{E}}_{t-\tau}\|m_{t}\|_{F}^{2},\quad\bar{X}_{t}=\frac{1}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}, where {\mathbb{E}}_{t-\tau} is the conditional expectation, conditioning on the randomness up to time t-\tau . Lemma 5.1 ((informal) Upper bound of local head errors ) . Let {\mathbf{P}}^{*}={\mathbf{B}}^{*}({\mathbf{B}}^{*})^{\top} , and {\mathbf{P}}_{t}={\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top} for each t . Choose \tau=\lceil 2\log_{\rho}(\zeta)\rceil , U_{\delta}U_{\omega}\zeta\leq\frac{1}{2} , where U_{\delta}=2U_{r}+2U_{\omega} , \beta=c\zeta for arbitrary c>0 that can be tuned. There exist arbitrary positive constants c_{3},c_{4},c_{5},c_{6} such that \displaystyle\bar{X}_{t+1}\leq\Bigg(1-2\lambda c\zeta+\Big(c_{3}^{2}\frac{c}{L}+c_{4}^{2}c+c_{5}^{2}\frac{1}{L}+c_{6}^{2}+4U_{\omega}^{4}\frac{1}{c_{5}^{2}L}\Big)\zeta \displaystyle\qquad\qquad\qquad\qquad\qquad+{\mathcal{C}}_{X,1}(\tau^{2})\zeta^{2}+{\mathcal{C}}_{X,2}(\tau^{2})\zeta^{3}\Bigg)\bar{X}_{t} \displaystyle\qquad+\left(\frac{36U_{\omega}^{2}}{c_{3}^{2}}\frac{c\zeta}{L}+6U_{\omega}U_{\delta}c\zeta\right)M_{t} \displaystyle\qquad+\Bigg(\Big[\frac{1}{c_{4}^{2}}c+\frac{U_{\omega}^{4}}{c_{6}^{2}}\Big]\zeta+{\mathcal{C}}_{X,3}(\tau^{2})\zeta^{2}+{\mathcal{C}}_{X,4}(\tau^{2})\zeta^{3}\Bigg)\bar{Y}_{t} \displaystyle\qquad+{\mathcal{C}}_{X,5}(\tau^{2})\frac{\zeta^{2}}{L}+{\mathcal{C}}_{X,6}(\tau^{2})\frac{\zeta^{2}}{K}+O(\zeta^{3})+O(\gamma^{2}\zeta^{2}), where {\mathcal{C}}_{X,1}(\tau^{2}),{\mathcal{C}}_{X,2}(\tau^{2}),{\mathcal{C}}_{X,3}(\tau^{2}),{\mathcal{C}}_{X,4}(\tau^{2}),{\mathcal{C}}_{X,5}(\tau^{2}), and {\mathcal{C}}_{X,6}(\tau^{2}) are on the order of \tau^{2} , formally defined in Appendix D.1 . Intuitively, when M_{t}\lesssim\bar{X}_{t} . The above upper bound is similar to the traditional single-agent setting ( 6 ) . Deriving Lemma 5.1 is highly nontrivial and requires a fundamental detour from the existing analysis ( 6 ) . Furthermore, existing analysis in PFL ( 8 ) is not applicable to our problem due to the Markovian sampling, the TD updates, and the lack of responses. Roughly speaking, the main drift of the decay in \|x_{t}^{k}\| may arise from \langle x_{t}^{k},{\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k})\delta_{t,L}^{k}\rangle . However, the environmental heterogeneity significantly complicates the characterization of this term. Specifically, \displaystyle\langle x_{t}^{k},{\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k})\delta_{t}^{k}\rangle=\langle x_{t}^{k},{\mathbf{P}}_{t}A_{L}^{k}x_{t}^{k}\rangle+\text{``perturbation''}. Zooming into this expression, the main negative drift arises from the term \langle x_{t}^{k},{\mathbf{P}}_{t}A_{L}^{k}x_{t}^{k}\rangle . Intuitively, in traditional single-agent or homogeneous environment settings, \langle x_{t}^{k},{\mathbf{P}}_{t}A_{L}^{k}x_{t}^{k}\rangle\approx\langle x_{t}^{k},A_{L}x_{t}^{k}\rangle , which is mainly controlled by the spectrum of A_{L} , with the desired property directly assumed in Assumption 1 . However, in the presence of environmental heterogeneity, Assumption 1 does not directly guarantee a negative drift of the x_{t}^{k} due to the existence of {\mathbf{P}}_{t} and its intricate interplay with the heterogeneous A_{L}^{k} . Specifically, (1) {\mathbf{P}}_{t} is not of full rank, (2) the local head error will be distorted in a different manner due to the product {\mathbf{P}}_{t}A_{L}^{k} , and (3) {\mathbf{P}}_{t} varies over time. We further show that it is impossible for M_{t}\gg\bar{X}_{t} . In fact, a slightly stronger result holds. Lemma 5.2 (Lower bound of local head errors) . Suppose that d\geq 2r . It holds that \displaystyle\frac{1}{K}\sum_{k=1}^{K}\|x_{t}^{k}\|^{2}\geq\frac{\|m_{t}\|^{2}}{K}\lambda^{+}_{\min}({\mathbf{Z}}^{*}{\mathbf{Z}}^{*\top}). Intuitively, Lemma 5.2 says that when the principal angle distance between {\mathbf{B}}_{t} and {\mathbf{B}}^{*} is large, the well coverage of all the r directions by the underlying truth \{z^{1,*},\cdots,z^{K,*}\} ensures that the aggregate errors in the local head estimates remain bounded away from zero. In other words, the local heads cannot be learned significantly faster than the subspace itself. Lemma 5.3 (Principal angle distance analysis) . Choosing \beta=c\zeta for some c>0 , and \zeta so that U_{\delta}^{2}U_{\omega}^{2}\zeta^{2}\leq\frac{1}{6} , when d\geq 2r , there exist arbitrary positive c_{1},c_{2},c_{3},c_{4} such that \displaystyle M_{t+1}\leq(1-\frac{\lambda\alpha\zeta}{r}+c_{1}^{2}\zeta+c_{2}^{2}\zeta)M_{t} \displaystyle+\left(\frac{32U_{\omega}^{2}\zeta^{2}}{L^{2}}+\frac{16U_{\omega}^{2}}{c_{2}^{2}}\frac{\zeta}{L^{2}}+4c_{3}^{2}\frac{\zeta}{L^{2}}\right)\bar{X}_{t} \displaystyle+\left(8U_{\omega}^{2}\zeta^{2}+\frac{4U_{\omega}^{2}}{c_{1}^{2}}\zeta+4c_{4}^{2}\zeta\right)\bar{Y}_{t}+{\mathcal{C}}_{M,1}\frac{\zeta^{2}}{K}+{\mathcal{C}}_{M,2}\zeta^{6} \displaystyle+{\mathcal{C}}_{M,3}\zeta^{4}+{\mathcal{C}}_{M,4}\zeta^{3}+4C_{15}\tau C_{5}c\frac{\zeta^{2}}{\sqrt{L}} \displaystyle+4C_{14}\tau C_{10}\frac{\zeta^{2}}{\sqrt{K}}+4C_{19}(\tau)\gamma\zeta^{2}, where the missing specification of the (nearly) constant terms can be found in the analysis. They either are problem-specific constants or depends only on problem-specific consants and \tau . Lemma 5.4 (Reward analysis) . Fixed a \tau . For t>\tau , \displaystyle{\mathbb{E}}(y_{t}^{k})^{2}\leq(y_{0}^{k})^{2}e^{-2\gamma t}+\frac{\gamma}{2-\gamma}4U_{r}^{2}+8U^{2}_{r}m\rho^{\tau}(1-\gamma)^{\tau} \displaystyle\qquad+8U_{r}^{2}\tau\gamma(1-\gamma)^{\tau}+8U_{r}^{2}\frac{\tau(\tau-1)}{2}\gamma^{2} \displaystyle\qquad+4|y_{0}^{k}|U_{r}e^{\gamma\tau}e^{-\gamma 2t}+4U_{r}(1-\gamma)^{T}|y_{0}^{k}|m\rho^{\tau}. Choosing \tau\leq\min\{\frac{2T}{\log(T)},\lceil\log_{\rho}(\gamma)\rceil,\lceil 2\log_{\rho}(\zeta)\rceil\} and \gamma=\frac{\log(T)}{2T} , the average reward can be bounded as \displaystyle{\mathbb{E}}(y_{T}^{k})^{2}\leq\frac{28U_{r}^{2}}{T}+\frac{8U_{r}^{2}\tau\log(T)}{T}+\frac{8U_{r}m\log(T)}{T}. Lemma 5.4 says that reward estimation error decays at the rate of \widetilde{\mathcal{O}}(\frac{1}{T}) , where \widetilde{\mathcal{O}} hides the polylog factor of T . It is worth noting that in Algorithm 1 the reward learning at different clients are independent of each other, and the dynamics of {\mathbf{B}} and local heads \omega^{k} do not affect y^{k} . Though there is no averaging over agents, the reward learning itself can be made faster than the learning of subspace and local heads. With minor changes, the same analysis of Lemma 5.4 can be extended to the conditional expectation, such as {\mathbb{E}}_{t-\tau}(y_{t}^{k})^{2} for sufficiently large t . Theorem 1 (Convergence (informal)) . Consider Algorithm 1 with \beta=c\zeta,\zeta=\frac{4r}{\sqrt{T}\lambda\alpha},\gamma=\frac{\log(T)}{2T} , where c>0 and satisfies \frac{\alpha}{4r}\leq c\leq\frac{\alpha}{2r} , K>0 satisfies \max\left\{\frac{30}{\lambda K},\frac{120U_{\omega}^{4}}{\lambda K}\right\}\leq\frac{\alpha}{2r},L=K. Let {\mathcal{V}}_{T}=\bar{X}_{T}+\kappa M_{T} be the Lyapunov function, where \kappa\geq\frac{2(1080U_{\omega}^{2}+6U_{\omega}U_{\delta}\lambda L^{2})}{\lambda^{2}L^{2}}=\Theta(1) . Let \tau=\lceil 2\log_{\rho}(\zeta)\rceil . We have for T\geq 2\tau , and for sufficiently large K \displaystyle{\mathcal{V}}_{T}=\widetilde{O}(\frac{1}{\sqrt{(TK)K}})+{\mathcal{O}}(\frac{\log(T)}{T}), where \widetilde{\mathcal{O}} hides the polylog factors. When T is sufficiently large, the first term dominates, which is a linear speedup. The threshold of sufficiency of “large” L or K (recalling that we choose K=L ) can be found in Eq. ( 95 ), which is independent of T . 6 Experiments 6.1 Numerical Experiments on Prediction Problems In this section, we evaluate the performance of PMAAR-TD for approximating value functions under a fixed policy. We compare our proposed approach, PMAAR-TD , against the baselines introduced in Sections 3 : Single TD and FedTD-Uniform . Single TD is the standard single-agent TD learning; FedTD-Uniform periodically averages the parameters of all agents and outputs a universal value function for all agents across different environments. The following is the specifications of how to evaluate the performance of value function approximation: 1. A pre-trained single-agent TD model, trained with standard environment parameters, is employed as the fixed policy under which the value function is evaluated. 2. A fixed initial state is selected and consistently applied across all methods throughout the entire duration of the experiment. 3. Value function convergence is monitored by periodically evaluating the preselected initial state via a forward pass, utilizing the resulting value estimate as a proxy for training progress. We adopt this scheme because the state space is continuous, and the design ensures that the preselected state appears in every training episode. The value of the preselected state is recorded during training serves as a metric to quantify the convergence rate toward the optimal value function for each approach. For the experiments, we utilize three distinct seeds and a synchronization rate of {E=50} as shared hyperparameters. The specific configuration of the Acrobot environment for each agent is detailed below: Parameters Agent 0 Agent 1 Agent 2 Agent 3 Agent 4 Agent 5 link length 1 2.2 1.5 2 3 2.1 1.5 link length 2 1 1 0.8 1 1 1 We analyze the results presented in Figure 1 from two primary perspectives: 1. Convergence Speed: Regarding the convergence speed of the value function (as indicated by the curves), PMAAR-TD exhibits a significant acceleration compared to Single TD across the majority of agents. 2. Approximation Accuracy: In terms of the reward (y-axis), it is evident that FedTD-Uniform converges to a suboptimal value in most agent configurations. This discrepancy suggests a lack of adaptability to heterogeneous environments, whereas PMAAR-TD maintains consistency with the optimal baseline (single agent). Evaluation of value functions is relatively simple when the policy is fixed and the corresponding true values are determined. Consequently, we consider control problems, where policies are learned and updated throughout training, leading to increased uncertainty in learning. 6.1.1 Comparison with two-timescale setting We further evaluate PMAAR-TD against the two-timescale reinforcement learning framework (see Figure 2 ) within the same fixed-policy Acrobot environment. For this comparison, a global learning rate of 2\times 10^{-4} is employed, while the critic layers are specifically configured with a learning rate of 2\times 10^{-5} . The results demonstrate that, consistent with theoretical expectations, the two-timescale approach exhibits significantly slower empirical convergence compared to our single-timescale setting, further validating the efficiency of our proposed synchronization mechanism. 6.2 Application to Control Problems In this section, we extend PMAAR-TD to the actor-critic framework by utilizing PMAAR-TD as the critic component and incorporate an additional actor component. The action heads are also distinct across agents in our setting, i.e., agents do not share the parameters of those layers. 6.2.1 Model Design The architecture of the model employed in our experiments is defined as follows: \mathbf{s}_{t} MLP Actor Head \pi(a_{t}|s_{t}) Critic Head V(s_{t}) State Feature Extraction Action Value The fundamental distinction between SingleAC and FedAC-Uniform lies in the synchronization mechanism; specifically, SingleAC operates independently, whereas both FedAC-Uniform and PMAAR-AC incorporate periodic synchronization. Furthermore, the key innovation of PMAAR-AC is its selective synchronization strategy. During the synchronization phase, PMAAR-AC updates only the shared representation layers, while retaining the actor and critic heads as personalized local parameters to preserve agent-specific policies. 6.2.2 Environment Configurations Regarding the environment configurations, we find that the settings used in prior work are not sufficiently heterogeneous ( 18 ; 42 ) , as a single uniform policy can perform well across all environments. We therefore propose a new design that introduces stronger heterogeneity. Specifically, we create a mirrored environment where the agent’s action will be mapped to a reversed/unwanted actual action. When agents operate in normal and mirrored environments respectively, a shared policy cannot be effectively learned, as the optimal behaviors for one group are adversarial to the other. In our experiments, we employ three distinct random seeds and set the synchronization interval E={50} as shared hyperparameters across all configurations. The environment-specific parameters are detailed as follows: Table 1 : CartPole environment setup for each agent Parameters Agent 0 Agent 1 Agent 2 Agent 3 Agent 4 Agent 5 pole length 0.5 1 0.5 1 0.5 1 gravity 9.8 15 9.8 15 9.8 15 force 10 10 10 10 10 10 start position \frac{\pi}{6} - \frac{\pi}{6} \frac{\pi}{6} - \frac{\pi}{6} \frac{\pi}{6} - \frac{\pi}{6} env normal mirrored normal mirrored normal mirrored Table 2 : Acrobot environment setup for each agent Parameters Agent 0 Agent 1 Agent 2 Agent 3 Agent 4 Agent 5 link length 1 1 1 1 1 1 1 link length 2 1 1 1 1 1 1 env normal mirrored normal mirrored normal mirrored 6.2.3 Result We discuss the results presented in Figure 5 and 5 from three primary perspectives: 1. Convergence Speed : In terms of convergence velocity, it is evident that PMAAR-AC outperforms both FedAC-Uniform and Single AC across the majority of agents in both Acrobot and CartPole environments. Compared with Single AC , although PMAAR-AC exhibits a relatively marginal performance lag during the initial exploration phase, it frequently surpasses the baselines in the mid-training stage and maintains its performance advantage until convergence. 2. Reward : Regarding the asymptotic reward obtained post-training, PMAAR-AC demonstrates superior performance, outperforming other baselines across the majority of agents. Particularly in the Acrobot environment, the integration of additional data samples facilitated by shared feature synchronization enables the model to converge closer to the theoretical optimum compared to alternative methods. 3. Training Stability : Regarding the training stability—represented by the shaded regions reflecting the variance across different seeds—it is evident that for all agents across both environments, PMAAR-AC exhibits a consistent narrowing of the confidence intervals as training progresses. Ultimately, our approach achieves the smallest shaded area, indicating significantly lower variance between individual runs and a more robust convergence toward the mean reward curve compared to the baselines. Figure 1 : Value function convergence comparison on Acrobot. Figure 2 : Value function convergence speed comparison between single-timescale and two-timescale settings on Acrobot. 7 Conclusion and Discussions In this work, we studied cooperative TD learning in heterogeneous environments, where agents share a low-dimensional common structure but face diverse local dynamics. By combining subspace estimation with personalized value function updates, we established finite-time convergence guarantees with linear speedup under Markovian sampling and clarified the limitations imposed by standard assumptions. We hope that our analysis framework and insights provide a foundation for future work on exploiting common structures in multi-agent reinforcement learning. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. Appendices Appendix A Additional Related Work MARL under homogeneous environments. When the environments of all agents are homogeneous, it has been shown that the federated version of classic reinforcement learning algorithms can significantly alleviate the data collection burden on individual agents ( 39 ; 19 ) . Specifically, the per-agent sample complexity decreases linearly in terms of the number of agents – an effect commonly referred to in existing literature as linear speedup. Recent work has made significant progress in understanding federated RL under homogeneous environments. ( 39 ) studied federated Q-learning. In addition to a linear speedup, ( 39 ) uncovered the blessing of heterogeneity in terms of state-action exploration – a completely different notion of heterogeneity from our focus. ( 28 ) studied the sample-communication complexity trade-off, and developed a new algorithm that achieves order-optimal sample and communication complexities. Model/Policy personalization. In FL, model personalization (often referred to as PFL) has garnered significant attention in recent years. The success of personalization techniques depends on balancing the bias introduced by using global knowledge that may not generalize to individual clients, and the variance inherent in learning from limited local datasets. Popular PFL techniques include regularized local objectives ( 32 ; 21 ) , local-global parameter interpolation ( 10 ) , meta-learning ( 13 ; 15 ) , and representation learning ( 8 ; 43 ) . Extending these ideas of PFL to RL introduces an even greater level of difficulty that arises from the inherent sample correlation induced by sequential decision-making, and the non-stationarity of sample quality as the policy evolves. In the RL literature, training personalized (or localized) policies has recently garnered considerable attention in the context of multi-agent Markov games or competitive MARL ( 14 ; 46 ; 27 ; 48 ; 9 ) , which depart fundamentally from our focus. Specifically, in their setting, agents interact with one another within a shared environment, and the rewards received by individual agents are determined by the joint actions of all agents. Each agent aims to learn a localized policy that contributes to a joint policy across all agents, with the goal of maximizing its own welfare. A commonly adopted objective among the participating agents is to learn some form of equilibrium, such as a Nash Equilibrium. Departing from Markov games, we focus on cooperative MARL, where we try to avoid the curse of heterogeneity in the joint learning. Appendix B Notations and Preliminaries Let v_{0:t}^{k} denote the Markovian trajectory up to time t generated under initial state distribution d_{0} , given policy \pi , and the local transition kernel P^{k} : v_{0:t}^{k}=(s_{0}^{k},a_{0}^{k},s_{1}^{k},...,s_{t+1}^{k}), and v_{0:t} denote the collection of Markovian trajectories generated by all agents: \displaystyle v_{0:t}=\{v_{0:t}^{k}\}_{k=1}^{K}. Similarly, we define v_{0:(t,\ell)}^{k} and v_{0:(t,\ell)} as the Markovian trajectory generated up to the \ell -th sample in time t . Let {\mathbb{E}}_{j}[X_{t}] denote the conditional expectation of any random variable X_{t} given the collection of Markovian trajectories up to time j : \displaystyle{\mathbb{E}}_{j}[X_{t}]={\mathbb{E}}\left[X_{t}\mid v_{0:j}\right]={\mathbb{E}}\left[X_{t}\mid\{v_{0:j}^{k}\}_{k=1}^{K}\right]. Denote \displaystyle\widetilde{b}_{t,L}^{k}=\sum_{\ell=0}^{L-1}(r_{t,\ell}^{k}-J^{k})\phi(s_{t,0}^{k});\quad\bar{b}_{L}^{k}={\mathbb{E}}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left[\sum_{\ell=0}^{L-1}(r(s^{(\ell)},a^{(\ell)})-J^{k})\phi(s^{(0)})\right]. (14) In addition, let \displaystyle\xi_{t,L}^{k}= \displaystyle\sum_{\ell=0}^{L-1}(r_{t,\ell}^{k}-J^{k})\phi(s_{t,0}^{k}) (15) \displaystyle+(\phi(s_{t,L}^{k})-\phi(s_{t,0}^{k}))^{\top}\mathbf{B}_{t}\omega_{t}^{k}\phi(s_{t,0}^{k}) \displaystyle-\sum_{\ell=0}^{L-1}{\mathbb{E}}\left[(r(s^{(\ell)},a^{(\ell)})-J^{k})\phi(s^{(0)})\right] \displaystyle-{\mathbb{E}}\left[(\phi(s^{(L)})-\phi(s^{(0)}))^{\top}\mathbf{B}_{t}\omega_{t}^{k}\phi(s^{(0)})\right], where the expectation is taken with respect to {s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell+1)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]} . This can be viewed as the Markovian noise of the TD error of agent k at time t . Moreover, let \displaystyle\widetilde{A}_{t,L}^{k}=\phi(s_{t,0}^{k})(\phi(s_{t,L}^{k})-\phi(s_{t,0}^{k}))^{\top}, and \displaystyle{\mathbf{b}}_{t,L}^{k}=\sum_{\ell=0}^{L-1}\big((r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k}) \displaystyle~~~~~~~~~-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\big)\phi(s_{t,0}^{k}). (16) The following rewrite appears repeatedly in our analysis. Proposition 1 . For any t and agent k , \delta_{t,L}^{k}\phi(s_{t,0}^{k}) can be rewritten as \displaystyle\delta_{t,L}^{k}\phi(s_{t}^{k})=\xi_{t,L}^{k}+A_{L}^{k}x_{t}^{k}+Ly_{t}^{k}\phi(s_{t}^{k}), (17) and \displaystyle\delta_{t,L}^{k}\phi(s_{t}^{k})=\widetilde{A}_{t,L}^{k}x_{t}^{k}+Ly_{t}^{k}\phi(s_{t,0}^{k})+{\mathbf{b}}_{t,L}^{k}. (18) The proof of Proposition 1 can be found in Appendix I . Appendix C Bound on QR Decomposition and Perturbation Recall from Algorithm 1 the update of \bar{\mathbf{B}} : \displaystyle\bar{\mathbf{B}}_{t+1}=\mathbf{B}_{t}+\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{t,\perp}{\mathbf{B}}_{t,\perp}^{\top}\delta_{t,L}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top} For ease of exposition, let \displaystyle{\mathbf{Q}}_{t}:=\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{t,\perp}{\mathbf{B}}_{t,\perp}^{\top}\delta_{t,L}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}. (19) The following couple of lemmas enable us to bound the “distortion” caused by the QR decomposition. Specifically, Lemma C.1 bounds the perturbation in terms of \|\bf Q_{t}\| , which is further bounded in Lemma C.2 . It is worth noting that, in the traditional PFL ( 8 ) , such distortion can be well controlled by the number of i.i.d. local samples that are freshly drawn in each iteration. However, their analysis does not apply to our setting due to the Markov sampling. We fundamentally depart from the analysis in ( 8 ) in deriving upper bounds on those distortions via constructing a fixed-point iteration. Lemma C.1 (Perturbation of QR) . For each t\geq 1 , when U_{\delta}U_{\omega}\zeta\leq 1/2 , it holds that \displaystyle\|{{\mathbf{R}}_{t+1}-{\mathbf{I}}}\|_{2}\leq 2\|{\mathbf{Q}}_{t}\|_{F}^{2}, (20) \displaystyle\|{\mathbf{R}}_{t+1}^{-1}\|\leq\frac{1}{1-\|{\mathbf{R}}_{t+1}-{\mathbf{I}}\|_{2}}\leq\frac{1}{1-2\|{\mathbf{Q}}_{t}\|_{F}^{2}}, (21) \displaystyle\|{{\mathbf{R}}_{t+1}^{-1}-{\mathbf{I}}}\|_{2}\leq 4\|{\mathbf{Q}}_{t}\|_{F}^{2}. (22) Proof. Recall that \bar{\mathbf{B}}_{t+1}={\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1} . We rewrite {\mathbf{R}}_{t+1}^{\top}{\mathbf{R}}_{t+1} as \displaystyle{\mathbf{R}}_{t+1}^{\top}{\mathbf{R}}_{t+1} \displaystyle=\bar{\mathbf{B}}_{t+1}^{\top}\bar{\mathbf{B}}_{t+1}=\mathbf{B}_{t}^{\top}\mathbf{B}_{t}+\mathbf{B}_{t}^{\top}\mathbf{Q}_{t}+\mathbf{Q}_{t}^{\top}\mathbf{B}_{t}+\mathbf{Q}_{t}^{\top}\mathbf{Q}_{t} \displaystyle={\mathbf{I}}+\mathbf{B}_{t}^{\top}\mathbf{Q}_{t}+\mathbf{Q}_{t}^{\top}\mathbf{B}_{t}+\mathbf{Q}_{t}^{\top}\mathbf{Q}_{t} \displaystyle={\mathbf{I}}+\mathbf{Q}_{t}^{\top}\mathbf{Q}_{t}. The last equality holds because {\mathbf{Q}}_{t} lives in \text{span}({\mathbf{B}}_{t,\perp}) . It can also be rewritten as \displaystyle{\mathbf{R}}_{t+1}^{\top}{\mathbf{R}}_{t+1} \displaystyle=({\mathbf{I}}+{\mathbf{R}}_{t+1}-{\mathbf{I}})^{\top}({\mathbf{I}}+{\mathbf{R}}_{t+1}-{\mathbf{I}}) \displaystyle={\mathbf{I}}+({\mathbf{R}}_{t+1}-{\mathbf{I}})^{\top}+({\mathbf{R}}_{t+1}-{\mathbf{I}})+({\mathbf{R}}_{t+1}-{\mathbf{I}})^{\top}({\mathbf{R}}_{t+1}-{\mathbf{I}}). Thus, we get ({\mathbf{R}}_{t+1}-{\mathbf{I}})^{\top}+({\mathbf{R}}_{t+1}-{\mathbf{I}})=\mathbf{Q}_{t}^{\top}\mathbf{Q}_{t}-({\mathbf{R}}_{t+1}-{\mathbf{I}})^{\top}({\mathbf{R}}_{t+1}-{\mathbf{I}}). (23) We construct a fixed-point iteration to “solve” {\mathbf{R}}_{t+1}-{\mathbf{I}} in terms of \mathbf{Q}_{t}^{\top}\mathbf{Q}_{t} under Eq. ( 23 ). Toward this, we decompose the symmetric matrix \mathbf{Q}_{t}^{\top}\mathbf{Q}_{t} into \mathbf{Q}_{t}^{\top}\mathbf{Q}_{t}=\mathbf{R}_{0}+\mathbf{R}_{0}^{\top}, where \mathbf{R}_{0} is upper triangular. It is easy to see that \displaystyle\left\|{\mathbf{R}_{0}}\right\|_{{F}}\leq\left\|{\mathbf{R}_{0}+\mathbf{R}_{0}^{\top}}\right\|_{{F}}\leq\|\mathbf{Q}_{t}\|_{F}^{2}\leq\zeta^{2}U_{\delta}^{2}U_{\omega}^{2}. Let \Delta_{t+1}\triangleq{\mathbf{R}}_{t+1}-{\mathbf{I}}-\mathbf{R}_{0} . Then Eq. ( 23 ) is equivalent to \Delta_{t+1}^{\top}+\Delta_{t+1}=-(\Delta_{t+1}^{\top}\Delta_{t+1}+\Delta_{t+1}^{\top}\mathbf{R}_{0}+\mathbf{R}_{0}^{\top}\Delta_{t+1}+\mathbf{R}_{0}^{\top}\mathbf{R}_{0}). (24) Next, we define a sequence of \Delta_{t+1,i} for i\geq 0 . Let \Delta_{t+1,0}=0 for i\geq 1 and let \Delta_{t+1,i} be an upper triangular matrix satisfying \Delta_{t+1,i}^{\top}+\Delta_{t+1,i}=-(\Delta_{t+1,i-1}^{\top}\Delta_{t+1,i-1}+\Delta_{t+1,i-1}^{\top}\mathbf{R}_{0}+\mathbf{R}_{0}^{\top}\Delta_{t+1,i-1}+\mathbf{R}_{0}^{\top}\mathbf{R}_{0}). We first show the boundedness of the sequence. Note that \left\|{\Delta_{t+1,i}}\right\|_{{F}}\leq\left\|{\Delta_{t+1,i}^{\top}+\Delta_{t+1,i}}\right\|_{{F}}\leq\left\|{\Delta_{t+1,i-1}}\right\|_{{F}}^{2}+2\left\|{\mathbf{R}_{0}}\right\|_{{F}}\left\|{\Delta_{t+1,i-1}}\right\|_{{F}}+\left\|{\mathbf{R}_{0}}\right\|_{{F}}^{2}, (25) where the second inequality applies the triangle inequality, and the first inequality holds because the absolute value of all entries in \Delta_{t+1,i}^{\top}+\Delta_{t+1,i} is larger than that in \Delta_{t+1,i} . Provided \zeta^{2}U_{\delta}^{2}U_{\omega}^{2}<1/4 , we have \left\|{\mathbf{R}_{0}}\right\|_{{F}}<1/4 , and for any \left\|{\Delta_{t+1,i-1}}\right\|_{{F}}\leq 1/4 , we have \left\|{\Delta_{t+1,i}}\right\|_{{F}}\leq 1/4 by Eq. ( 25 ). By induction, the entire sequence satisfies \left\|{\Delta_{t+1,i}}\right\|_{{F}}\leq 1/4 for all i\geq 0 . Next, we show the convergence of the sequence \Delta_{t+1,i} . By definition, \displaystyle(\Delta_{t+1,i+1}^{\top}+\Delta_{t+1,i+1})-(\Delta_{t+1,i}^{\top}+\Delta_{t+1,i}) \displaystyle=~ \displaystyle-(\Delta_{t+1,i}^{\top}(\Delta_{t+1,i}-\Delta_{t+1,i-1})+(\Delta_{t+1,i}-\Delta_{t+1,i-1})^{\top}\Delta_{t+1,i-1} \displaystyle\qquad+(\Delta_{t+1,i}-\Delta_{t+1,i-1})^{\top}\mathbf{R}_{0}+\mathbf{R}_{0}^{\top}(\Delta_{t+1,i}-\Delta_{t+1,i-1})). By taking the norm on both sides, we obtain \displaystyle\left\|{\Delta_{t+1,i+1}-\Delta_{t+1,i}}\right\|_{{F}} \displaystyle\leq\left\|{(\Delta_{t+1,i+1}-\Delta_{t+1,i})^{\top}+(\Delta_{t+1,i+1}-\Delta_{t+1,i})}\right\|_{{F}} \displaystyle\leq\frac{1}{4}\left\|{\Delta_{t+1,i}-\Delta_{t+1,i-1}}\right\|_{{F}}+\frac{1}{4}\left\|{\Delta_{t+1,i}-\Delta_{t+1,i-1}}\right\|_{{F}} \displaystyle\quad+2\left\|{\mathbf{R}_{0}}\right\|_{{F}}\left\|{\Delta_{t+1,i}-\Delta_{t+1,i-1}}\right\|_{{F}}. Therefore, since \left\|{\mathbf{R}_{0}}\right\|_{{F}}<\frac{1}{4} , the difference \left\|{\Delta_{t+1,i+1}-\Delta_{t+1,i}}\right\|_{{F}} geometrically converges to zero, which implies that the sequence \Delta_{t+1,i} converges. The limit of the sequence satisfies the fixed-point Eq.  ( 24 ). Taking norm on both sides of Eq. ( 24 ) yields \left\|{\Delta_{t+1}}\right\|_{{F}}\leq\left\|{\Delta_{t+1}^{\top}+\Delta_{t+1}}\right\|_{{F}}\leq\left\|{\Delta_{t+1}}\right\|_{{F}}^{2}+2\left\|{\mathbf{R}_{0}}\right\|_{{F}}\left\|{\Delta_{t+1}}\right\|_{{F}}+\left\|{\mathbf{R}_{0}}\right\|_{{F}}^{2}. Since \left\|{\Delta_{t+1}}\right\|_{{F}}\leq 1/4 , we obtain that \left\|{\Delta_{t+1}}\right\|_{{F}}\leq\frac{1-2\left\|{\mathbf{R}_{0}}\right\|_{{F}}-\sqrt{1-4\left\|{\mathbf{R}_{0}}\right\|_{{F}}}}{2}\leq 4\left\|{\mathbf{R}_{0}}\right\|_{{F}}^{2}. We conclude that \left\|{{\mathbf{R}}_{t+1}-{\mathbf{I}}}\right\|_{{F}}=\left\|{\mathbf{R}_{0}+\Delta_{t+1}}\right\|_{{F}}\leq\left\|{\mathbf{R}_{0}}\right\|_{{F}}+4\left\|{\mathbf{R}_{0}}\right\|_{{F}}^{2}\leq 2\left\|{\mathbf{R}_{0}}\right\|_{{F}}\leq 2\left\|{{\mathbf{Q}}_{t}}\right\|_{{F}}^{2}. Moreover, \sigma_{\min}({\mathbf{R}}_{t+1})\geq 1-\|{{\mathbf{R}}_{t+1}-{\mathbf{I}}}\|_{2},~~~\text{by Weyl's inequality}, \sigma_{\max}({\mathbf{R}}_{t+1})\leq 1+\|{{\mathbf{R}}_{t+1}-{\mathbf{I}}}\|_{2},~~~\text{by triangle inequality}, and \|{{\mathbf{R}}_{t+1}^{-1}-{\mathbf{I}}}\|_{2}=\|{{\mathbf{R}}_{t+1}^{-1}({\mathbf{I}}-{\mathbf{R}}_{t+1})}\|_{2}\leq\|{{\mathbf{R}}_{t+1}^{-1}}\|_{2}\|{{\mathbf{I}}-{\mathbf{R}}_{t+1}}\|_{2}. Therefore, \displaystyle\left\|{{\mathbf{R}}_{t+1}-{\mathbf{I}}}\right\|_{{2}}\leq\left\|{{\mathbf{R}}_{t+1}-{\mathbf{I}}}\right\|_{{F}}\leq 2\|{\mathbf{Q}}_{t}\|_{F}^{2}, \displaystyle\|{\mathbf{R}}_{t+1}^{-1}\|\leq\frac{1}{1-\|{\mathbf{R}}_{t+1}-{\mathbf{I}}\|_{2}}\leq\frac{1}{1-\|{\mathbf{R}}_{t+1}-{\mathbf{I}}\|_{F}}\leq\frac{1}{1-2\left\|{\bf Q_{t}}\right\|_{{F}}^{2}}, \displaystyle\|{{\mathbf{R}}_{t+1}^{-1}-{\mathbf{I}}}\|_{2}\leq 4\|{\mathbf{Q}}_{t}\|_{F}^{2}. For the last inequality, we used 2U_{\delta}^{2}U_{\omega}^{2}\zeta^{2}\leq 1/2 so that \|{\mathbf{R}}_{t+1}^{-1}\|\leq 2 . ∎ Lemma C.2 . For any t\geq\tau , \displaystyle{\mathbb{E}}_{t-\tau}\|{\mathbf{Q}}_{t}\|_{F}^{2}\leq \displaystyle\frac{16U_{\omega}^{2}\zeta^{2}}{L^{2}}\frac{1}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+4\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{2}\frac{\zeta^{2}}{K}+8U_{\omega}^{2}\frac{U_{\delta,L}^{2}}{L^{2}}\zeta^{2}m^{2}\rho^{2\tau} \displaystyle+4\tau^{2}\frac{U_{\delta,L}^{4}}{L^{4}}\zeta^{2}\beta^{2}+\frac{4U_{\omega}^{2}\zeta^{2}}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}. (26) Proof. For any t\geq\tau , we have \displaystyle\|{\mathbf{Q}}_{t}\|_{F}^{2}=\|\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{P}}_{t,\perp}\delta_{t,L}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}\|_{F}^{2}\leq\|\frac{\zeta}{KL}\sum_{k=1}^{K}\delta_{t,L}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}\|_{F}^{2} \displaystyle= \displaystyle\frac{\zeta^{2}}{K^{2}L^{2}}\|\sum_{k=1}^{K}\left(\widetilde{A}_{t,L}^{k}x_{t}^{k}+{\mathbf{b}}_{t,L}^{k}+Ly_{t}^{k}\phi(s_{t,0}^{k})\right)(\omega_{t}^{k})^{\top}\|_{F}^{2}~~~(\text{by Proposition \ref{prop: key intermediate}}) \displaystyle\leq \displaystyle\underbrace{\frac{4\zeta^{2}L^{2}}{K^{2}}\|\sum_{k=1}^{K}\widetilde{A}_{t,L}^{k}x_{t}^{k}(\omega_{t}^{k})^{\top}\|_{F}^{2}}_{I_{1}}+\underbrace{\frac{4\zeta^{2}}{K^{2}L^{2}}\|\sum_{k=1}^{K}{\mathbf{b}}_{t,L}^{k}(\omega_{t-\tau}^{k})^{\top}\|_{F}^{2}}_{I_{2}} \displaystyle+\underbrace{\frac{4\zeta^{2}}{K^{2}L^{2}}\|\sum_{k=1}^{K}{\mathbf{b}}_{t,L}^{k}(\omega_{t}^{k}-\omega_{t-\tau}^{k})^{\top}\|_{F}^{2}}_{I_{3}}+\underbrace{\frac{4\zeta^{2}L^{2}}{K^{2}L^{2}}\|\sum_{k=1}^{K}y_{t}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}\|_{F}^{2}}_{I_{4}}. Recall that \widetilde{A}_{t,L}^{k}=\sum_{\ell=0}^{L-1}\phi(s_{t,0}^{k})(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top},{\mathbf{b}}_{t,L}^{k}=\sum_{\ell=0}^{L-1}\left((r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\right)\phi(s_{t,0}^{k}) . Then, \displaystyle\|\widetilde{A}_{t,L}^{k}\|=\|\phi(s_{t,0}^{k})(\phi(s_{t,L}^{k})-\phi(s_{t,0}^{k}))^{\top}\|\leq 2. (27) For I_{1} , we have, \displaystyle I_{1}\leq\frac{4\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}\|\widetilde{A}_{t,L}^{k}x_{t}^{k}(\omega_{t}^{k})^{\top}\|^{2}\leq\frac{16U_{\omega}^{2}\zeta^{2}}{L^{2}}\frac{1}{K}\sum_{k=1}^{K}\|x_{t}^{k}\|^{2}. For I_{2} , the Markovian noise, we have under expectation that, \displaystyle\frac{4\zeta^{2}}{K^{2}L^{2}}{\mathbb{E}}_{t-\tau}\|\sum_{k=1}^{K}{\mathbf{b}}_{t,L}^{k}(\omega_{t-\tau}^{k})^{\top}\|_{F}^{2} \displaystyle\leq \displaystyle\frac{4\zeta^{2}}{K^{2}L^{2}}{\mathbb{E}}_{t-\tau}\sum_{k=1}^{K}\|{\mathbf{b}}_{t,L}^{k}(\omega_{t-\tau}^{k})^{\top}\|^{2}+\frac{8\zeta^{2}}{K^{2}L^{2}}\sum_{k=1}^{K}\sum_{k^{\prime}=k+1}^{K}{\mathbb{E}}_{t-\tau}\left\langle{\mathbf{b}}_{t,L}^{k}(\omega_{t-\tau}^{k})^{\top},{\mathbf{b}}_{t,L}^{k^{\prime}}(\omega_{t-\tau}^{k^{\prime}})^{\top}\right\rangle \displaystyle\leq \displaystyle 4\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{2}\frac{\zeta^{2}}{K}+8\zeta^{2}U_{\omega}^{2}\frac{U_{\delta,L}^{2}}{L^{2}}m^{2}\rho^{2\tau}.~~~~~(\text{by Lemma \ref{lmm: U delta}, independence and the {ergodicity assumption}}) For I_{3} , \displaystyle\frac{4\zeta^{2}}{K^{2}L^{2}}\|\sum_{k=1}^{K}{\mathbf{b}}_{t,L}^{k}(\omega_{t}^{k}-\omega_{t-\tau}^{k})^{\top}\|_{F}^{2} \displaystyle\leq \displaystyle\frac{4\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}\|{\mathbf{b}}_{t,L}^{k}(\omega_{t}^{k}-\omega_{t-\tau}^{k})^{\top}\|^{2} \displaystyle\leq \displaystyle\frac{4\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}\|{\mathbf{b}}_{t,L}^{k}\|^{2}\|\omega_{t}^{k}-\omega_{t-\tau}^{k}\|^{2} \displaystyle\leq \displaystyle 4\frac{U_{\delta,L}^{2}}{L^{2}}\zeta^{2}\left(\tau\frac{U_{\delta,L}}{L}\beta\right)^{2}~~~~~(\text{by Lemma \ref{lmm: crude parameter bound}}) \displaystyle= \displaystyle 4\tau^{2}\frac{U_{\delta,L}^{4}}{L^{4}}\zeta^{2}\beta^{2}. Lastly, for I_{4} , we have \displaystyle I_{4}\leq \displaystyle\frac{4\zeta^{2}}{K}\sum_{k=1}^{K}\|y_{t}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}\|^{2}\leq\frac{4U_{\omega}^{2}\zeta^{2}}{K}\sum_{k=1}^{K}|y_{t}^{k}|^{2}. In summary, \displaystyle{\mathbb{E}}_{t-\tau}\|{\mathbf{Q}}_{t}\|_{F}^{2}\leq \displaystyle\frac{16U_{\omega}^{2}\zeta^{2}}{L^{2}}\frac{1}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+4\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{2}\frac{\zeta^{2}}{K}+8U_{\omega}^{2}\frac{U_{\delta,L}^{2}}{L^{2}}\zeta^{2}m^{2}\rho^{2\tau} \displaystyle+4\tau^{2}\frac{U_{\delta,L}^{4}}{L^{4}}\zeta^{2}\beta^{2}+\frac{4U_{\omega}^{2}\zeta^{2}}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}. ∎ Appendix D Bounds on Local Head Errors D.1 Upper bound: Proof of Lemma 5.1 We first provide a formal statement of Lemma 5.1 , which involves a set of constants whose formal definitions are deferred to the lemma proof. Lemma D.1 (Upper bound of local head errors) . Let {\mathbf{P}}^{*}={\mathbf{B}}^{*}({\mathbf{B}}^{*})^{\top} , and {\mathbf{P}}_{t}={\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top} for each t . Choose \tau=\lceil 2\log_{\rho}(\zeta)\rceil , U_{\delta}U_{\omega}\zeta\leq\frac{1}{2} , where U_{\delta}=2U_{r}+2U_{\omega} , \beta=c\zeta for arbitrary c>0 that can be tuned. There exist arbitrary positive constants c_{3},c_{4},c_{5},c_{6} such that \displaystyle\bar{X}_{t+1}\leq\Bigg(1-2\lambda c\zeta+\Big(c_{3}^{2}\frac{c}{L}+c_{4}^{2}c+c_{5}^{2}\frac{1}{L}+c_{6}^{2}+4U_{\omega}^{4}\frac{1}{c_{5}^{2}L}\Big)\zeta+{\mathcal{C}}_{X,1}(\tau^{2})\zeta^{2}+{\mathcal{C}}_{X,2}(\tau^{2})\zeta^{3}\Bigg)\bar{X}_{t} \displaystyle\qquad+\left(\frac{36U_{\omega}^{2}}{c_{3}^{2}}\frac{c\zeta}{L}+6U_{\omega}U_{\delta}c\zeta\right)M_{t} \displaystyle\qquad+\Bigg(\Big[\frac{1}{c_{4}^{2}}c+\frac{U_{\omega}^{4}}{c_{6}^{2}}\Big]\zeta+{\mathcal{C}}_{X,3}(\tau^{2})\zeta^{2}+{\mathcal{C}}_{X,4}(\tau^{2})\zeta^{3}\Bigg)\bar{Y}_{t} \displaystyle\qquad+{\mathcal{C}}_{X,5}(\tau^{2})\frac{\zeta^{2}}{L}+{\mathcal{C}}_{X,6}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{C}}_{X,7}\zeta^{3}+{\mathcal{C}}_{X,8}\zeta^{4}+{\mathcal{C}}_{X,9}\gamma^{2}\zeta^{2}+{\mathcal{C}}_{X,10}\zeta^{5}+{\mathcal{C}}_{X,11}\gamma^{2}\zeta^{3}+{\mathcal{C}}_{X,12}\zeta^{7}+{\mathcal{C}}_{X,13}\zeta^{6}, where \displaystyle{\mathcal{C}}_{X,1}(\tau^{2})=\bigg(C_{31}c+(36+40\tau^{2})\frac{c^{2}}{L^{2}}+C_{33}\frac{c}{L}+\left(36U_{\omega}^{4}+\left(2.5+3U_{\omega}\right)C_{18}\right)\frac{1}{L^{2}}+\frac{96c^{2}}{L^{2}}+\frac{256U_{\omega}^{4}}{L^{2}}+96U_{\delta}c\frac{U_{\omega}^{3}}{c_{2}^{2}}\frac{1}{L^{2}}\bigg), \displaystyle{\mathcal{C}}_{X,2}(\tau^{2})=6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,4}(\tau^{2}), \displaystyle{\mathcal{C}}_{X,3}(\tau^{2})=\Bigg[C_{32}c+(9+10\tau^{2})c^{2}+C_{34}c+9U_{\omega}^{4}+\left(2.5+3U_{\omega}\right)C_{23}+24c^{2}+64U_{\omega}^{4}+24U_{\delta}c\frac{U_{\omega}^{3}}{c_{1}^{2}}\Bigg], \displaystyle{\mathcal{C}}_{X,4}(\tau^{2})=6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,5}(\tau^{2}),{\mathcal{C}}_{X,5}(\tau^{2})=\left[C_{39}(\tau^{2})+24c^{2}U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})\right],{\mathcal{C}}_{X,6}(\tau^{2})=\left[C_{41}(\tau^{2})+64\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{4}\right], \displaystyle{\mathcal{C}}_{X,7}=\left[C_{42}+C_{43}(\tau)+24U_{\omega}U_{\delta}cC_{25}(\tau^{2})\frac{1}{L}+6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,2}(\tau^{2})\frac{1}{K}\right], \displaystyle{\mathcal{C}}_{X,8}=\left[C_{40}(\tau^{4})+100U_{\omega}^{6}U_{\delta}^{4}+144U_{\omega}^{2}\tau^{2}\frac{U_{\delta,L}^{4}}{L^{4}}c^{2}+6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,6}\right],{\mathcal{C}}_{X,9}=C_{45}(\tau^{4}),{\mathcal{C}}_{X,10}=6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,1}, \displaystyle{\mathcal{C}}_{X,11}=24U_{\omega}U_{\delta}cC_{28}(\tau^{4}),{\mathcal{C}}_{X,12}=6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,3},{\mathcal{C}}_{X,13}=\left[128U_{\omega}^{4}\frac{U_{\delta,L}^{2}}{L^{2}}m^{2}+C_{44}(\tau^{2})\right], and the missing definition of constants such as C_{3}1 can be found in the analysis. Proof. Recall that \omega_{t+1}^{k}=\Pi_{U_{\omega}}(\omega_{t}^{k}+\frac{\beta}{L}\delta_{t,L}^{k}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k})) , {\mathbf{B}}_{t+1}=\bar{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}^{-1} , and \widetilde{\omega}_{t+1}^{k}=\omega_{t}^{k}+\frac{\beta}{L}\delta_{t,L}^{k}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k}) . We have \displaystyle\|x_{t+1}^{k}\|= \displaystyle\|{\mathbf{B}}_{t+1}\omega_{t+1}^{k}-z^{k,*}\| \displaystyle= \displaystyle\|{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-z^{k,*}\| \displaystyle= \displaystyle\|{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}-z^{k,*}+{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\| \displaystyle= \displaystyle\|\bar{\mathbf{B}}_{t+1}\widetilde{\omega}_{t+1}^{k}-z^{k,*}+{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\| \displaystyle= \displaystyle\|\widetilde{x}_{t+1}^{k}+{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\|. Taking the square on both sides, we get \displaystyle\|x_{t+1}^{k}\|^{2}= \displaystyle\|\widetilde{x}_{t+1}^{k}\|^{2}+\|{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\|^{2} (28) \displaystyle+2\langle{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k},\widetilde{x}_{t+1}^{k}\rangle. We bound \|{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\|^{2} as follows: We have \displaystyle\|{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\| \displaystyle= \displaystyle\|\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\| \displaystyle= \displaystyle\|\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-\widetilde{\omega}_{t+1}^{k}+\widetilde{\omega}_{t+1}^{k}-{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\| \displaystyle\leq \displaystyle\|\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-\widetilde{\omega}_{t+1}^{k}\|+\|({\mathbf{I}}-{\mathbf{R}}_{t+1})\widetilde{\omega}_{t+1}^{k}\| \displaystyle\leq \displaystyle\|\frac{\beta}{L}\delta_{t,L}^{k}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k})\|+\|({\mathbf{I}}-{\mathbf{R}}_{t+1})\widetilde{\omega}_{t+1}^{k}\| where the last inequality follows from Eq. ( 20 ) and we use \frac{U_{\delta,L}}{L}\beta\leq U_{\omega} for large T such that \|\widetilde{\omega}_{t+1}^{k}\|\leq 2U_{\omega} . Then, \displaystyle\|{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\|^{2} \displaystyle\leq \displaystyle 2\frac{\beta^{2}}{L^{2}}\|\delta_{t,L}^{k}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k})\|^{2}+32U_{\omega}^{2}\|{\mathbf{Q}}\|_{F}^{4}. For the first term, we further decompose it as: \displaystyle\|\delta_{t,L}^{k}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k})\|^{2}\leq\|\delta_{t,L}^{k}\phi(s_{t,0}^{k})\|^{2} \displaystyle= \displaystyle\|\widetilde{A}_{t,L}^{k}x_{t}^{k}+{\mathbf{b}}_{t,L}^{k}+Ly_{t}^{k}\phi(s_{t,0}^{k})\|^{2}~~~~(\text{by Proposition \ref{prop: key intermediate}}) \displaystyle\leq \displaystyle 3\|\widetilde{A}_{t,L}^{k}x_{t}^{k}\|^{2}+3\|{\mathbf{b}}_{t,L}^{k}\|^{2}+3\|Ly_{t}^{k}\phi(s_{t,0}^{k})\|^{2} \displaystyle\leq \displaystyle 12\|x_{t}^{k}\|^{2}+3\|{\mathbf{b}}_{t,L}^{k}\|^{2}+3L^{2}|y_{t}^{k}|^{2}. Taking conditional expectation over the Markovian trajectory up to t-\tau , we get \displaystyle{\mathbb{E}}_{t-\tau}\|\delta_{t,L}^{k}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k})\|^{2} \displaystyle\leq \displaystyle 12{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+3{\mathbb{E}}_{t-\tau}\|{\mathbf{b}}_{t,L}^{k}\|^{2}+3L^{2}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2} \displaystyle\leq \displaystyle 12{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+3U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})L+3L^{2}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}.~~(\text{by Lemma \ref{lmm: local noise reduction} and tower property}) (29) Finally, the second term in Eq. ( 28 ) can be bounded as \displaystyle{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\|^{2} \displaystyle\leq \displaystyle 24\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+6\frac{\beta^{2}}{L}U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})+6\beta^{2}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}+32U_{\omega}^{2}{\mathbb{E}}_{t-\tau}\|{\mathbf{Q}}\|_{F}^{4}. (30) As for 2\langle{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k},\widetilde{x}_{t+1}^{k}\rangle , we first write \widetilde{x}_{t+1}^{k} as \displaystyle\widetilde{x}_{t+1}^{k}=\bar{\mathbf{B}}_{t+1}\widetilde{\omega}_{t+1}^{k}-{\mathbf{B}}^{*}\omega^{*,k}={\mathbf{B}}_{t+1}\omega_{t+1}^{k}-{\mathbf{B}}^{*}\omega^{k,*}+\bar{\mathbf{B}}_{t+1}\widetilde{\omega}_{t+1}^{k}-{\mathbf{B}}_{t+1}\omega_{t+1}^{k}. Thus, \displaystyle\langle{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k},\widetilde{x}_{t+1}^{k}\rangle \displaystyle=\underbrace{\langle{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k},{\mathbf{B}}_{t+1}\omega_{t+1}^{k}-{\mathbf{B}}^{*}\omega^{k,*}\rangle}_{I_{1}} \displaystyle\quad+\underbrace{\langle{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k},\bar{\mathbf{B}}_{t+1}\widetilde{\omega}_{t+1}^{k}-{\mathbf{B}}_{t+1}\omega_{t+1}^{k}\rangle}_{I_{2}}. Both I_{1} and I_{2} can be well controlled by the projection error of \widetilde{w} and the perturbation error that arises from the QR decomposition of \bar{{\mathbf{B}}} . We bound I_{1} and I_{2} separately. On I_{2} , we have \displaystyle\bar{\mathbf{B}}_{t+1}\widetilde{\omega}_{t+1}^{k}-{\mathbf{B}}_{t+1}\omega_{t+1}^{k} \displaystyle={\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}-{\mathbf{B}}_{t+1}\omega_{t+1}^{k} \displaystyle={\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\omega_{t+1}^{k}+{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\omega_{t+1}^{k}-{\mathbf{B}}_{t+1}\omega_{t+1}^{k} \displaystyle={\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\left(\widetilde{\omega}_{t+1}^{k}-\omega_{t+1}^{k}\right)+{\mathbf{B}}_{t+1}\left({\mathbf{R}}_{t+1}-{\mathbf{I}}\right)\omega_{t+1}^{k}. So, I_{2} can be rewritten as \displaystyle\langle{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k},\bar{\mathbf{B}}_{t+1}\widetilde{\omega}_{t+1}^{k}-{\mathbf{B}}_{t+1}\omega_{t+1}^{k}\rangle \displaystyle=\underbrace{\langle{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k},{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\left(\widetilde{\omega}_{t+1}^{k}-\omega_{t+1}^{k}\right)\rangle}_{I_{21}} \displaystyle\quad+\underbrace{\langle{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k},{\mathbf{B}}_{t+1}\left({\mathbf{R}}_{t+1}-{\mathbf{I}}\right)\omega_{t+1}^{k}\rangle}_{I_{22}}. For I_{21} , by Cauchy Schwartz, we have \displaystyle I_{21}\leq\|{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\|\|{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\left(\widetilde{\omega}_{t+1}^{k}-\omega_{t+1}^{k}\right)\| \displaystyle\leq \displaystyle\frac{1}{2}\|{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\|^{2}+\frac{1}{2}\|{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\left(\widetilde{\omega}_{t+1}^{k}-\omega_{t+1}^{k}\right)\|^{2} \displaystyle\leq \displaystyle\frac{1}{2}\|{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\|^{2}+\|\widetilde{\omega}_{t+1}^{k}-\omega_{t+1}^{k}\|^{2}. For the last inequality, we use \|{\mathbf{R}}_{t+1}\|\leq 2 for when 2U_{\delta}^{2}U_{\omega}^{2}\zeta^{2}\leq 1/2 . As for \|\widetilde{\omega}_{t+1}^{k}-\omega_{t+1}^{k}\|^{2} , by Eq. ( D.1 ), we have \displaystyle{\mathbb{E}}_{t-\tau}\|\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k}\|^{2}\leq{\mathbb{E}}_{t-\tau}\|\frac{\beta}{L}\delta_{t,L}^{k}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k})\|^{2} \displaystyle\leq \displaystyle 12\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+3\frac{\beta^{2}}{L}U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})+3\beta^{2}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}. For I_{22} , we have \displaystyle I_{22}\leq\|{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\|\|{\mathbf{B}}_{t+1}\left({\mathbf{R}}_{t+1}-{\mathbf{I}}\right)\omega_{t+1}^{k}\| \displaystyle\leq \displaystyle\frac{1}{2}\|{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\|^{2}+\frac{1}{2}\|{\mathbf{B}}_{t+1}\left({\mathbf{R}}_{t+1}-{\mathbf{I}}\right)\omega_{t+1}^{k}\|^{2} \displaystyle\leq \displaystyle\frac{1}{2}\|{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\|^{2}+\frac{1}{2}U_{\omega}^{2}\|{\mathbf{R}}_{t+1}-{\mathbf{I}}\|^{2} \displaystyle\leq \displaystyle\frac{1}{2}\|{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\|^{2}+2U_{\omega}^{2}\|{\mathbf{Q}}_{t}\|_{F}^{4}.~~~~\text{(by Eq.\,(\ref{eqn: R related bounds: R-I}))} Combining I_{21} and I_{22} , we get \displaystyle{\mathbb{E}}_{t-\tau}[I_{2}]\leq{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\|^{2} \displaystyle+12\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+3\frac{\beta^{2}}{L}U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})+3\beta^{2}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}+2U_{\omega}^{2}{\mathbb{E}}_{t-\tau}\|{\mathbf{Q}}_{t}\|_{F}^{4} \displaystyle\leq \displaystyle 36\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+9\frac{\beta^{2}}{L}U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})+9\beta^{2}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}+34U_{\omega}^{2}{\mathbb{E}}_{t-\tau}\|{\mathbf{Q}}_{t}\|_{F}^{4}.~~~~~~\text{(by Eq.\,(\ref{eq: upper bound of qr-proj squared}))} Bounding I_{1} is slightly more involved. We have \displaystyle I_{1}=\langle{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k},{\mathbf{B}}_{t+1}\omega_{t+1}^{k}-{\mathbf{B}}^{*}\omega^{k,*}\rangle \displaystyle=\underbrace{\langle{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}\widetilde{\omega}_{t+1}^{k},{\mathbf{B}}_{t+1}\omega_{t+1}^{k}-{\mathbf{B}}^{*}\omega^{k,*}\rangle}_{I_{11}} \displaystyle\quad+\underbrace{\langle{\mathbf{B}}_{t+1}\widetilde{\omega}_{t+1}^{k}-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k},{\mathbf{B}}_{t+1}\omega_{t+1}^{k}-{\mathbf{B}}^{*}\omega^{k,*}\rangle}_{I_{12}} For I_{11} , let {\mathcal{R}}={\mathbf{U}}{\mathbf{V}}^{\top}, where {\mathbf{U}}\Sigma{\mathbf{V}}^{\top}={\mathbf{B}}_{t+1}^{\top}{\mathbf{B}}^{*} is the SVD. Note that {\mathcal{R}} is the optimal rotation matrix that aligns {\mathbf{B}}_{t+1} and {\mathbf{B}}^{*} in the sense that {\mathcal{R}}=\text{argmin}_{{\mathcal{R}}^{\top}{\mathcal{R}}={\mathbf{I}}}\|{\mathbf{B}}_{t+1}{\mathcal{R}}-{\mathbf{B}}^{*}\|_{F} . Then, we have \displaystyle I_{11}=\langle{\mathbf{B}}_{t+1}(\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k}),{\mathbf{B}}_{t+1}{\mathcal{R}}{\mathcal{R}}^{-1}\omega_{t+1}^{k}-{\mathbf{B}}^{*}\omega^{k,*}\rangle~~~(\text{by rotation}) \displaystyle= \displaystyle\langle{\mathbf{B}}_{t+1}{\mathcal{R}}{\mathcal{R}}^{-1}(\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k}),({\mathbf{B}}_{t+1}{\mathcal{R}}-{\mathbf{B}}^{*}){\mathcal{R}}^{-1}\omega_{t+1}^{k}\rangle \displaystyle+\langle{\mathbf{B}}_{t+1}(\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k}),{\mathbf{B}}^{*}({\mathcal{R}}^{-1}\omega_{t+1}^{k}-\omega^{k,*})\rangle \displaystyle= \displaystyle({\mathcal{R}}^{-1}(\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k}))^{\top}({\mathbf{I}}-({\mathbf{B}}_{t+1}{\mathcal{R}})^{\top}{\mathbf{B}}^{*}){\mathcal{R}}^{-1}\omega_{t+1}^{k} \displaystyle+\langle{\mathbf{B}}_{t+1}{\mathcal{R}}{\mathcal{R}}^{-1}(\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k}),({\mathbf{B}}^{*}-{\mathbf{B}}_{t+1}{\mathcal{R}})({\mathcal{R}}^{-1}\omega_{t+1}^{k}-\omega^{k,*})\rangle \displaystyle+\langle{\mathbf{B}}_{t+1}{\mathcal{R}}{\mathcal{R}}^{-1}(\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k}),{\mathbf{B}}_{t+1}{\mathcal{R}}({\mathcal{R}}^{-1}\omega_{t+1}^{k}-\omega^{k,*})\rangle \displaystyle= \displaystyle({\mathcal{R}}^{-1}(\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k}))^{\top}({\mathbf{I}}-({\mathbf{B}}_{t+1}{\mathcal{R}})^{\top}{\mathbf{B}}^{*}){\mathcal{R}}^{-1}\omega_{t+1}^{k} \displaystyle+\langle{\mathcal{R}}^{-1}(\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k}),(({\mathbf{B}}_{t+1}{\mathcal{R}})^{\top}{\mathbf{B}}^{*}-{\mathbf{I}})({\mathcal{R}}^{-1}\omega_{t+1}^{k}-\omega^{k,*})\rangle \displaystyle+\langle\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k},\omega_{t+1}^{k}-{\mathcal{R}}\omega^{k,*}\rangle. We have the following facts: • Fact 1: Since the ball with radius U_{\omega} is convex, and \omega^{k,*} belongs to the ball, it holds that \langle\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k},\omega_{t+1}^{k}-{\mathcal{R}}\omega^{k,*}\rangle\leq 0 . • Fact 2: \displaystyle\|{\mathbf{I}}-({\mathbf{B}}_{t+1}{\mathcal{R}})^{\top}{\mathbf{B}}^{*}\|\leq\|{\mathbf{B}}_{\perp}^{*\top}{\mathbf{B}}_{t+1}\|^{2}. (31) The proof of Fact 2 is postponed to Appendix I.1 . Therefore, \displaystyle I_{11}\leq \displaystyle\left\|{\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k}}\right\|_{2}\left\|{{\mathbf{I}}-({\mathbf{B}}_{t+1}{\mathcal{R}})^{\top}{\mathbf{B}}^{*}}\right\|_{2}\left\|{\omega_{t+1}^{k}}\right\|_{2} \displaystyle\quad+\left\|{\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k}}\right\|_{2}\left\|{{\mathbf{I}}-({\mathbf{B}}_{t+1}{\mathcal{R}})^{\top}{\mathbf{B}}^{*}}\right\|_{2}\left\|{{\mathcal{R}}^{-1}\omega_{t+1}^{k}-\omega^{k,*}}\right\|_{2} \displaystyle\leq \displaystyle 3U_{\omega}\left\|{\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k}}\right\|_{2}\|m_{t+1}\|^{2}. Similarly, for I_{12} , we have \displaystyle I_{12}=\langle{\mathbf{B}}_{t+1}\widetilde{\omega}_{t+1}^{k}-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k},{\mathbf{B}}_{t+1}\omega_{t+1}^{k}-{\mathbf{B}}^{*}\omega^{k,*}\rangle \displaystyle=\langle{\mathbf{B}}_{t+1}({\mathbf{I}}-{\mathbf{R}}_{t+1})\widetilde{\omega}_{t+1}^{k},{\mathbf{B}}_{t+1}\omega_{t+1}^{k}-{\mathbf{B}}^{*}\omega^{k,*}\rangle \displaystyle\leq 4U_{\omega}^{2}\|{\mathbf{I}}-{\mathbf{R}}_{t+1}\| \displaystyle\leq 8U_{\omega}^{2}\|{\mathbf{Q}}_{t}\|_{F}^{2}. Summing up I_{1} and I_{2} and multiplying by 2, we have \displaystyle 2{\mathbb{E}}_{t-\tau}\langle{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k},\widetilde{x}_{t+1}^{k}\rangle \displaystyle\leq \displaystyle 72\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+18\frac{\beta^{2}}{L}U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})+18\beta^{2}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}+68U_{\omega}^{2}{\mathbb{E}}_{t-\tau}\|{\mathbf{Q}}_{t}\|_{F}^{4} \displaystyle+6U_{\omega}{\mathbb{E}}_{t-\tau}\left\|{\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k}}\right\|_{2}\|m_{t+1}\|^{2}+16U_{\omega}^{2}{\mathbb{E}}_{t-\tau}\|{\mathbf{Q}}_{t}\|_{F}^{2}. (32) We bound \|\widetilde{x}_{t+1}^{k}\| as follows: \displaystyle\widetilde{x}_{t+1}^{k} \displaystyle=\bar{\mathbf{B}}_{t+1}\widetilde{\omega}_{t+1}^{k}-z^{k,*} \displaystyle=\left({\mathbf{B}}_{t}+\frac{\zeta}{KL}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}\delta_{t,L}^{i}\phi(s_{t,0}^{i})(\omega_{t}^{i})^{\top}\right)\left(\omega_{t}^{k}+\frac{\beta}{L}\delta_{t,L}^{k}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k})\right)-z^{k,*} \displaystyle={\mathbf{B}}_{t}\omega_{t}^{k}-z^{k,*}+\frac{\beta}{L}{\mathbf{B}}_{t}(\delta_{t,L}^{k}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k}))+\frac{\zeta}{KL}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}\delta_{t,L}^{i}\phi(s_{t,0}^{i})(\omega_{t}^{i})^{\top}\omega_{t}^{k} \displaystyle\qquad+\zeta\beta(\frac{1}{KL^{2}}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}\delta_{t,L}^{i}\phi(s_{t,0}^{i})(\omega_{t}^{i})^{\top})\left(\delta_{t,L}^{k}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k})\right). Squaring both sides, we have \displaystyle\|\widetilde{x}_{t+1}^{k}\|^{2}= \displaystyle\|\bar{\mathbf{B}}_{t+1}\widetilde{\omega}_{t+1}^{k}-z^{k,*}\|^{2} \displaystyle\leq \displaystyle\|{\mathbf{B}}_{t}\omega_{t}^{k}-z^{k,*}\|^{2}+\underbrace{2\frac{\beta}{L}\langle{\mathbf{B}}_{t}\omega_{t}^{k}-z^{k,*},{\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top}\delta_{t,L}^{k}\phi(s_{t,0}^{k})\rangle}_{I_{1}} \displaystyle\quad+\underbrace{2\zeta\langle{\mathbf{B}}_{t}\omega_{t}^{k}-z^{k,*},\frac{1}{KL}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}\phi(s_{t,0}^{i})\delta_{t,L}^{i}(\omega_{t}^{i})^{\top}\omega_{t}^{k}\rangle}_{I_{2}} \displaystyle\quad+\underbrace{2\frac{\zeta\beta}{L^{2}}\left\langle{\mathbf{B}}_{t}\omega_{t}^{k}-z^{k,*},(\frac{1}{K}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}\delta_{t,L}^{i}\phi(s_{t,0}^{i})(\omega_{t}^{i})^{\top})\left(\delta_{t,L}^{k}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k})\right)\right\rangle}_{I_{3}} \displaystyle\quad+\underbrace{3\frac{\beta^{2}}{L^{2}}\|{\mathbf{B}}_{t}(\delta_{t,L}^{k}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k}))\|^{2}}_{I_{4}}+\underbrace{3\frac{\zeta^{2}}{L^{2}}\|\frac{1}{K}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}\delta_{t,L}^{i}\phi(s_{t,0}^{i})(\omega_{t}^{i})^{\top}\omega_{t}^{k}\|^{2}}_{I_{5}} \displaystyle\quad+\underbrace{3\frac{\beta^{2}\zeta^{2}}{L^{4}}\|(\frac{1}{K}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}\delta_{t,L}^{i}\phi(s_{t,0}^{i})(\omega_{t}^{i})^{\top})\left(\delta_{t,L}^{k}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k})\right)\|^{2}}_{I_{6}}. (33) We bound these terms separately. Bounding I_{1} . Roughly speaking, the main drift of the decay in \|x_{t}^{k}\| may arise from 2\beta\langle x_{t}^{k},{\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top}\phi(s_{t}^{k})\delta_{t}^{k}\rangle (i.e., I_{1} in Eq. ( D.1 )). However, the environmental heterogeneity significantly complicates the characterization of this term. Specifically, by Proposition 1 , I_{1} can be written as: \displaystyle 2\frac{\beta}{L}\langle x_{t}^{k},{\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k})\delta_{t,L}^{k}\rangle \displaystyle= \displaystyle 2\frac{\beta}{L}\langle x_{t}^{k},{\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top}\xi_{t,L}^{k}\rangle+2\frac{\beta}{L}\langle x_{t}^{k},{\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top}A_{L}^{k}x_{t}^{k}\rangle+2\beta\langle x_{t}^{k},{\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top}y_{t}^{k}\phi(s_{t,0}^{k})\rangle \displaystyle= \displaystyle\underbrace{2\frac{\beta}{L}\langle x_{t}^{k},{\mathbf{P}}_{t}A_{L}^{k}x_{t}^{k}\rangle}_{I_{11}}+\underbrace{2\frac{\beta}{L}\langle x_{t}^{k},{\mathbf{P}}_{t}\xi_{t,L}^{k}\rangle}_{I_{12}}+\underbrace{2{\beta}\langle x_{t}^{k},{\mathbf{P}}_{t}y_{t}^{k}\phi(s_{t,0}^{k})\rangle}_{I_{13}}, (34) recalling that {\mathbf{P}}_{t}={\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top} . Inside I_{1} , the main negative drift arises from the term \langle x_{t}^{k},{\mathbf{P}}_{t}A_{L}^{k}x_{t}^{k}\rangle . Intuitively, in traditional single-agent or homogeneous environment settings, \langle x_{t}^{k},{\mathbf{P}}_{t}A_{L}^{k}x_{t}^{k}\rangle\approx\langle x_{t}^{k},Ax_{t}^{k}\rangle , which is mainly controlled by the spectrum of A , with the desired property directly assumed in Assumption 1 . However, in the presence of environmental heterogeneity, Assumption 1 does not directly guarantee a negative drift of the x_{t}^{k} due to the existence of {\mathbf{P}}_{t} and its intricate interplay with the heterogeneous A_{L}^{k} . Specifically, (1) {\mathbf{P}}_{t} is not of full rank, (2) the local head error will be distorted in a different manner due to the product {\mathbf{P}}_{t}A_{L}^{k} , and (3) {\mathbf{P}}_{t} varies over time. To address this, we further decompose I_{11} as \displaystyle 2\frac{\beta}{L}\langle x_{t}^{k},{\mathbf{P}}_{t}A_{L}^{k}x_{t}^{k}\rangle \displaystyle= \displaystyle 2\frac{\beta}{L}\langle x_{t}^{k},({\mathbf{P}}_{t}-{\mathbf{P}}^{*})A_{L}^{k}x_{t}^{k}\rangle+2\frac{\beta}{L}\langle x_{t}^{k},{\mathbf{P}}^{*}A_{L}^{k}x_{t}^{k}\rangle \displaystyle= \displaystyle\underbrace{2\frac{\beta}{L}\langle x_{t}^{k},A_{L}^{k}x_{t}^{k}\rangle}_{I_{111}}+\underbrace{2\frac{\beta}{L}\langle x_{t}^{k},({\mathbf{P}}_{t}-{\mathbf{P}}^{*})A_{L}^{k}x_{t}^{k}\rangle}_{I_{112}}-\underbrace{2\frac{\beta}{L}\langle x_{t}^{k},{\mathbf{P}}_{\perp}^{*}A_{L}^{k}x_{t}^{k}\rangle}_{I_{113}}. (35) The first term of Eq. ( D.1 ), I_{111} , is a contraction by Assumption 1 . Specifically, \displaystyle 2\frac{\beta}{L}\langle x_{t}^{k},A_{L}^{k}x_{t}^{k}\rangle\leq-2\lambda\beta\|x_{t}^{k}\|^{2}. I_{112} can be controlled via the principal angle distance between the subspace estimate {\mathbf{B}}_{t} and the underlying truth {\mathbf{B}}^{*} . Since \displaystyle\|A_{L}^{k}\|\leq\mathbb{E}_{s^{0}\sim\mu^{k},a^{0}\sim\pi,...,s^{(L)}\sim P^{k}}\left\|{\phi(s^{(0)})(\phi(s^{(L)})-\phi(s^{(0)}))^{\top}}\right\|\leq 2, (36) it holds that \displaystyle 2\frac{\beta}{L}\langle x_{t}^{k},({\mathbf{P}}_{t}-{\mathbf{P}}^{*})A_{L}^{k}x_{t}^{k}\rangle\leq 8U_{\omega}\frac{\beta}{L}\|{\mathbf{P}}_{t}-{\mathbf{P}}^{*}\|\|x_{t}^{k}\|. The third term, I_{113} , can be bounded as \displaystyle-2\frac{\beta}{L}\langle x_{t}^{k},{\mathbf{P}}_{\perp}^{*}A_{L}^{k}x_{t}^{k}\rangle \displaystyle=-2\frac{\beta}{L}({\mathbf{P}}_{\perp}^{*}x_{t}^{k})^{\top}A_{L}^{k}x_{t}^{k}=-2\frac{\beta}{L}({\mathbf{P}}_{\perp}^{*}\left({\mathbf{B}}_{t}\omega_{t}^{k}-{\mathbf{B}}^{*}\omega^{k,*}\right))^{\top}A_{L}^{k}x_{t}^{k} \displaystyle=-2\frac{\beta}{L}\left({\mathbf{B}}_{\perp}^{*}{\mathbf{B}}_{\perp}^{*\top}\left({\mathbf{B}}_{t}\omega_{t}^{k}-{\mathbf{B}}^{*}\omega^{k,*}\right)\right)^{\top}A_{L}^{k}x_{t}^{k} \displaystyle=-2\frac{\beta}{L}({\mathbf{B}}_{\perp}^{*}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{B}}_{t}\omega_{t}^{k})^{\top}A_{L}^{k}x_{t}^{k} \displaystyle\leq 4\frac{\beta}{L}U_{\omega}\|{\mathbf{B}}_{\perp}^{*\top}{\mathbf{B}}_{t}\|\|x_{t}^{k}\|. (37) wherein the local head error x_{t}^{k} and the principal angle distance \|{\mathbf{B}}_{\perp}^{*\top}{\mathbf{B}}_{t}\| are also coupled. Then, \displaystyle I_{11}\leq-2\lambda\beta\|x_{t}^{k}\|^{2}+12U_{\omega}\frac{\beta}{L}\|m_{t}\|\|x_{t}^{k}\|. (38) It is worth noting that when \|{\mathbf{B}}_{\perp}^{*\top}{\mathbf{B}}_{t}\|=0 , i.e., when {\mathbf{B}}_{t} and {\mathbf{B}}^{*} span the same r -dimensional subspace, the second and the third terms in Eq. ( D.1 ) become zero, and Eq. ( D.1 ) reduces to the standard negative drift term. We postpone the characterization of the convergence of \|{\mathbf{B}}_{\perp}^{*\top}{\mathbf{B}}_{t}\| to Lemma 5.3 . For I_{13} ,we have \displaystyle I_{13}\leq 2\beta\|x_{t}^{k}\||y_{t}^{k}|. (39) Recall \widetilde{b}_{t,L}^{k}=\sum_{\ell=0}^{L-1}(r_{t,\ell}^{k}-J^{k})\phi(s_{t,0}^{k});\quad\bar{b}_{L}^{k}={\mathbb{E}}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left[\sum_{\ell=0}^{L-1}(r(s^{(\ell)},a^{(\ell)})-J^{k})\phi(s^{(0)})\right]. For I_{12} , recall from Eq. ( 15 ) that, \displaystyle\xi_{t,L}^{k}=\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}+(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t}\omega_{t}^{k}. (40) Then, \displaystyle 2\frac{\beta}{L}\left\langle x_{t}^{k},{\mathbf{P}}_{t}\xi_{t,L}^{k}\right\rangle \displaystyle= \displaystyle 2\frac{\beta}{L}\left\langle x_{t}^{k},{\mathbf{P}}_{t}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}+(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t}\omega_{t}^{k}\big]\right\rangle \displaystyle= \displaystyle 2\frac{\beta}{L}\left\langle x_{t}^{k},{\mathbf{P}}_{t}(\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k})\right\rangle+2\frac{\beta}{L}\left\langle x_{t}^{k},{\mathbf{P}}_{t}(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t}\omega_{t}^{k}\right\rangle \displaystyle= \displaystyle\underbrace{2\frac{\beta}{L}\left\langle x_{t}^{k}-x_{t-\tau}^{k},{\mathbf{P}}_{t}(\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k})\right\rangle}_{I_{121}}+\underbrace{2\frac{\beta}{L}\left\langle x_{t-\tau}^{k},({\mathbf{P}}_{t}-{\mathbf{P}}_{t-\tau})(\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k})\right\rangle}_{I_{122}}+\underbrace{2\frac{\beta}{L}\left\langle x_{t-\tau}^{k},{\mathbf{P}}_{t-\tau}(\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k})\right\rangle}_{I_{123}} \displaystyle+\underbrace{2\frac{\beta}{L}\left\langle x_{t}^{k}-x_{t-\tau}^{k},{\mathbf{P}}_{t}(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t}\omega_{t}^{k}\right\rangle}_{I_{124}}+\underbrace{2\frac{\beta}{L}\left\langle x_{t-\tau}^{k},({\mathbf{P}}_{t}-{\mathbf{P}}_{t-\tau})(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t}\omega_{t}^{k}\right\rangle}_{I_{125}} \displaystyle+\underbrace{2\frac{\beta}{L}\left\langle x_{t-\tau}^{k},{\mathbf{P}}_{t-\tau}(\widetilde{A}_{t,L}^{k}-A_{L}^{k})({\mathbf{B}}_{t}\omega_{t}^{k}-{\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k})\right\rangle}_{I_{126}}+\underbrace{2\frac{\beta}{L}\left\langle x_{t-\tau}^{k},{\mathbf{P}}_{t-\tau}(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}\right\rangle}_{I_{127}}. For I_{121} , we have under conditional expectation, \displaystyle 2\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\left\langle x_{t}^{k}-x_{t-\tau}^{k},{\mathbf{P}}_{t}\big(\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big)\right\rangle \displaystyle\leq \displaystyle 2\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}-x_{t-\tau}^{k}\|\|{\mathbf{P}}_{t}\|\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\| \displaystyle\leq \displaystyle 2\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}\omega_{t}^{k}-{\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}\|\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\| \displaystyle\leq \displaystyle\frac{1}{2}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}\omega_{t}^{k}-{\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}\|^{2}+2\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\|^{2} \displaystyle\leq \displaystyle{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}\omega_{t}^{k}-{\mathbf{B}}_{t}\omega_{t-\tau}^{k}\|^{2}+{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}\omega_{t-\tau}^{k}-{\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}\|^{2}+2\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\|^{2} \displaystyle\leq \displaystyle{\mathbb{E}}_{t-\tau}\|\omega_{t}^{k}-\omega_{t-\tau}^{k}\|^{2}+U_{\omega}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2}+2\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\|^{2}. (41) Similarly, for I_{122} , we have \displaystyle 2\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\left\langle x_{t-\tau}^{k},({\mathbf{P}}_{t}-{\mathbf{P}}_{t-\tau})\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big]\right\rangle \displaystyle\leq \displaystyle 2\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\|x_{t-\tau}^{k}\|\|{\mathbf{P}}_{t}-{\mathbf{P}}_{t-\tau}\|\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\|~~~~(\text{by Cauchy Schwartz}) \displaystyle\leq \displaystyle 8U_{\omega}\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\| \displaystyle\leq \displaystyle\frac{1}{2}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2}+32U_{\omega}^{2}\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\|^{2}. (42) For I_{124} , \displaystyle 2\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\left\langle x_{t}^{k}-x_{t-\tau}^{k},{\mathbf{P}}_{t}(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t}\omega_{t}^{k}\right\rangle \displaystyle\leq \displaystyle 2U_{\omega}\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}\omega_{t}^{k}-{\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}\|\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\| \displaystyle\leq \displaystyle\frac{1}{2}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}\omega_{t}^{k}-{\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}\|^{2}+2U_{\omega}^{2}\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2} \displaystyle\leq \displaystyle{\mathbb{E}}_{t-\tau}\|\omega_{t}^{k}-\omega_{t-\tau}^{k}\|^{2}+U_{\omega}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2}+2U_{\omega}^{2}\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2}. (43) For I_{125} , \displaystyle 2\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\left\langle x_{t-\tau}^{k},({\mathbf{P}}_{t}-{\mathbf{P}}_{t-\tau})(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t}\omega_{t}^{k}\right\rangle \displaystyle\leq \displaystyle 8\frac{\beta}{L}U_{\omega}^{2}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\| \displaystyle\leq \displaystyle\frac{1}{2}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2}+32U_{\omega}^{4}\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2}. (44) For I_{126} , we have, \displaystyle 2\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\left\langle x_{t-\tau}^{k},{\mathbf{P}}_{t-\tau}(\widetilde{A}_{t,L}^{k}-A_{L}^{k})({\mathbf{B}}_{t}\omega_{t}^{k}-{\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k})\right\rangle \displaystyle\leq \displaystyle 4\frac{\beta}{L}U_{\omega}{\mathbb{E}}_{t-\tau}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|\|{\mathbf{B}}_{t}\omega_{t}^{k}-{\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}\| \displaystyle\leq \displaystyle 8\frac{\beta^{2}}{L^{2}}U_{\omega}^{2}{\mathbb{E}}_{t-\tau}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2}+{\mathbb{E}}_{t-\tau}\|\omega_{t}^{k}-\omega_{t-\tau}^{k}\|^{2}+U_{\omega}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2} (45) Summing up the upper bounds of I_{121} (Eq. ( D.1 )) ,I_{122} (Eq. ( D.1 )) ,I_{124} (Eq. ( D.1 )), I_{125} (Eq. ( D.1 )) and I_{126} (Eq. ( D.1 )), conditioning on v_{t-\tau} , we get, \displaystyle{\mathbb{E}}_{t-\tau}\left[I_{121}+I_{122}+I_{124}+I_{125}+I_{126}\right] \displaystyle\leq \displaystyle 3{\mathbb{E}}_{t-\tau}\|\omega_{t}^{k}-\omega_{t-\tau}^{k}\|^{2}+\left(1+3U_{\omega}\right){\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2} \displaystyle+\left(10U_{\omega}^{2}+32U_{\omega}^{4}\right)\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2}+\left(2+32U_{\omega}^{2}\right)\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\|^{2} \displaystyle\leq \displaystyle 3\tau\sum_{j=t-\tau}^{t-1}{\mathbb{E}}_{t-\tau}\|\omega_{j+1}^{k}-\omega_{j}^{k}\|^{2}+\left(1+3U_{\omega}\right)\tau\sum_{j=t-\tau}^{t-1}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{j+1}-{\mathbf{B}}_{j}\|^{2} \displaystyle+\left(10U_{\omega}^{2}+32U_{\omega}^{4}\right)\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2}+\left(2+32U_{\omega}^{2}\right)\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\|^{2}. (46) For I_{123} , by Assumption 2 and Lemma I.6 , we have \displaystyle 2\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\left\langle x_{t-\tau}^{k},{\mathbf{P}}_{t-\tau}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big]\right\rangle \displaystyle\leq 2\frac{\beta}{L}\left\langle x_{t-\tau}^{k},{\mathbf{P}}_{t-\tau}{\mathbb{E}}_{t-\tau}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big]\right\rangle \displaystyle\leq 2\frac{\beta}{L}\|x_{t-\tau}^{k}\|\|{\mathbf{P}}_{t-\tau}\|\|{\mathbb{E}}_{t-\tau}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big]\| \displaystyle\leq 2\frac{\beta}{L}2U_{\omega}4LU_{r}m\rho^{\tau}=16U_{\omega}U_{r}\beta m\rho^{\tau}. For I_{127} , similarly, by Assumption 2 and Lemma I.6 , we have \displaystyle 2\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\left\langle x_{t-\tau}^{k},{\mathbf{P}}_{t-\tau}(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}\right\rangle\leq 16U_{\omega}^{2}\frac{\beta}{L}m\rho^{\tau}. Then, by the tower property, we have \displaystyle{\mathbb{E}}_{t-\tau}[I_{123}+I_{127}]\leq 16U_{\omega}(U_{r}+\frac{U_{\omega}}{L})\beta m\rho^{\tau}\leq 16U_{\omega}(U_{r}+U_{\omega})\beta m\rho^{\tau}. (47) Putting Eq. ( 47 ) and Eq. ( D.1 ) together, we get \displaystyle{\mathbb{E}}_{t-\tau}[I_{12}]\leq 3\tau\sum_{j=t-\tau}^{t-1}{\mathbb{E}}_{t-\tau}\|\omega_{j+1}^{k}-\omega_{j}^{k}\|^{2}+\left(1+3U_{\omega}\right)\tau\sum_{j=t-\tau}^{t-1}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{j+1}-{\mathbf{B}}_{j}\|^{2} \displaystyle+\left(10U_{\omega}^{2}+32U_{\omega}^{4}\right)\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2}+\left(2+32U_{\omega}^{2}\right)\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\|^{2}+16U_{\omega}(U_{r}+U_{\omega})\beta m\rho^{\tau}. (48) Invoking Assumption 1 , putting Eq. ( 38 ), Eq. ( D.1 ), and Eq. ( 39 ), back to Eq. ( 34 ), and taking conditional expectation on v_{0:t-\tau} , we get \displaystyle{\mathbb{E}}_{t-\tau}[I_{1}]=2\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\langle x_{t}^{k},{\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k})\delta_{t,L}^{k}\rangle \displaystyle\leq \displaystyle-2\lambda\beta{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+12U_{\omega}\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\|m_{t}\|\|x_{t}^{k}\|+2\beta{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\||y_{t}^{k}| \displaystyle+3\tau\sum_{j=t-\tau}^{t-1}{\mathbb{E}}_{t-\tau}\|\omega_{j+1}^{k}-\omega_{j}^{k}\|^{2}+\left(1+3U_{\omega}\right)\tau\sum_{j=t-\tau}^{t-1}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{j+1}-{\mathbf{B}}_{j}\|^{2} \displaystyle+\left(10U_{\omega}^{2}+32U_{\omega}^{4}\right)\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2}+\left(2+32U_{\omega}^{2}\right)\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\|^{2}+16U_{\omega}(U_{r}+U_{\omega})\beta m\rho^{\tau}. (49) Apply Young’s inequality, for arbitrary c_{3}>0 , c_{4}>0 , we have \displaystyle 12U_{\omega}\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\|m_{t}\|\|x_{t}^{k}\|\leq c_{3}^{2}\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+\frac{36U_{\omega}^{2}}{c_{3}^{2}}\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\|m_{t}\|^{2}, \displaystyle 2\beta{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\||y_{t}^{k}|\leq c_{4}^{2}\beta{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+\frac{1}{c_{4}^{2}}\beta{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}, we get \displaystyle{\mathbb{E}}_{t-2\tau}[I_{1}]\leq-2\lambda\beta{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|^{2}+c_{3}^{2}\frac{\beta}{L}{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|^{2}+\frac{36U_{\omega}^{2}}{c_{3}^{2}}\frac{\beta}{L}{\mathbb{E}}_{t-2\tau}\|m_{t}\|^{2}+c_{4}^{2}\beta{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|^{2}+\frac{1}{c_{4}^{2}}\beta{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|^{2} \displaystyle+3\tau\sum_{j=t-\tau}^{t-1}{\mathbb{E}}_{t-2\tau}\|\omega_{j+1}^{k}-\omega_{j}^{k}\|^{2}+\left(1+3U_{\omega}\right)\tau\sum_{j=t-\tau}^{t-1}{\mathbb{E}}_{t-2\tau}\|{\mathbf{B}}_{j+1}-{\mathbf{B}}_{j}\|^{2} \displaystyle+\left(10U_{\omega}^{2}+32U_{\omega}^{4}\right)\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-2\tau}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2}+\left(2+32U_{\omega}^{2}\right)\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-2\tau}\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\|^{2}+16U_{\omega}(U_{r}+U_{\omega})\beta m\rho^{\tau}. (50) Bounding I_{2} . By Proposition 1 , I_{2} can be written as, \displaystyle\frac{2\zeta}{KL}\langle x_{t}^{k},\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}\phi(s_{t,0}^{i})\delta_{t,L}^{i}(\omega_{t}^{i})^{\top}\omega_{t}^{k}\rangle \displaystyle= \displaystyle\frac{2\zeta}{KL}\langle x_{t}^{k},\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}\widetilde{A}_{t,L}^{i}x_{t}^{i}(\omega_{t}^{i})^{\top}\omega_{t}^{k}\rangle+\frac{2\zeta}{KL}\langle x_{t}^{k},\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}{\mathbf{b}}_{t,L}^{i}(\omega_{t}^{i})^{\top}\omega_{t}^{k}\rangle+\frac{2\zeta}{K}\langle x_{t}^{k},\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}y_{t}^{i}\phi(s_{t,0}^{i})(\omega_{t}^{i})^{\top}\omega_{t}^{k}\rangle \displaystyle\leq \displaystyle\frac{4U_{\omega}^{2}\zeta}{KL}\sum_{i=1}^{K}\|x_{t}^{k}\|\|x_{t}^{i}\|+\underbrace{\frac{2\zeta}{KL}\langle x_{t}^{k},\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}{\mathbf{b}}_{t,L}^{i}(\omega_{t}^{i})^{\top}\omega_{t}^{k}\rangle}_{I_{21}}+\frac{2U_{\omega}^{2}\zeta}{K}\sum_{i=1}^{K}\|x_{t}^{k}\||y_{t}^{i}|. (51) For I_{21} , similar to the derivations of the upper bounds of terms from I_{121} to I_{126} , we have \displaystyle\frac{2\zeta}{KL}\sum_{i=1}^{K}\langle x_{t}^{k},{\mathbf{P}}_{t,\perp}{\mathbf{b}}_{t,L}^{i}(\omega_{t}^{i})^{\top}\omega_{t}^{k}\rangle \displaystyle= \displaystyle\frac{2\zeta}{KL}\sum_{i=1}^{K}\langle x_{t}^{k}-x_{t-\tau}^{k},{\mathbf{P}}_{t,\perp}{\mathbf{b}}_{t,L}^{i}(\omega_{t}^{i})^{\top}\omega_{t}^{k}\rangle+\frac{2\zeta}{KL}\sum_{i=1}^{K}\langle x_{t-\tau}^{k},({\mathbf{P}}_{t,\perp}-{\mathbf{P}}_{t-\tau,\perp}){\mathbf{b}}_{t,L}^{i}(\omega_{t}^{i})^{\top}\omega_{t}^{k}\rangle \displaystyle+\frac{2\zeta}{KL}\sum_{i=1}^{K}\langle x_{t-\tau}^{k},{\mathbf{P}}_{t-\tau,\perp}{\mathbf{b}}_{t,L}^{i}(\omega_{t}^{i}-\omega_{t-\tau}^{i})^{\top}\omega_{t}^{k}\rangle+\frac{2\zeta}{KL}\sum_{i=1}^{K}\langle x_{t-\tau}^{k},{\mathbf{P}}_{t-\tau,\perp}{\mathbf{b}}_{t,L}^{i}(\omega_{t-\tau}^{i})^{\top}(\omega_{t}^{k}-\omega_{t-\tau}^{k})\rangle \displaystyle+\frac{2\zeta}{KL}\sum_{i=1}^{K}\langle x_{t-\tau}^{k},{\mathbf{P}}_{t-\tau,\perp}{\mathbf{b}}_{t,L}^{i}(\omega_{t-\tau}^{i})^{\top}\omega_{t-\tau}^{k}\rangle \displaystyle\leq \displaystyle\frac{3}{2}\|\omega_{t}^{k}-\omega_{t-\tau}^{k}\|^{2}+\frac{3}{2}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2}+\left(18U_{\omega}^{4}+32U_{\omega}^{6}\right)\frac{\zeta^{2}}{KL^{2}}\sum_{i=1}^{K}\|{\mathbf{b}}_{t,L}^{i}\|^{2}+\frac{1}{2}\frac{1}{K}\sum_{i=1}^{K}\|\omega_{t}^{i}-\omega_{t-\tau}^{i}\|^{2} \displaystyle+\frac{2\zeta}{KL}\sum_{i=1}^{K}\langle x_{t-\tau}^{k},{\mathbf{P}}_{t-\tau,\perp}{\mathbf{b}}_{t,L}^{i}(\omega_{t-\tau}^{i})^{\top}\omega_{t-\tau}^{k}\rangle. (52) For the last term in Eq. ( D.1 ), by Cauchy–Schwarz inequality and Lemma I.6 , we have \displaystyle\frac{2\zeta}{KL}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\langle x_{t-\tau}^{k},{\mathbf{P}}_{t-\tau,\perp}{\mathbf{b}}_{t,L}^{i}(\omega_{t-\tau}^{i})^{\top}\omega_{t-\tau}^{k}\rangle \displaystyle\leq\frac{2\zeta}{KL}\sum_{i=1}^{K}2U_{\omega}^{3}\|{\mathbb{E}}_{t-\tau}[{\mathbf{b}}_{t,L}^{i}]\| \displaystyle\leq 8\frac{U_{\delta,L}}{L}U_{\omega}^{3}\zeta m\rho^{\tau}. (53) Then, putting Eq. ( D.1 ) back to Eq. ( D.1 ) and then putting the result back to Eq. ( D.1 ) and using tower property, we have \displaystyle{\mathbb{E}}_{t-\tau}[I_{2}]\leq 4U_{\omega}^{2}\frac{\zeta}{KL}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|\|x_{t}^{i}\|+2U_{\omega}^{2}\frac{\zeta}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\||y_{t}^{i}|+\frac{3}{2}{\mathbb{E}}_{t-\tau}\|\omega_{t}^{k}-\omega_{t-\tau}^{k}\|^{2} \displaystyle+\frac{3}{2}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2}+\left(18U_{\omega}^{4}+32U_{\omega}^{6}\right)\frac{\zeta^{2}}{KL^{2}}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|{\mathbf{b}}_{t,L}^{i}\|^{2}+\frac{1}{2}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|\omega_{t}^{i}-\omega_{t-\tau}^{i}\|^{2} \displaystyle+8\frac{U_{\delta,L}}{L}U_{\omega}^{3}\zeta m\rho^{\tau}. (54) Applying Young’s inequality, for arbitrary c_{5}>0 and c_{6}>0 , we have \displaystyle 4U_{\omega}^{2}\frac{\zeta}{KL}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|\|x_{t}^{i}\|\leq c_{5}^{2}\frac{\zeta}{L}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+\frac{4U_{\omega}^{4}}{c_{5}^{2}}\frac{\zeta}{KL}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{i}\|^{2}, \displaystyle 2U_{\omega}^{2}\frac{\zeta}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\||y_{t}^{i}|\leq c_{6}^{2}\zeta{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+\frac{U_{\omega}^{4}}{c_{6}^{2}}\frac{\zeta}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{i}|^{2}. We get \displaystyle{\mathbb{E}}_{t-\tau}[I_{2}]\leq c_{5}^{2}\frac{\zeta}{L}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+\frac{4U_{\omega}^{4}}{c_{5}^{2}}\frac{\zeta}{KL}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{i}\|^{2}+c_{6}^{2}\zeta{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+\frac{U_{\omega}^{4}}{c_{6}^{2}}\frac{\zeta}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{i}|^{2} \displaystyle+\frac{3}{2}{\mathbb{E}}_{t-\tau}\|\omega_{t}^{k}-\omega_{t-\tau}^{k}\|^{2}+\frac{3}{2}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2}+\left(18U_{\omega}^{4}+32U_{\omega}^{6}\right)\frac{\zeta^{2}}{KL^{2}}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|{\mathbf{b}}_{t,L}^{i}\|^{2} \displaystyle+\frac{1}{2}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|\omega_{t}^{i}-\omega_{t-\tau}^{i}\|^{2}+8\frac{U_{\delta,L}}{L}U_{\omega}^{3}\zeta m\rho^{\tau}. (55) For I_{3} , by Proposition 1 , we have \displaystyle I_{3}= \displaystyle\underbrace{\left\langle x_{t}^{k},\left(\frac{1}{K}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}\delta_{t,L}^{i}\phi(s_{t,0}^{i})(\omega_{t}^{i})^{\top}\right)\left({\mathbf{B}}_{t}^{\top}\widetilde{A}_{t,L}^{k}x_{t}^{k}\right)\right\rangle}_{I_{31}}+\underbrace{\left\langle x_{t}^{k},\left(\frac{1}{K}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}\delta_{t,L}^{i}\phi(s_{t,0}^{i})(\omega_{t}^{i})^{\top}\right)\left({\mathbf{B}}_{t}^{\top}Ly_{t}^{k}\phi(s_{t,0}^{k})\right)\right\rangle}_{I_{32}} \displaystyle+\underbrace{\left\langle x_{t}^{k},\left(\frac{1}{K}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}\delta_{t,L}^{i}\phi(s_{t,0}^{i})(\omega_{t}^{i})^{\top}\right)\left({\mathbf{B}}_{t}^{\top}{\mathbf{b}}_{t,L}^{k}\right)\right\rangle}_{I_{33}}. By Lemma H.2 , I_{31} and I_{32} can be bounded as \displaystyle I_{31} \displaystyle\leq 2U_{\delta,L}U_{\omega}\|x_{t}^{k}\|^{2}, (56) \displaystyle I_{32} \displaystyle\leq LU_{\delta,L}U_{\omega}\|x_{t}^{k}\||y_{t}^{k}|. (57) For I_{33} , we have, \displaystyle I_{33}= \displaystyle\left\langle x_{t}^{k},\left(\frac{1}{K}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}\widetilde{A}_{t,L}^{i}x_{t}^{i}(\omega_{t}^{i})^{\top}\right)\left({\mathbf{B}}_{t}^{\top}{\mathbf{b}}_{t,L}^{k}\right)\right\rangle+\left\langle x_{t}^{k},\left(\frac{1}{K}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}Ly_{t}^{i}\phi(s_{t,0}^{i})(\omega_{t}^{i})^{\top}\right)\left({\mathbf{B}}_{t}^{\top}{\mathbf{b}}_{t,L}^{k}\right)\right\rangle \displaystyle+\left\langle x_{t}^{k},\left(\frac{1}{K}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}{\mathbf{b}}_{t,L}^{i}(\omega_{t}^{i})^{\top}\right)\left({\mathbf{B}}_{t}^{\top}{\mathbf{b}}_{t,L}^{k}\right)\right\rangle \displaystyle\leq \displaystyle 2U_{\delta,L}U_{\omega}\frac{1}{K}\sum_{i=1}^{K}\|x_{t}^{k}\|\|x_{t}^{i}\|+LU_{\delta,L}U_{\omega}\frac{1}{K}\sum_{i=1}^{K}\|x_{t}^{k}\||y_{t}^{i}|+\left\langle x_{t}^{k},\left(\frac{1}{K}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}{\mathbf{b}}_{t,L}^{i}(\omega_{t}^{i})^{\top}\right)\left({\mathbf{B}}_{t}^{\top}{\mathbf{b}}_{t,L}^{k}\right)\right\rangle. (58) For the last term, we have, \displaystyle{\mathbb{E}}_{t-\tau}\left\langle x_{t}^{k},\left(\frac{1}{K}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}{\mathbf{b}}_{t,L}^{i}(\omega_{t}^{i})^{\top}\right)\left({\mathbf{B}}_{t}^{\top}{\mathbf{b}}_{t,L}^{k}\right)\right\rangle \displaystyle= \displaystyle{\mathbb{E}}_{t-\tau}(x_{t}^{k}-x_{t-\tau}^{k})^{\top}{\mathbf{P}}_{t,\perp}\frac{1}{K}\sum_{i=1}^{K}{\mathbf{b}}_{t,L}^{i}(\omega_{t}^{i})^{\top}\left({\mathbf{B}}_{t}^{\top}{\mathbf{b}}_{t,L}^{k}\right) \displaystyle+{\mathbb{E}}_{t-\tau}(x_{t-\tau}^{k})^{\top}({\mathbf{P}}_{t,\perp}-{\mathbf{P}}_{t-\tau,\perp})\frac{1}{K}\sum_{i=1}^{K}{\mathbf{b}}_{t,L}^{i}(\omega_{t}^{i})^{\top}\left({\mathbf{B}}_{t}^{\top}{\mathbf{b}}_{t,L}^{k}\right) \displaystyle+{\mathbb{E}}_{t-\tau}(x_{t-\tau}^{k})^{\top}{\mathbf{P}}_{t-\tau,\perp}\frac{1}{K}\sum_{i=1}^{K}{\mathbf{b}}_{t,L}^{i}(\omega_{t}^{i}-\omega_{t-\tau}^{i})^{\top}\left({\mathbf{B}}_{t}^{\top}{\mathbf{b}}_{t,L}^{k}\right) \displaystyle+{\mathbb{E}}_{t-\tau}(x_{t-\tau}^{k})^{\top}{\mathbf{P}}_{t-\tau,\perp}\frac{1}{K}\sum_{i=1}^{K}{\mathbf{b}}_{t,L}^{i}(\omega_{t-\tau}^{i})^{\top}\left(({\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau})^{\top}{\mathbf{b}}_{t,L}^{k}\right) \displaystyle+{\mathbb{E}}_{t-\tau}(x_{t-\tau}^{k})^{\top}{\mathbf{P}}_{t-\tau,\perp}\frac{1}{K}\sum_{i=1}^{K}{\mathbf{b}}_{t,L}^{i}(\omega_{t-\tau}^{i})^{\top}\left({\mathbf{B}}_{t-\tau}^{\top}{\mathbf{b}}_{t,L}^{k}\right)~~~~~(\text{bridging}) \displaystyle\leq \displaystyle{\mathbb{E}}_{t-\tau}\|\omega_{t}^{k}-\omega_{t-\tau}^{k}\|U_{\delta,L}^{2}U_{\omega}+7U_{\delta,L}^{2}U_{\omega}^{2}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|+2U_{\omega}U_{\delta,L}^{2}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|\omega_{t}^{i}-\omega_{t-\tau}^{i}\| \displaystyle+2U_{\omega}^{2}U_{\delta,L}^{2}\frac{1}{K}+{\mathbb{E}}_{t-\tau}(x_{t-\tau}^{k})^{\top}{\mathbf{P}}_{t-\tau,\perp}\frac{1}{K}\sum_{i\neq k}{\mathbf{b}}_{t,L}^{i}(\omega_{t-\tau}^{i})^{\top}\left({\mathbf{B}}_{t-\tau}^{\top}{\mathbf{b}}_{t,L}^{k}\right) \displaystyle\leq \displaystyle 3\tau U_{\omega}\frac{U_{\delta,L}^{3}}{L}\beta+14\frac{U_{\delta,L}^{3}}{L}U_{\omega}^{3}\tau\zeta+28\frac{U_{\delta,L}^{4}}{L^{2}}U_{\omega}^{4}\tau\zeta^{2}+2U_{\omega}^{2}U_{\delta,L}^{2}\frac{1}{K}+8U_{\omega}^{2}U_{\delta,L}^{2}m^{2}\rho^{2\tau}, where the last inequality follows from (a) the independence across agents, (b) Assumption 2 , and (c) Lemma I.1 . Putting this bound back to I_{33} (Eq. ( D.1 )), and taking conditional expectation, we get, \displaystyle{\mathbb{E}}_{t-\tau}[I_{33}]\leq 2U_{\delta,L}U_{\omega}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|\|x_{t}^{i}\|+LU_{\delta,L}U_{\omega}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\||y_{t}^{i}| \displaystyle+3\tau U_{\omega}\frac{U_{\delta,L}^{3}}{L}\beta+14\frac{U_{\delta,L}^{3}}{L}U_{\omega}^{3}\tau\zeta+28\frac{U_{\delta,L}^{4}}{L^{2}}U_{\omega}^{4}\tau\zeta^{2}+2U_{\omega}^{2}U_{\delta,L}^{2}\frac{1}{K}+8U_{\omega}^{2}U_{\delta,L}^{2}m^{2}\rho^{2\tau}. (59) Putting I_{31} (Eq. ( 56 )), I_{32} (Eq. ( 57 )), and I_{33} (Eq. ( D.1 )) back to I_{3} , and taking expectation, we have \displaystyle{\mathbb{E}}_{t-\tau}[I_{3}]\leq 2\frac{\zeta\beta}{L^{2}}\bigg(2U_{\delta,L}U_{\omega}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+LU_{\delta,L}U_{\omega}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\||y_{t}^{k}| \displaystyle\qquad+2U_{\delta,L}U_{\omega}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|\|x_{t}^{i}\|+LU_{\delta,L}U_{\omega}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\||y_{t}^{i}| \displaystyle\qquad+3\tau U_{\omega}\frac{U_{\delta,L}^{3}}{L}\beta+14\frac{U_{\delta,L}^{3}}{L}U_{\omega}^{3}\tau\zeta+28\frac{U_{\delta,L}^{4}}{L^{2}}U_{\omega}^{4}\tau\zeta^{2}+2U_{\omega}^{2}U_{\delta,L}^{2}\frac{1}{K}+8U_{\omega}^{2}U_{\delta,L}^{2}m^{2}\rho^{2\tau}\bigg) \displaystyle\leq \displaystyle 4\frac{U_{\delta,L}}{L}U_{\omega}\frac{\zeta\beta}{L}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+2\frac{U_{\delta,L}}{L}U_{\omega}\zeta\beta{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\||y_{t}^{k}| \displaystyle\qquad+4\frac{U_{\delta,L}}{L}U_{\omega}\frac{\zeta\beta}{L}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|\|x_{t}^{i}\|+2\frac{U_{\delta,L}}{L}U_{\omega}\zeta\beta\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\||y_{t}^{i}| \displaystyle\qquad+6\tau U_{\omega}\frac{U_{\delta,L}^{3}}{L^{3}}\zeta\beta^{2}+28\frac{U_{\delta,L}^{3}}{L^{3}}U_{\omega}^{3}\tau\zeta^{2}\beta+56\frac{U_{\delta,L}^{4}}{L^{4}}U_{\omega}^{4}\tau\zeta^{3}\beta+4\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{2}\frac{\zeta\beta}{K}+16\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{2}m^{2}\zeta\beta\rho^{2\tau} \displaystyle\leq \displaystyle\underbrace{(6\frac{U_{\delta,L}}{L^{2}}U_{\omega}+2\frac{U_{\delta,L}}{L}U_{\omega})}_{:=C_{31}}\zeta\beta{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+\underbrace{\frac{U_{\delta,L}}{L}U_{\omega}}_{:=C_{32}}\zeta\beta{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2} \displaystyle\qquad+\underbrace{2\frac{U_{\delta,L}}{L}U_{\omega}}_{:=C_{33}}\frac{\zeta\beta}{L}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{i}\|^{2}+\underbrace{\frac{U_{\delta,L}}{L}U_{\omega}}_{:=C_{34}}\zeta\beta\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{i}|^{2} \displaystyle+\underbrace{6\tau U_{\omega}\frac{U_{\delta,L}^{3}}{L^{3}}}_{:=C_{35}(\tau)}\zeta\beta^{2}+\underbrace{28\frac{U_{\delta,L}^{3}}{L^{3}}U_{\omega}^{3}\tau}_{C_{36}(\tau)}\zeta^{2}\beta+\underbrace{56\frac{U_{\delta,L}^{4}}{L^{4}}U_{\omega}^{4}\tau}_{C_{37}(\tau)}\zeta^{3}\beta+\underbrace{4\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{2}}_{C_{38}}\frac{\zeta\beta}{K}+4\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{2}m^{2}\zeta\beta\rho^{2\tau}~~~~(\text{Young's inequality}) \displaystyle= \displaystyle C_{31}\zeta\beta{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+C_{32}\zeta\beta{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}+C_{33}\frac{\zeta\beta}{L}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{i}\|^{2}+C_{34}\zeta\beta\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{i}|^{2} \displaystyle+C_{35}(\tau)\zeta\beta^{2}+C_{36}(\tau)\zeta^{2}\beta+C_{37}(\tau)\zeta^{3}\beta+C_{38}\frac{\zeta\beta}{K}+4U_{\omega}^{2}\frac{U_{\delta,L}^{2}}{L^{2}}m^{2}\zeta\beta\rho^{2\tau}. (60) For I_{4} , we have, \displaystyle\|{\mathbf{B}}_{t}(\delta_{t,L}^{k}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k}))\|^{2}\leq \displaystyle 3\|{\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top}(\widetilde{A}_{t,L}^{k}x_{t}^{k})\|^{2}+3L^{2}\|{\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top}y_{t}^{k}\phi(s_{t,0}^{k})\|^{2}+3\|{\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top}{\mathbf{b}}_{t,L}^{k}\|^{2}~~~~\text{(by Proposition \ref{prop: key intermediate})} \displaystyle\leq \displaystyle 12\|x_{t}^{k}\|^{2}+3L^{2}|y_{t}^{k}|^{2}+3\|{\mathbf{b}}_{t,L}^{k}\|^{2}. (61) From Lemma I.5 , we know \displaystyle{\mathbb{E}}_{t-\tau}\|{\mathbf{b}}_{t,L}^{k}\|^{2}\leq U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})L. (62) Then, taking conditional expectation on Eq. ( D.1 ), we get \displaystyle{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}(\delta_{t,L}^{k}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k}))\|^{2}\leq 12{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+3L^{2}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}+3U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})L. (63) Therefore, we have \displaystyle{\mathbb{E}}_{t-\tau}[I_{4}]\leq 36\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+9\beta^{2}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}+9U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})\frac{\beta^{2}}{L}. (64) For I_{5} , similarly, under conditional expectation, we have \displaystyle{\mathbb{E}}_{t-\tau}\|\frac{1}{K}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}\delta_{t,L}^{i}\phi(s_{t,0}^{i})(\omega_{t}^{i})^{\top}\omega_{t}^{k}\|^{2} \displaystyle\leq \displaystyle 3{\mathbb{E}}_{t-\tau}\|\frac{1}{K}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}\widetilde{A}_{t,L}^{i}x_{t}^{i}(\omega_{t}^{i})^{\top}\omega_{t}^{k}\|^{2}+3L^{2}{\mathbb{E}}_{t-\tau}\|\frac{1}{K}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}y_{t}^{i}\phi(s_{t,0}^{i})(\omega_{t}^{i})^{\top}\omega_{t}^{k}\|^{2} \displaystyle+3{\mathbb{E}}_{t-\tau}\|\frac{1}{K}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}{\mathbf{b}}_{t,L}^{i}(\omega_{t}^{i})^{\top}\omega_{t}^{k}\|^{2} \displaystyle\leq \displaystyle 12U_{\omega}^{4}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{i}\|^{2}+3L^{2}U_{\omega}^{4}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{i}|^{2}+3U_{\omega}^{4}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|{\mathbf{b}}_{t,L}^{i}\|^{2} \displaystyle\leq \displaystyle 12U_{\omega}^{4}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{i}\|^{2}+3L^{2}U_{\omega}^{4}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{i}|^{2}+3U_{\omega}^{4}U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})L.~~~~~(\text{by Eq.\,(\ref{eqn: tower property for bfb})}) Then, we have \displaystyle{\mathbb{E}}_{t-\tau}[I_{5}]\leq 36U_{\omega}^{4}\frac{\zeta^{2}}{L^{2}}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{i}\|^{2}+9\zeta^{2}U_{\omega}^{4}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{i}|^{2}+9U_{\omega}^{4}U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})\frac{\zeta^{2}}{L}. (65) For I_{6} , we have, \displaystyle I_{6}\leq 3\frac{\beta^{2}\zeta^{2}}{L^{4}}\|(\frac{1}{K}\sum_{i=1}^{K}{\mathbf{P}}_{t,\perp}\delta_{t,L}^{i}\phi(s_{t,0}^{i})(\omega_{t}^{i})^{\top})\left(\delta_{t,L}^{k}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k})\right)\|^{2}\leq 3\frac{U_{\delta,L}^{4}}{L^{4}}U_{\omega}^{2}\beta^{2}\zeta^{2}. (66) Plugging the upper bounds of I_{1} , I_{2} , I_{3} , I_{4} , I_{5} , and I_{6} (Eq. ( D.1 ), Eq. ( D.1 ), Eq. ( D.1 ), Eq. ( 64 ), Eq. ( 65 ), and Eq. ( 66 )) back to Eq. ( D.1 ), and taking expectation, we have \displaystyle{\mathbb{E}}_{t-\tau}\|\widetilde{x}_{t+1}^{k}\|^{2}\leq(1-2\lambda\beta){\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+c_{3}^{2}\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+\frac{36U_{\omega}^{2}}{c_{3}^{2}}\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\|m_{t}\|^{2}+c_{4}^{2}\beta{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+\frac{1}{c_{4}^{2}}\beta{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2} \displaystyle+3\tau\sum_{j=t-\tau}^{t-1}{\mathbb{E}}_{t-\tau}\|\omega_{j+1}^{k}-\omega_{j}^{k}\|^{2}+\left(1+3U_{\omega}\right)\tau\sum_{j=t-\tau}^{t-1}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{j+1}-{\mathbf{B}}_{j}\|^{2} \displaystyle+\left(10U_{\omega}^{2}+32U_{\omega}^{4}\right)\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2}+\left(2+32U_{\omega}^{2}\right)\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\|^{2}+16U_{\omega}(U_{r}+U_{\omega})m\beta\rho^{\tau} \displaystyle+c_{5}^{2}\frac{\zeta}{L}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+\frac{4U_{\omega}^{4}}{c_{5}^{2}}\frac{\zeta}{KL}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{i}\|^{2}+c_{6}^{2}\zeta{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+\frac{U_{\omega}^{4}}{c_{6}^{2}}\zeta\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{i}|^{2} \displaystyle+\frac{3}{2}{\mathbb{E}}_{t-\tau}\|\omega_{t}^{k}-\omega_{t-\tau}^{k}\|^{2}+\frac{3}{2}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2}+\left(18U_{\omega}^{4}+32U_{\omega}^{6}\right)U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})\frac{\zeta^{2}}{L} \displaystyle+\frac{1}{2K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|\omega_{t}^{i}-\omega_{t-\tau}^{i}\|^{2}+4\frac{U_{\delta,L}}{L}U_{\omega}^{3}\zeta m\rho^{\tau}+C_{31}\zeta\beta{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+C_{32}\zeta\beta{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}+C_{33}\frac{\zeta\beta}{L}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{i}\|^{2} \displaystyle+C_{34}\zeta\beta\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{i}|^{2}+C_{35}(\tau)\zeta\beta^{2}+C_{36}(\tau)\zeta^{2}\beta+C_{37}(\tau)\zeta^{3}\beta+C_{38}\frac{\zeta\beta}{K}+16U_{\omega}^{2}\frac{U_{\delta,L}^{2}}{L^{2}}m^{2}\zeta\beta\rho^{2\tau} \displaystyle+36\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+9\beta^{2}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}+9U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})\frac{\beta^{2}}{L} \displaystyle+36U_{\omega}^{4}\frac{\zeta^{2}}{L^{2}}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{i}\|^{2}+9\zeta^{2}U_{\omega}^{4}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{i}|^{2}+9U_{\omega}^{4}U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})\frac{\zeta^{2}}{L}+3\frac{U_{\delta,L}^{4}}{L^{4}}U_{\omega}^{2}\beta^{2}\zeta^{2}. Combining terms, we get, \displaystyle{\mathbb{E}}_{t-\tau}\|\widetilde{x}_{t+1}^{k}\|^{2}\leq\left(1-2\lambda\beta+\left(c_{3}^{2}\frac{\beta}{L}+c_{4}^{2}\beta+c_{5}^{2}\frac{\zeta}{L}+c_{6}^{2}\zeta+C_{31}\zeta\beta+36\frac{\beta^{2}}{L^{2}}\right)\right){\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2} \displaystyle+\frac{36U_{\omega}^{2}}{c_{3}^{2}}\frac{\beta}{L}{\mathbb{E}}_{t-\tau}\|m_{t}\|^{2}+\left(\frac{1}{c_{4}^{2}}\beta+C_{32}\zeta\beta+9\beta^{2}\right){\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2} \displaystyle+\left(\frac{4U_{\omega}^{4}}{c_{5}^{2}}\frac{\zeta}{L}+C_{33}\frac{\zeta\beta}{L}+36U_{\omega}^{4}\frac{\zeta^{2}}{L^{2}}\right)\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{i}\|^{2} \displaystyle+\left(\frac{U_{\omega}^{4}}{c_{6}^{2}}\zeta+C_{34}\zeta\beta+9\zeta^{2}U_{\omega}^{4}\right)\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{i}|^{2} \displaystyle+16U_{\omega}(U_{r}+U_{\omega})m\beta\rho^{\tau}+\left(18U_{\omega}^{4}+32U_{\omega}^{6}\right)U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})\frac{\zeta^{2}}{L}+4\frac{U_{\delta,L}}{L}U_{\omega}^{3}\zeta m\rho^{\tau} \displaystyle+C_{35}(\tau)\zeta\beta^{2}+C_{36}(\tau)\zeta^{2}\beta+C_{37}(\tau)\zeta^{3}\beta+C_{38}\frac{\zeta\beta}{K}+16U_{\omega}^{2}\frac{U_{\delta,L}^{2}}{L^{2}}m^{2}\zeta\beta\rho^{2\tau}+9U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})\frac{\beta^{2}}{L} \displaystyle+9U_{\omega}^{4}U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})\frac{\zeta^{2}}{L}+3\frac{U_{\delta,L}^{4}}{L^{4}}U_{\omega}^{2}\beta^{2}\zeta^{2} \displaystyle+\frac{9}{2}\tau\sum_{j=t-\tau}^{t-1}{\mathbb{E}}_{t-\tau}\|\omega_{j+1}^{k}-\omega_{j}^{k}\|^{2}+\left(\frac{5}{2}+3U_{\omega}\right)\tau\sum_{j=t-\tau}^{t-1}{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{j+1}-{\mathbf{B}}_{j}\|^{2} \displaystyle+\left(10U_{\omega}^{2}+32U_{\omega}^{4}\right)\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2}+\left(2+32U_{\omega}^{2}\right)\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\|^{2} \displaystyle+\frac{1}{2K}\tau\sum_{j=t-\tau}^{t-1}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|\omega_{j+1}^{i}-\omega_{j}^{i}\|^{2}. Applying Lemma I.5 , Lemma I.2 , Lemma I.3 , and using \beta=c\zeta we further get, \displaystyle{\mathbb{E}}_{t-\tau}\|\widetilde{x}_{t+1}^{k}\|^{2}\leq\left(1-2\lambda c\zeta+\left(c_{3}^{2}\frac{c\zeta}{L}+c_{4}^{2}c\zeta+c_{5}^{2}\frac{\zeta}{L}+c_{6}^{2}\zeta+C_{31}c\zeta^{2}+(36+40\tau^{2})\frac{c^{2}\zeta^{2}}{L^{2}}\right)\right){\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2} \displaystyle+\frac{36U_{\omega}^{2}}{c_{3}^{2}}\frac{c\zeta}{L}{\mathbb{E}}_{t-\tau}\|m_{t}\|^{2}+\left(\frac{1}{c_{4}^{2}}c\zeta+C_{32}c\zeta^{2}+(9+10\tau^{2})c^{2}\zeta^{2}\right){\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2} \displaystyle+\left(\frac{4U_{\omega}^{4}}{c_{5}^{2}}\frac{\zeta}{L}+C_{33}\frac{c\zeta^{2}}{L}+\left(36U_{\omega}^{4}+\left(\frac{5}{2}+3U_{\omega}\right)C_{18}\right)\frac{\zeta^{2}}{L^{2}}\right)\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{i}\|^{2} \displaystyle+\left(\frac{U_{\omega}^{4}}{c_{6}^{2}}\zeta+C_{34}c\zeta^{2}+\left(9U_{\omega}^{4}+\left(\frac{5}{2}+3U_{\omega}\right)C_{23}\right)\zeta^{2}\right)\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{i}|^{2} \displaystyle+C_{39}(\tau^{2})\frac{\zeta^{2}}{L}+C_{41}(\tau^{2})\frac{\zeta^{2}}{K}+C_{42}\zeta\rho^{\tau}+C_{43}(\tau)\zeta^{3}+C_{44}(\tau^{2})\zeta^{2}\rho^{2\tau}+C_{40}(\tau^{4})\zeta^{4}+C_{45}(\tau^{4})\gamma^{2}\zeta^{2}, (67) where \displaystyle C_{39}(\tau^{2})=\bigg[\left(10U_{\omega}^{2}+32U_{\omega}^{4}\right)\left(16+32\frac{m\rho}{1-\rho}\right)c^{2}+\left(2+32U_{\omega}^{2}\right)U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})c^{2} \displaystyle\qquad+\left(18U_{\omega}^{4}+32U_{\omega}^{6}\right)U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})+9U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})c^{2}+9U_{\omega}^{4}U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})+5\tau^{2}C_{7}c^{2}\bigg], \displaystyle C_{40}(\tau^{4})=\bigg[C_{37}(\tau)c+3\frac{U_{\delta,L}^{4}}{L^{4}}U_{\omega}^{2}c^{2}+5\tau^{2}C_{5}(\tau^{2})c^{4}\frac{1}{L^{2}}+5\tau^{2}C_{6}(\tau^{2})c^{2}\frac{1}{L^{2}} \displaystyle\qquad+\left(\frac{5}{2}+3U_{\omega}\right)\tau^{2}\bigg(C_{16}(\tau^{2})c^{2}\frac{1}{L^{2}}+C_{17}(\tau^{2})\frac{1}{L^{2}}+C_{19}(\tau^{2})c^{2}+C_{24}\bigg)\bigg] \displaystyle C_{41}(\tau^{2})=\left[\left(\frac{5}{2}+3U_{\omega}\right)\tau^{2}C_{20}+C_{38}c\right],C_{42}=\left[16U_{\omega}(U_{r}+U_{\omega})mc+4\frac{U_{\delta,L}}{L}U_{\omega}^{3}m\right],C_{43}(\tau)=\left[C_{35}(\tau)c^{2}+C_{36}(\tau)c\right], \displaystyle C_{44}(\tau^{2})=\left[16U_{\omega}^{2}\frac{U_{\delta,L}^{2}}{L^{2}}m^{2}c+\left(\frac{5}{2}+3U_{\omega}\right)\tau^{2}C_{21}m^{2}\right],C_{45}(\tau^{4})=\left[5\tau^{2}C_{8}(\tau^{2})c^{2}+\left(\frac{5}{2}+3U_{\omega}\right)\tau^{2}C_{22}(\tau^{2})\right]. Putting Eq. ( D.1 ) and Eq. ( D.1 ), back to Eq. ( 28 ), and by tower property, we get, \displaystyle{\mathbb{E}}_{t-\tau}\|x_{t+1}^{k}\|^{2}={\mathbb{E}}_{t-\tau}\|\widetilde{x}_{t+1}^{k}\|^{2}+{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k}\|^{2} \displaystyle+2{\mathbb{E}}_{t-\tau}\langle{\mathbf{B}}_{t+1}\Pi_{U_{\omega}}(\widetilde{\omega}_{t+1}^{k})-{\mathbf{B}}_{t+1}{\mathbf{R}}_{t+1}\widetilde{\omega}_{t+1}^{k},\widetilde{x}_{t+1}^{k}\rangle \displaystyle\leq \displaystyle{\mathbb{E}}_{t-\tau}\|\widetilde{x}_{t+1}^{k}\|^{2}+96\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+24\frac{\beta^{2}}{L}U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})+24\beta^{2}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}+100U_{\omega}^{2}{\mathbb{E}}_{t-\tau}\|{\mathbf{Q}}_{t}\|_{F}^{4} \displaystyle+6U_{\omega}{\mathbb{E}}_{t-\tau}\left\|{\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k}}\right\|_{2}\|m_{t+1}\|^{2}+16U_{\omega}^{2}{\mathbb{E}}_{t-\tau}\|{\mathbf{Q}}_{t}\|_{F}^{2}. (68) For \left\|{\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k}}\right\|_{2} , we have \displaystyle\left\|{\omega_{t+1}^{k}-\widetilde{\omega}_{t+1}^{k}}\right\|_{2}\leq\|\frac{\beta}{L}\delta_{t,L}^{k}{\mathbf{B}}_{t}^{\top}\phi(s_{t,0}^{k})\|\leq\beta U_{\delta},~~~~(\text{ since }\frac{U_{\delta,L}}{L}\leq U_{\delta}) for {\mathbb{E}}_{t-\tau}\|{\mathbf{Q}}_{t}\|_{F}^{4} , we use the crude bound that \|{\mathbf{Q}}_{t}\|_{F}\leq U_{\delta}U_{\omega}\zeta , and for {\mathbb{E}}_{t-\tau}\|{\mathbf{Q}}_{t}\|_{F}^{2} , we invoke Eq. ( C.2 ). Then, Eq. ( D.1 ) can be written as \displaystyle{\mathbb{E}}_{t-\tau}\|x_{t+1}^{k}\|^{2} \displaystyle\leq \displaystyle{\mathbb{E}}_{t-\tau}\|\widetilde{x}_{t+1}^{k}\|^{2}+96\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+24\frac{\beta^{2}}{L}U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})+24\beta^{2}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}+100U_{\omega}^{6}U_{\delta}^{4}\zeta^{4} \displaystyle+6U_{\omega}U_{\delta}\beta{\mathbb{E}}_{t-\tau}\|m_{t+1}\|^{2}+\frac{256U_{\omega}^{4}\zeta^{2}}{L^{2}}\frac{1}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{i}\|^{2}+64\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{4}\frac{\zeta^{2}}{K}+128U_{\omega}^{4}\frac{U_{\delta,L}^{2}}{L^{2}}\zeta^{2}m^{2}\rho^{2\tau} \displaystyle+144U_{\omega}^{2}\tau^{2}\frac{U_{\delta,L}^{4}}{L^{4}}\zeta^{2}\beta^{2}+\frac{64U_{\omega}^{4}\zeta^{2}}{K}\sum_{i=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{i}|^{2}. (69) Recall M_{t}={\mathbb{E}}_{t-\tau}\|m_{t}\|_{F}^{2};\bar{X}_{t}=\frac{1}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2};\bar{Y}_{t}=\frac{1}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}. Invoking Eq. ( D.1 ), and taking average over agents, Eq. ( D.1 ) can be written as, \displaystyle\bar{X}_{t+1} \displaystyle\leq \displaystyle\Bigg(1-2\lambda c\zeta+\Big(c_{3}^{2}\frac{c}{L}+c_{4}^{2}c+c_{5}^{2}\frac{1}{L}+c_{6}^{2}+4U_{\omega}^{4}\frac{1}{c_{5}^{2}L}\Big)\zeta \displaystyle+\bigg(C_{31}c+(36+40\tau^{2})\frac{c^{2}}{L^{2}}+C_{33}\frac{c}{L}+\left(36U_{\omega}^{4}+\left(2.5+3U_{\omega}\right)C_{18}\right)\frac{1}{L^{2}}+\frac{96c^{2}}{L^{2}}+\frac{256U_{\omega}^{4}}{L^{2}}\bigg)\zeta^{2}\Bigg)\bar{X}_{t} \displaystyle+\frac{36U_{\omega}^{2}}{c_{3}^{2}}\frac{c\zeta}{L}{\mathbb{E}}_{t-\tau}\|m_{t}\|^{2} \displaystyle+\Bigg(\Big[\frac{1}{c_{4}^{2}}c+\frac{U_{\omega}^{4}}{c_{6}^{2}}\Big]\zeta+\Big[C_{32}c+(9+10\tau^{2})c^{2}+C_{34}c+9U_{\omega}^{4}+\left(2.5+3U_{\omega}\right)C_{23}+24c^{2}+64U_{\omega}^{4}\big]\zeta^{2}\Bigg)\bar{Y}_{t} \displaystyle+\left[C_{39}(\tau^{2})+24c^{2}U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})\right]\frac{\zeta^{2}}{L}+\left[C_{41}(\tau^{2})+64\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{4}\right]\frac{\zeta^{2}}{K} \displaystyle+C_{42}\zeta\rho^{\tau}+C_{43}(\tau)\zeta^{3}+\left[C_{40}(\tau^{4})+100U_{\omega}^{6}U_{\delta}^{4}+144U_{\omega}^{2}\tau^{2}\frac{U_{\delta,L}^{4}}{L^{4}}c^{2}\right]\zeta^{4}+C_{45}(\tau^{4})\gamma^{2}\zeta^{2} \displaystyle+6U_{\omega}U_{\delta}\beta{\mathbb{E}}_{t-\tau}\|m_{t+1}\|^{2}+\left[128U_{\omega}^{4}\frac{U_{\delta,L}^{2}}{L^{2}}m^{2}+C_{44}(\tau^{2})\right]\zeta^{2}\rho^{2\tau}. (70) By Eq. ( E ), Eq. ( D.1 ) can be written as, \displaystyle\bar{X}_{t+1}\leq \displaystyle\Bigg(1-2\lambda c\zeta+\Big(c_{3}^{2}\frac{c}{L}+c_{4}^{2}c+c_{5}^{2}\frac{1}{L}+c_{6}^{2}+4U_{\omega}^{4}\frac{1}{c_{5}^{2}L}\Big)\zeta \displaystyle+\bigg(C_{31}c+(36+40\tau^{2})\frac{c^{2}}{L^{2}}+C_{33}\frac{c}{L}+\left(36U_{\omega}^{4}+\left(2.5+3U_{\omega}\right)C_{18}\right)\frac{1}{L^{2}} \displaystyle\qquad+\frac{96c^{2}}{L^{2}}+\frac{256U_{\omega}^{4}}{L^{2}}+96U_{\delta}c\frac{U_{\omega}^{3}}{c_{2}^{2}}\frac{1}{L^{2}}\bigg)\zeta^{2}+6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,4}(\tau^{2})\zeta^{3}\Bigg)\bar{X}_{t} \displaystyle+\frac{36U_{\omega}^{2}}{c_{3}^{2}}\frac{c\zeta}{L}M_{t}+6U_{\omega}U_{\delta}c\zeta M_{t} \displaystyle+\Bigg(\Big[\frac{1}{c_{4}^{2}}c+\frac{U_{\omega}^{4}}{c_{6}^{2}}\Big]\zeta+\Bigg[C_{32}c+(9+10\tau^{2})c^{2}+C_{34}c+9U_{\omega}^{4}+\left(2.5+3U_{\omega}\right)C_{23} \displaystyle\qquad+24c^{2}+64U_{\omega}^{4}+24U_{\delta}c\frac{U_{\omega}^{3}}{c_{1}^{2}}\Bigg]\zeta^{2}+6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,5}(\tau^{2})\zeta^{3}\Bigg)\bar{Y}_{t} \displaystyle+\left[C_{39}(\tau^{2})+24c^{2}U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})\right]\frac{\zeta^{2}}{L}+\left[C_{41}(\tau^{2})+64\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{4}\right]\frac{\zeta^{2}}{K} \displaystyle+C_{42}\zeta\rho^{\tau}+\left[C_{43}(\tau)+24U_{\omega}U_{\delta}cC_{25}(\tau^{2})\frac{1}{L}+6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,2}(\tau^{2})\frac{1}{K}\right]\zeta^{3} \displaystyle+\left[C_{40}(\tau^{4})+100U_{\omega}^{6}U_{\delta}^{4}+144U_{\omega}^{2}\tau^{2}\frac{U_{\delta,L}^{4}}{L^{4}}c^{2}+6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,6}\right]\zeta^{4}+C_{45}(\tau^{4})\gamma^{2}\zeta^{2} \displaystyle+6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,1}\zeta^{5}+24U_{\omega}U_{\delta}cC_{28}(\tau^{4})\gamma^{2}\zeta^{3}+6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,3}\zeta^{7}+\left[128U_{\omega}^{4}\frac{U_{\delta,L}^{2}}{L^{2}}m^{2}+C_{44}(\tau^{2})\right]\zeta^{2}\rho^{2\tau}. (71) Choosing \tau=\lceil 2\log_{\rho}(\zeta)\rceil , we have \displaystyle\bar{X}_{t+1}\leq\Bigg(1-2\lambda c\zeta+\Big(c_{3}^{2}\frac{c}{L}+c_{4}^{2}c+c_{5}^{2}\frac{1}{L}+c_{6}^{2}+4U_{\omega}^{4}\frac{1}{c_{5}^{2}L}\Big)\zeta+{\mathcal{C}}_{X,1}(\tau^{2})\zeta^{2}+{\mathcal{C}}_{X,2}(\tau^{2})\zeta^{3}\Bigg)\bar{X}_{t} \displaystyle\qquad+\left(\frac{36U_{\omega}^{2}}{c_{3}^{2}}\frac{c\zeta}{L}+6U_{\omega}U_{\delta}c\zeta\right)M_{t} \displaystyle\qquad+\Bigg(\Big[\frac{1}{c_{4}^{2}}c+\frac{U_{\omega}^{4}}{c_{6}^{2}}\Big]\zeta+{\mathcal{C}}_{X,3}(\tau^{2})\zeta^{2}+{\mathcal{C}}_{X,4}(\tau^{2})\zeta^{3}\Bigg)\bar{Y}_{t} \displaystyle\qquad+{\mathcal{C}}_{X,5}(\tau^{2})\frac{\zeta^{2}}{L}+{\mathcal{C}}_{X,6}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{C}}_{X,7}\zeta^{3}+{\mathcal{C}}_{X,8}\zeta^{4}+{\mathcal{C}}_{X,9}\gamma^{2}\zeta^{2}+{\mathcal{C}}_{X,10}\zeta^{5}+{\mathcal{C}}_{X,11}\gamma^{2}\zeta^{3}+{\mathcal{C}}_{X,12}\zeta^{7}+{\mathcal{C}}_{X,13}\zeta^{6}, (72) where \displaystyle{\mathcal{C}}_{X,1}(\tau^{2})=\bigg(C_{31}c+(36+40\tau^{2})\frac{c^{2}}{L^{2}}+C_{33}\frac{c}{L}+\left(36U_{\omega}^{4}+\left(2.5+3U_{\omega}\right)C_{18}\right)\frac{1}{L^{2}}+\frac{96c^{2}}{L^{2}}+\frac{256U_{\omega}^{4}}{L^{2}}+96U_{\delta}c\frac{U_{\omega}^{3}}{c_{2}^{2}}\frac{1}{L^{2}}\bigg), \displaystyle{\mathcal{C}}_{X,2}(\tau^{2})=6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,4}(\tau^{2}), \displaystyle{\mathcal{C}}_{X,3}(\tau^{2})=\Bigg[C_{32}c+(9+10\tau^{2})c^{2}+C_{34}c+9U_{\omega}^{4}+\left(2.5+3U_{\omega}\right)C_{23}+24c^{2}+64U_{\omega}^{4}+24U_{\delta}c\frac{U_{\omega}^{3}}{c_{1}^{2}}\Bigg], \displaystyle{\mathcal{C}}_{X,4}(\tau^{2})=6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,5}(\tau^{2}),{\mathcal{C}}_{X,5}(\tau^{2})=\left[C_{39}(\tau^{2})+24c^{2}U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})\right],{\mathcal{C}}_{X,6}(\tau^{2})=\left[C_{41}(\tau^{2})+64\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{4}\right], \displaystyle{\mathcal{C}}_{X,7}=\left[C_{42}+C_{43}(\tau)+24U_{\omega}U_{\delta}cC_{25}(\tau^{2})\frac{1}{L}+6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,2}(\tau^{2})\frac{1}{K}\right], \displaystyle{\mathcal{C}}_{X,8}=\left[C_{40}(\tau^{4})+100U_{\omega}^{6}U_{\delta}^{4}+144U_{\omega}^{2}\tau^{2}\frac{U_{\delta,L}^{4}}{L^{4}}c^{2}+6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,6}\right],{\mathcal{C}}_{X,9}=C_{45}(\tau^{4}),{\mathcal{C}}_{X,10}=6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,1}, \displaystyle{\mathcal{C}}_{X,11}=24U_{\omega}U_{\delta}cC_{28}(\tau^{4}),{\mathcal{C}}_{X,12}=6U_{\omega}U_{\delta}c{\mathcal{C}}_{M,3},{\mathcal{C}}_{X,13}=\left[128U_{\omega}^{4}\frac{U_{\delta,L}^{2}}{L^{2}}m^{2}+C_{44}(\tau^{2})\right]. ∎ D.2 Lower bound: Proof of Lemma 5.2 Recall that x_{t}^{k}={\mathbf{B}}_{t}\omega_{t}^{k}-z^{k,*}={\mathbf{B}}_{t}\omega_{t}^{k}-{\mathbf{B}}^{*}\omega^{k,*} . By definition, for each k\in[K] , {\mathbf{B}}_{t}\omega_{t}^{k} lies in the subspace spanned by {\mathbf{B}}_{t} . As illustrated in Fig. 3 , the distance between {\mathbf{B}}_{t}\omega_{t}^{k} and z^{k,*} is no smaller than the distance Figure 3 : Geometry illustration. between z^{k,*} and the subspace spanned by {\mathbf{B}}_{t} . Hence, \displaystyle\|x_{t}^{k}\| \displaystyle=\|{\mathbf{B}}_{t}\omega_{t}^{k}-z^{k,*}\|\geq\|{\mathbf{P}}_{t,\perp}z^{k,*}\| \displaystyle=\|({\mathbf{I}}-{\mathbf{P}}_{t})z^{k,*}\|=\|({\mathbf{P}}^{*}-{\mathbf{P}}_{t})z^{k,*}\| \displaystyle=\|({\mathbf{P}}_{t}-{\mathbf{P}}^{*})z^{k,*}\|. Recall that {\mathbf{P}}_{t}={\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top} and {\mathbf{P}}^{*}={\mathbf{B}}^{*}({\mathbf{B}}^{*})^{\top} . We first rewrite {\mathbf{P}}_{t}-{\mathbf{P}}^{*} to relate it to the cross terms {\mathbf{B}}_{t}^{\top}{\mathbf{B}}_{\perp}^{*} and {\mathbf{B}}_{t,\perp}^{\top}{\mathbf{B}}^{*} . Observe that \displaystyle\begin{bmatrix}{\mathbf{B}}_{t}^{\top}\\ {\mathbf{B}}_{t,\perp}^{\top}\end{bmatrix}({\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top}-{\mathbf{B}}^{*}{\mathbf{B}}^{*\top})\begin{bmatrix}{\mathbf{B}}_{\perp}^{*}&{\mathbf{B}}^{*}\end{bmatrix}=\begin{bmatrix}{\mathbf{B}}_{t}^{\top}{\mathbf{B}}_{\perp}^{*}&\textbf{0}\\ \textbf{0}&-{\mathbf{B}}_{t,\perp}^{\top}{\mathbf{B}}^{*}\end{bmatrix}. It is easy to check that \displaystyle{\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top}-{\mathbf{B}}^{*}{\mathbf{B}}^{*\top}=\begin{bmatrix}{\mathbf{B}}_{t}&{\mathbf{B}}_{t,\perp}\end{bmatrix}\begin{bmatrix}{\mathbf{B}}_{t}^{\top}{\mathbf{B}}_{\perp}^{*}&\textbf{0}\\ \textbf{0}&-{\mathbf{B}}_{t,\perp}^{\top}{\mathbf{B}}^{*}\end{bmatrix}\begin{bmatrix}{\mathbf{B}}_{\perp}^{*\top}\\ {\mathbf{B}}^{*\top}.\end{bmatrix} We can do singular value decomposition for the diagonal blocks; for ease of exposition, we drop the time index t in the decomposition. That is \displaystyle{\mathbf{B}}_{t}^{\top}{\mathbf{B}}_{\perp}^{*}={\mathbf{X}}_{1}\bf\Sigma_{1}{\mathbf{Y}}_{1}^{\top},~~~\text{and}~~~-{\mathbf{B}}_{t,\perp}^{\top}{\mathbf{B}}^{*}={\mathbf{X}}_{2}\bf\Sigma_{2}{\mathbf{Y}}_{2}^{\top}. Then, we have \displaystyle{\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top}-{\mathbf{B}}^{*}{\mathbf{B}}^{*\top}=\begin{bmatrix}{\mathbf{B}}_{t}&{\mathbf{B}}_{t,\perp}\end{bmatrix}\begin{bmatrix}{\mathbf{X}}_{1}&\textbf{0}\\ \textbf{0}&{\mathbf{X}}_{2}\end{bmatrix}\begin{bmatrix}\bf\Sigma_{1}&\textbf{0}\\ \textbf{0}&\bf\Sigma_{2}\end{bmatrix}\begin{bmatrix}{\mathbf{Y}}_{1}^{\top}&\textbf{0}\\ \textbf{0}&{\mathbf{Y}}_{2}^{\top}\end{bmatrix}\begin{bmatrix}{\mathbf{B}}_{\perp}^{*\top}\\ {\mathbf{B}}^{*\top}\end{bmatrix} \displaystyle= \displaystyle\begin{bmatrix}{\mathbf{B}}_{t}{\mathbf{X}}_{1}&{\mathbf{B}}_{t,\perp}{\mathbf{X}}_{2}\end{bmatrix}\begin{bmatrix}\bf\Sigma_{1}&\textbf{0}\\ \textbf{0}&\bf\Sigma_{2}\end{bmatrix}\begin{bmatrix}({\mathbf{B}}_{\perp}^{*}{\mathbf{Y}}_{1})^{\top}\\ ({\mathbf{B}}^{*}{\mathbf{Y}}_{2})^{\top}\end{bmatrix}. So, \displaystyle\|({\mathbf{P}}_{t}-{\mathbf{P}}^{*})z^{k,*}\|=\left\|{\begin{bmatrix}{\mathbf{B}}_{t}{\mathbf{X}}_{1}&{\mathbf{B}}_{t,\perp}{\mathbf{X}}_{2}\end{bmatrix}\begin{bmatrix}\bf\Sigma_{1}&\textbf{0}\\ \textbf{0}&\bf\Sigma_{2}\end{bmatrix}\begin{bmatrix}({\mathbf{B}}_{\perp}^{*}{\mathbf{Y}}_{1})^{\top}\\ ({\mathbf{B}}^{*}{\mathbf{Y}}_{2})^{\top}\end{bmatrix}z^{k,*}}\right\| \displaystyle= \displaystyle\left\|{\begin{bmatrix}\bf\Sigma_{1}&\textbf{0}\\ \textbf{0}&\bf\Sigma_{2}\end{bmatrix}\begin{bmatrix}({\mathbf{B}}_{\perp}^{*}{\mathbf{Y}}_{1})^{\top}\\ ({\mathbf{B}}^{*}{\mathbf{Y}}_{2})^{\top}\end{bmatrix}z^{k,*}}\right\| From the proof of ( 7 , Lemma 2.4) , we know that when d\geq 2r , \bf\Sigma_{1} and \bf\Sigma_{2} are identical up to permutation. Let \sigma_{1} denote the largest singular value. From ( 7 , Lemma 2.5) , we know that \displaystyle\sigma_{1}=\|{\mathbf{B}}_{t}^{\top}{\mathbf{B}}_{\perp}^{*}\|=\|{\mathbf{B}}_{t,\perp}^{\top}{\mathbf{B}}^{*}\|=\|{\mathbf{B}}_{t}{\mathbf{B}}_{t}^{\top}-{\mathbf{B}}^{*}{\mathbf{B}}^{*\top}\|=\|m_{t}\|. (73) Let {\mathbf{v}}_{1} and {\mathbf{v}}_{1}^{*} denote the corresponding right singular vectors of {\mathbf{B}}_{t}^{\top}{\mathbf{B}}_{\perp}^{*} and {\mathbf{B}}_{t,\perp}^{\top}{\mathbf{B}}^{*} . Let \widetilde{{\mathbf{v}}}_{1}=B^{*}{\mathbf{v}}_{1}\in{\mathbb{R}}^{d} , and \widetilde{{\mathbf{v}}}^{*}_{1}=B^{*}_{\perp}{\mathbf{v}}_{1}^{*}\in{\mathbb{R}}^{d} . It is easy to see that \widetilde{{\mathbf{v}}}_{1}\in\text{span}({\mathbf{B}}^{*}) and \widetilde{{\mathbf{v}}}_{1}^{*}\in\text{span}({\mathbf{B}}^{*}_{\perp}) . Hence, we get \displaystyle\|({\mathbf{P}}_{t}-{\mathbf{P}}^{*})z^{k,*}\|= \displaystyle\left\|{\begin{bmatrix}\bf\Sigma_{1}&\textbf{0}\\ \textbf{0}&\bf\Sigma_{2}\end{bmatrix}\begin{bmatrix}({\mathbf{B}}_{\perp}^{*}{\mathbf{Y}}_{1})^{\top}\\ ({\mathbf{B}}^{*}{\mathbf{Y}}_{2})^{\top}\end{bmatrix}z^{k,*}}\right\| \displaystyle\geq \displaystyle\sqrt{(\sigma_{1}\widetilde{{\mathbf{v}}}_{1}^{\top}z^{k,*})^{2}+(\sigma_{1}(\widetilde{{\mathbf{v}}}_{1}^{*})^{\top}z^{k,*})^{2}} \displaystyle= \displaystyle\sigma_{1}|\widetilde{{\mathbf{v}}}_{1}^{\top}z^{k,*}|, where the last line follows from z^{k,*}\in\text{span}({\mathbf{B}}^{*}) and thus (\widetilde{{\mathbf{v}}}_{1}^{*})^{\top}z^{k,*}=0,\forall k . Recall from Eq. ( 10 ) that {\mathbf{Z}}^{*}\in{\mathbb{R}}^{d\times K} stacks z^{k,*} as columnes. Averaging the reconstruction error across all agents, we get \displaystyle\frac{1}{K}\sum_{k=1}^{K}\|x_{t}^{k}\|^{2}\geq\frac{1}{K}\sum_{k=1}^{K}\sigma_{1}^{2}(\widetilde{{\mathbf{v}}}_{1}^{\top}z^{k,*})^{2}=\sigma_{1}^{2}\frac{1}{K}\sum_{k=1}^{K}(\widetilde{{\mathbf{v}}}_{1}^{\top}z^{k,*})^{2} \displaystyle= \displaystyle\sigma_{1}^{2}\frac{1}{K}\left\|{\widetilde{{\mathbf{v}}}_{1}^{\top}{\mathbf{Z}}^{*}}\right\|^{2} \displaystyle= \displaystyle\sigma_{1}^{2}\frac{1}{K}\widetilde{{\mathbf{v}}}_{1}^{\top}{\mathbf{Z}}^{*}{\mathbf{Z}}^{*\top}\widetilde{{\mathbf{v}}}_{1}. Recall from Eq. ( 4 ) that z^{k,*}={\mathbf{B}}^{*}\omega^{k,*} . Then, \displaystyle\frac{\sigma_{1}^{2}}{K}\widetilde{{\mathbf{v}}}_{1}^{\top}{\mathbf{Z}}^{*}{\mathbf{Z}}^{*\top}\widetilde{{\mathbf{v}}}_{1}\geq\frac{\sigma_{1}^{2}}{K}\lambda^{+}_{\min}({\mathbf{Z}}^{*}{\mathbf{Z}}^{*\top}). Therefore, we have \displaystyle\frac{1}{K}\sum_{k=1}^{K}\|x_{t}^{k}\|^{2}\geq\frac{\sigma_{1}^{2}}{K}\lambda^{+}_{\min}({\mathbf{Z}}^{*}{\mathbf{Z}}^{*\top})=\frac{\|m_{t}\|^{2}}{K}\lambda^{+}_{\min}({\mathbf{Z}}^{*}{\mathbf{Z}}^{*\top}), where the last equality follows from Eq. ( 73 ). Appendix E Proof of Lemma 5.3 (Bounds on Principal Angle Distance) Recall from Algorithm 1 of the update of \bar{\mathbf{B}} that \bar{\mathbf{B}}_{t+1}=\mathbf{B}_{t}+\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{t,\perp}{\mathbf{B}}_{t,\perp}^{\top}\delta_{t,L}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}. Since \mathbf{B}_{t+1}=\bar{\mathbf{B}}_{t+1}\mathbf{R}_{t+1}^{-1} , we have \displaystyle\|{\mathbf{B}}_{\perp}^{*\top}\mathbf{B}_{t+1}\|^{2}_{F}=\|{\mathbf{B}}_{\perp}^{*\top}\bar{\mathbf{B}}_{t+1}{\mathbf{R}}^{-1}_{t+1}\|^{2}_{F}\leq\|{\mathbf{B}}_{\perp}^{*\top}\bar{\mathbf{B}}_{t+1}\|^{2}_{F}\|{\mathbf{R}}^{-1}_{t+1}\|^{2}. That is, \displaystyle\|m_{t+1}\|_{F}^{2}\leq\|\bar{m}_{t+1}\|^{2}_{F}\|{\mathbf{R}}^{-1}_{t+1}\|^{2}. For \|\bar{m}_{t+1}\|^{2}_{F} , we have \displaystyle\|\bar{m}_{t+1}\|^{2}_{F}=\|{\mathbf{B}}_{\perp}^{*\top}\bar{\mathbf{B}}_{t+1}\|^{2}_{F}=\|{\mathbf{B}}_{\perp}^{*\top}\mathbf{B}_{t}+\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{B}}_{t,\perp}{\mathbf{B}}_{t,\perp}^{\top}\delta_{t,L}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}\|^{2}_{F} \displaystyle= \displaystyle\|{\mathbf{B}}_{\perp}^{*\top}\mathbf{B}_{t}\|_{F}^{2}+2\langle{\mathbf{B}}_{\perp}^{*\top}\mathbf{B}_{t},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{B}}_{t,\perp}{\mathbf{B}}_{t,\perp}^{\top}\delta_{t,L}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}\rangle+\|\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{B}}_{t,\perp}{\mathbf{B}}_{t,\perp}^{\top}\delta_{t,L}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}\|_{F}^{2} \displaystyle= \displaystyle\|m_{t}\|_{F}^{2}+\underbrace{\|\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{B}}_{t,\perp}{\mathbf{B}}_{t,\perp}^{\top}\delta_{t,L}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}\|_{F}^{2}}_{I.1}+2\langle{\mathbf{B}}_{\perp}^{*\top}\mathbf{B}_{t},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t,\perp}\xi_{t,L}^{k}(\omega_{t}^{k})^{\top}\rangle \displaystyle+\underbrace{2\langle{\mathbf{B}}_{\perp}^{*\top}\mathbf{B}_{t},\frac{\zeta}{K}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t,\perp}y_{t}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}\rangle}_{I.2}+\underbrace{2\langle{\mathbf{B}}_{\perp}^{*\top}\mathbf{B}_{t},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t,\perp}A_{L}^{k}x_{t}^{k}(\omega_{t}^{k})^{\top}\rangle}_{I.3}, (74) where the last equality follows from Proposition 1 . For I.1 in Eq. ( E ), recalling that \|{\mathbf{Q}}_{t}\|_{F}^{2}=\|\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{P}}_{t,\perp}\delta_{t,L}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}\|_{F}^{2} , we have \displaystyle I.1= \displaystyle\|\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t,\perp}\delta_{t,L}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}\|_{F}^{2} \displaystyle= \displaystyle\|{\mathbf{B}}_{\perp}^{*\top}\frac{\zeta}{K}\sum_{k=1}^{K}{\mathbf{P}}_{t,\perp}\delta_{t,L}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}\|_{F}^{2} \displaystyle\leq \displaystyle\|{\mathbf{B}}_{\perp}^{*\top}\|^{2}\|{\mathbf{Q}}_{t}\|_{F}^{2} \displaystyle\leq \displaystyle\|{\mathbf{Q}}_{t}\|_{F}^{2}. Similar to the derivation in the proof of Lemma 5.1 , I.2 in Eq. ( E ) can be bounded as \displaystyle I.2=2\langle m_{t},\frac{\zeta}{K}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t,\perp}y_{t}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}\rangle\leq 2\|m_{t}\|\frac{\zeta}{K}\sum_{k=1}^{K}\|{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t,\perp}y_{t}^{k}\phi(s_{t,0}^{k})(\omega_{t}^{k})^{\top}\|\leq 2U_{\omega}\frac{\zeta}{K}\sum_{k=1}^{K}\|m_{t}\||y_{t}^{k}|. I.3 in Eq. ( E ) is related to the negative drift of the m_{t} . Specifically, \displaystyle\frac{1}{2}I.3= \displaystyle\langle{\mathbf{B}}_{\perp}^{*\top}\mathbf{B}_{t},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t,\perp}A_{L}^{k}x_{t}^{k}(\omega_{t}^{k})^{\top}\rangle \displaystyle=\frac{\zeta}{KL}\sum_{k=1}^{K}\left({\mathbf{P}}_{t,\perp}{\mathbf{P}}_{\perp}^{*}{\mathbf{B}}_{t}\omega_{t}^{k}\right)^{\top}A_{L}^{k}\left({\mathbf{P}}_{t}+{\mathbf{P}}_{t,\perp}\right)x_{t}^{k}, which is true because \left\langle AB,vw^{\top}\right\rangle=\text{trace}\left(A^{\top}vw^{\top}\right)=w^{\top}B^{\top}A^{\top}v for any r\times r matrix A and r -dimensional vectors v and w . Additionally, \displaystyle\left({\mathbf{P}}_{t,\perp}{\mathbf{P}}_{\perp}^{*}{\mathbf{B}}_{t}\omega_{t}^{k}\right)^{\top}A_{L}^{k}\left({\mathbf{P}}_{t}+{\mathbf{P}}_{t,\perp}\right)x_{t}^{k} \displaystyle\leq\left({\mathbf{P}}_{t,\perp}{\mathbf{P}}_{\perp}^{*}x_{t}^{k}\right)^{\top}A_{L}^{k}{\mathbf{P}}_{t,\perp}x_{t}^{k}+2U_{\omega}\|m_{t}\|\|x_{t}^{k}\| \displaystyle\leq\left({\mathbf{P}}_{t,\perp}x_{t}^{k}\right)^{\top}A_{L}^{k}{\mathbf{P}}_{t,\perp}x_{t}^{k}+6U_{\omega}\|m_{t}\|\|x_{t}^{k}\| \displaystyle\leq-\lambda L\|{\mathbf{P}}_{t,\perp}x_{t}^{k}\|^{2}+6U_{\omega}\|m_{t}\|\|x_{t}^{k}\|. For each k\in[K] , it holds that \displaystyle\|{\mathbf{P}}_{t,\perp}x_{t}^{k}\|^{2}=\|{\mathbf{P}}_{t,\perp}(\mathbf{B}_{t}\omega_{t}^{k}-{\mathbf{B}}^{*}\omega^{k,*})\|^{2}=\|{\mathbf{P}}_{t,\perp}z^{k,*}\|=\|({\mathbf{P}}_{t}-{\mathbf{P}}^{*})z^{k,*}\|. From the analysis in the proof of Lemma 5.2 , we know that \frac{1}{K}\sum_{k=1}^{K}\|({\mathbf{P}}_{t}-{\mathbf{P}}^{*})z^{k,*}\|^{2}\geq\|m_{t}\|_{F}^{2}\frac{\lambda_{\min}({\mathbf{Z}}^{*}{\mathbf{Z}}^{*\top})}{rK}. So, \displaystyle\frac{1}{K}\sum_{k=1}^{K}\|{\mathbf{P}}_{\perp}^{*}(\mathbf{B}_{t}\omega_{t}^{k}-{\mathbf{B}}^{*}\omega^{k,*})\|^{2} \displaystyle\geq\|m_{t}\|_{F}^{2}\frac{\lambda_{\min}({\mathbf{Z}}^{*}{\mathbf{Z}}^{*\top})}{rK}. Thus, \displaystyle I.3\leq-2\lambda\zeta\frac{\lambda_{\min}({\mathbf{Z}}^{*}{\mathbf{Z}}^{*})^{\top}}{rK}\|m_{t}\|_{F}^{2}+12U_{\omega}\zeta\frac{1}{KL}\sum_{k=1}^{K}\|m_{t}\|\|x_{t}^{k}\|. (75) Combining the bounds of I.1, I.2, and I.3 of Eq. ( E ), and applying Assumption 4 , we have \displaystyle\|\bar{m}_{t+1}\|_{F}^{2}\leq(1-\frac{2\lambda\alpha\zeta}{r})\|m_{t}\|_{F}^{2}+\|{\mathbf{Q}}_{t}\|_{F}^{2}+2\langle m_{t},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t,\perp}\xi_{t,L}^{k}(\omega_{t}^{k})^{\top}\rangle \displaystyle~~~~~~~~+2U_{\omega}\frac{\zeta}{K}\sum_{k=1}^{K}\|m_{t}\||y_{t}^{k}|+12U_{\omega}\zeta\frac{1}{KL}\sum_{k=1}^{K}\|m_{t}\|\|x_{t}^{k}\|. (76) Recall that \|m_{t+1}\|_{F}^{2}\leq\|\bar{m}_{t+1}\|^{2}_{F}\|{\mathbf{R}}^{-1}_{t+1}\|^{2} , we have \displaystyle\|m_{t+1}\|_{F}^{2}\leq(1-\frac{2\lambda\alpha\zeta}{r})\|m_{t}\|_{F}^{2}\|{\mathbf{R}}^{-1}_{t+1}\|^{2} \displaystyle~~~~~~~~+\|{\mathbf{Q}}_{t}\|_{F}^{2}\|{\mathbf{R}}^{-1}_{t+1}\|^{2}+2\langle m_{t},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t,\perp}\xi_{t,L}^{k}(\omega_{t}^{k})^{\top}\rangle\|{\mathbf{R}}^{-1}_{t+1}\|^{2} \displaystyle~~~~~~~~+2U_{\omega}\frac{\zeta}{K}\sum_{k=1}^{K}\|m_{t}\||y_{t}^{k}|\|{\mathbf{R}}^{-1}_{t+1}\|^{2}+12U_{\omega}\zeta\frac{1}{KL}\sum_{k=1}^{K}\|m_{t}\|\|x_{t}^{k}\|\|{\mathbf{R}}^{-1}_{t+1}\|^{2}. For the first term, we use Eq. ( 21 ) that {\|{\mathbf{R}}_{t+1}^{-1}\|^{2}\leq(\frac{1}{1-U_{\delta}^{2}U_{\omega}^{2}\zeta^{2}})^{2}} , and for the remaining terms, we use a crude bound that \|{\mathbf{R}}_{t+1}^{-1}\|^{2}\leq(\frac{1}{1-U_{\delta}^{2}U_{\omega}^{2}\zeta^{2}})^{2}\leq 2 when U_{\delta}^{2}U_{\omega}^{2}\zeta^{2}\leq\frac{1}{6} , then, \displaystyle\|m_{t+1}\|_{F}^{2}\leq(1-\frac{\lambda\alpha\zeta}{r})(\frac{1}{1-U_{\delta}^{2}U_{\omega}^{2}\zeta^{2}})^{2}\|m_{t}\|_{F}^{2}+2\|{\mathbf{Q}}_{t}\|_{F}^{2}+4\langle m_{t},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t,\perp}\xi_{t,L}^{k}(\omega_{t}^{k})^{\top}\rangle \displaystyle~~~~~~~~+4U_{\omega}\frac{\zeta}{K}\sum_{k=1}^{K}\|m_{t}\||y_{t}^{k}|+24U_{\omega}\zeta\frac{1}{KL}\sum_{k=1}^{K}\|m_{t}\|\|x_{t}^{k}\|. Moreover, since U_{\delta}^{2}U_{\omega}^{2}\zeta^{2}\leq\frac{1}{6} , we have (\frac{1}{1-U_{\delta}^{2}U_{\omega}^{2}\zeta^{2}})^{2}\leq 1+3U_{\delta}^{2}U_{\omega}^{2}\zeta^{2} . Providing \frac{\lambda\alpha}{r}\geq 3U_{\delta}^{2}U_{\omega}^{2}\zeta , it holds that \displaystyle(1-\frac{2\lambda\alpha\zeta}{r})(\frac{1}{1-U_{\delta}^{2}U_{\omega}^{2}\zeta^{2}})^{2}\leq(1-\frac{2\lambda\alpha\zeta}{r})(1+3U_{\delta}^{2}U_{\omega}^{2}\zeta^{2})\leq(1-\frac{\lambda\alpha\zeta}{r}). Then, \displaystyle\|m_{t+1}\|_{F}^{2}\leq(1-\frac{\lambda\alpha\zeta}{r})\|m_{t}\|_{F}^{2}+2\|{\mathbf{Q}}_{t}\|_{F}^{2}+4\langle m_{t},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t,\perp}\xi_{t,L}^{k}(\omega_{t}^{k})^{\top}\rangle \displaystyle~~~~~~~~+4U_{\omega}\frac{\zeta}{K}\sum_{k=1}^{K}\|m_{t}\||y_{t}^{k}|+24U_{\omega}\zeta\frac{1}{KL}\sum_{k=1}^{K}\|m_{t}\|\|x_{t}^{k}\|. (77) By Young’s inequality, we have for any c_{1}>0,c_{2}>0 , \displaystyle 4U_{\omega}\frac{\zeta}{K}\sum_{k=1}^{K}\|m_{t}\||y_{t}^{k}|\leq c_{1}^{2}\zeta\|m_{t}\|^{2}+\frac{4U_{\omega}^{2}}{c_{1}^{2}}\zeta\frac{1}{K}\sum_{k=1}^{K}|y_{t}^{k}|^{2}, \displaystyle 16U_{\omega}\zeta\frac{1}{KL}\sum_{k=1}^{K}\|m_{t}\|\|x_{t}^{k}\|\leq c_{2}^{2}\zeta\|m_{t}\|^{2}+\frac{144U_{\omega}^{2}}{c_{2}^{2}}\frac{\zeta}{L^{2}}\frac{1}{K}\sum_{k=1}^{K}\|x_{t}^{k}\|^{2}. Putting this back to Eq. ( E ), we get, \displaystyle\|m_{t+1}\|_{F}^{2}\leq(1-\frac{\lambda\alpha\zeta}{r}+c_{1}^{2}\zeta+c_{2}^{2}\zeta)\|m_{t}\|_{F}^{2}+2\|{\mathbf{Q}}_{t}\|_{F}^{2}+4\langle m_{t},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t,\perp}\xi_{t,L}^{k}(\omega_{t}^{k})^{\top}\rangle \displaystyle~~~~~~~~+\frac{4U_{\omega}^{2}}{c_{1}^{2}}\zeta\frac{1}{K}\sum_{k=1}^{K}|y_{t}^{k}|^{2}+\frac{144U_{\omega}^{2}}{c_{2}^{2}}\frac{\zeta}{L^{2}}\frac{1}{K}\sum_{k=1}^{K}\|x_{t}^{k}\|^{2}. (78) For the third term of Eq. ( E ), we have \displaystyle\langle m_{t},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}\xi_{t,L}^{k}(\omega_{t}^{k})^{\top}\rangle \displaystyle= \displaystyle\langle m_{t},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}+(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t}\omega_{t}^{k}\big](\omega_{t}^{k})^{\top}\rangle~~~~~(\text{by Eq.\,(\ref{eqn: xi decomp})}) \displaystyle= \displaystyle\langle m_{t},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big](\omega_{t}^{k})^{\top}\rangle+\langle m_{t},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t}\omega_{t}^{k}(\omega_{t}^{k})^{\top}\rangle \displaystyle= \displaystyle\langle m_{t},{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}\frac{\zeta}{KL}\sum_{k=1}^{K}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big](\omega_{t}^{k}-\omega_{t-\tau}^{k})^{\top}\rangle+\langle m_{t}-m_{t-\tau},{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}\frac{\zeta}{KL}\sum_{k=1}^{K}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big](\omega_{t-\tau}^{k})^{\top}\rangle \displaystyle+\langle m_{t-\tau},{\mathbf{B}}_{\perp}^{*\top}({\mathbf{P}}_{t,\perp}-{\mathbf{P}}_{t-\tau,\perp})\frac{\zeta}{KL}\sum_{k=1}^{K}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big](\omega_{t-\tau}^{k})^{\top}\rangle+\underbrace{\langle m_{t-\tau},{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t-\tau,\perp}\frac{\zeta}{KL}\sum_{k=1}^{K}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big](\omega_{t-\tau}^{k})^{\top}\rangle}_{I_{1}} \displaystyle+\langle m_{t},{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}\frac{\zeta}{KL}\sum_{k=1}^{K}(\widetilde{A}_{t,L}^{k}-A_{L}^{k})\left({\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\right)\omega_{t}^{k}(\omega_{t}^{k})^{\top}\rangle \displaystyle+\langle m_{t},{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}\frac{\zeta}{KL}\sum_{k=1}^{K}(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t-\tau}(\omega_{t}^{k}-\omega_{t-\tau}^{k})(\omega_{t}^{k})^{\top}\rangle \displaystyle+\langle m_{t},{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}\frac{\zeta}{KL}\sum_{k=1}^{K}(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}(\omega_{t}^{k}-\omega_{t-\tau}^{k})^{\top}\rangle \displaystyle+\langle m_{t}-m_{t-\tau},{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}\frac{\zeta}{KL}\sum_{k=1}^{K}(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}(\omega_{t-\tau}^{k})^{\top}\rangle \displaystyle+\langle m_{t-\tau},{\mathbf{B}}_{\perp}^{*\top}\left({\mathbf{P}}_{t,\perp}-{\mathbf{P}}_{t-\tau,\perp}\right)\frac{\zeta}{KL}\sum_{k=1}^{K}(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}(\omega_{t-\tau}^{k})^{\top}\rangle \displaystyle+\underbrace{\langle m_{t-\tau},{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t-\tau,\perp}}\frac{\zeta}{KL}\sum_{k=1}^{K}(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}(\omega_{t-\tau}^{k})^{\top}\rangle}_{I_{2}}. (79) Note that I_{1} and I_{2} in the above decomposition can be bounded under expectation conditioned on v_{0:t-\tau} , and other terms are in terms of \|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2} , \|\omega_{t}^{k}-\omega_{t-\tau}^{k}\|^{2} , \left\|{\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}}\right\|^{2} , and \left\|{\widetilde{A}_{t,L}^{k}-A_{L}^{k}}\right\|^{2} by applying Cauchy-Schwartz. Specifically, for the first term, we have \displaystyle\langle m_{t},{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}\frac{\zeta}{KL}\sum_{k=1}^{K}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big](\omega_{t}^{k}-\omega_{t-\tau}^{k})^{\top}\rangle \displaystyle\leq \displaystyle\frac{\zeta}{KL}\sum_{k=1}^{K}\left\|{\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}}\right\|\left\|{\omega_{t}^{k}-\omega_{t-\tau}^{k}}\right\| \displaystyle\leq \displaystyle\frac{\zeta}{KL}\sum_{k=1}^{K}\left(\frac{\zeta}{2L}\left\|{\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}}\right\|^{2}+\frac{L}{2\zeta}\left\|{\omega_{t}^{k}-\omega_{t-\tau}^{k}}\right\|^{2}\right) \displaystyle\leq \displaystyle\frac{1}{2}\frac{\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}\left\|{\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}}\right\|^{2}+\frac{1}{2K}\sum_{k=1}^{K}\left\|{\omega_{t}^{k}-\omega_{t-\tau}^{k}}\right\|^{2}. Applying similar analysis to all other terms in Eq. ( E ), we obtain \displaystyle\langle m_{t}-m_{t-\tau},{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}\frac{\zeta}{KL}\sum_{k=1}^{K}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big](\omega_{t-\tau}^{k})^{\top}\rangle\leq\frac{1}{2}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2}+\frac{U_{\omega}^{2}}{2}\frac{\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}\left\|{\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}}\right\|^{2}, \displaystyle\langle m_{t-\tau},{\mathbf{B}}_{\perp}^{*\top}({\mathbf{P}}_{t,\perp}-{\mathbf{P}}_{t-\tau,\perp})\frac{\zeta}{KL}\sum_{k=1}^{K}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big](\omega_{t-\tau}^{k})^{\top}\rangle\leq 2\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2}+\frac{1}{2}U_{\omega}^{2}\frac{\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}\left\|{\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}}\right\|^{2}, \displaystyle\langle m_{t},{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}\frac{\zeta}{KL}\sum_{k=1}^{K}(\widetilde{A}_{t,L}^{k}-A_{L}^{k})\left({\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\right)\omega_{t}^{k}(\omega_{t}^{k})^{\top}\rangle\leq\frac{1}{2}U_{\omega}^{4}\frac{\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2}+\frac{1}{2}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2}, \displaystyle\langle m_{t},{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}\frac{\zeta}{KL}\sum_{k=1}^{K}(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t-\tau}(\omega_{t}^{k}-\omega_{t-\tau}^{k})(\omega_{t}^{k})^{\top}\rangle \displaystyle\qquad\qquad\qquad\qquad\leq\frac{U_{\omega}}{2}\frac{\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2}+\frac{U_{\omega}}{2}\!\frac{1}{K}\sum_{k=1}^{K}\|\omega_{t}^{k}-\omega_{t-\tau}^{k}\|^{2}, \displaystyle\langle m_{t},{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}\frac{\zeta}{KL}\sum_{k=1}^{K}(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}(\omega_{t}^{k}-\omega_{t-\tau}^{k})^{\top}\rangle \displaystyle\qquad\qquad\qquad\qquad\leq\frac{U_{\omega}}{2}\frac{\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2}+\frac{U_{\omega}}{2}\frac{1}{K}\sum_{k=1}^{K}\|\omega_{t}^{k}-\omega_{t-\tau}^{k}\|^{2}, \displaystyle\langle m_{t}-m_{t-\tau},{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}\frac{\zeta}{KL}\sum_{k=1}^{K}(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}(\omega_{t-\tau}^{k})^{\top}\rangle \displaystyle\qquad\qquad\qquad\qquad\leq\frac{1}{2}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2}+\frac{1}{2}U_{\omega}^{4}\frac{\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2}, \displaystyle\langle m_{t-\tau},{\mathbf{B}}_{\perp}^{*\top}\left({\mathbf{P}}_{t,\perp}-{\mathbf{P}}_{t,\perp}\right)\frac{\zeta}{KL}\sum_{k=1}^{K}(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}(\omega_{t-\tau}^{k})^{\top}\rangle \displaystyle\qquad\qquad\qquad\qquad\leq 2\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2}+\frac{U_{\omega}^{4}}{2}\frac{\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2} Combining the above bounds and pulgging them back to Eq. ( E ), and taking expectation conditioned on v_{0:t-\tau} , we get \displaystyle{\mathbb{E}}_{t-\tau}\langle m_{t},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}\xi_{t,L}^{k}(\omega_{t}^{k})^{\top}\rangle \displaystyle\leq \displaystyle\left(\frac{1}{2}+U_{\omega}^{2}\right)\frac{\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}{\mathbb{E}}_{t-\tau}\left\|{\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}}\right\|^{2}+\left(\frac{1}{2}+U_{\omega}\right)\frac{1}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-\tau}\left\|{\omega_{t}^{k}-\omega_{t-\tau}^{k}}\right\|^{2} \displaystyle+\left(U_{\omega}+\frac{3}{2}U_{\omega}^{4}\right)\frac{\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}{\mathbb{E}}_{t-\tau}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2}+8{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2} \displaystyle+\underbrace{\langle m_{t-\tau},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t-\tau,\perp}{\mathbb{E}}_{t-\tau}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big](\omega_{t-\tau}^{k})^{\top}\rangle}_{I_{1}} \displaystyle+\underbrace{\langle m_{t-\tau},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t-\tau,\perp}{\mathbb{E}}_{t-\tau}[\widetilde{A}_{t,L}^{k}-A_{L}^{k}]{\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}(\omega_{t-\tau}^{k})^{\top}\rangle}_{I_{2}}. (80) Note that Lemma I.5 holds for any t\geq 0 and any state s_{t,k}^{0} . Applying Lemma I.5 , we have \displaystyle{\mathbb{E}}_{t-\tau}\left\|{\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}}\right\|^{2}\leq\left(16U_{r}^{2}+32U_{r}^{2}\frac{m\rho}{1-\rho}\right)L. (81) Similarly, we have, \displaystyle{\mathbb{E}}_{t-\tau}\|\widetilde{A}_{t,L}^{k}-A_{L}^{k}\|^{2}\leq\left(16+32\frac{m\rho}{1-\rho}\right)L.~~~~(\text{ Lemma \ref{lmm: local noise reduction}, Eq.\,(\ref{eqn: Atilde-Abar ^2 local noise})}) (82) Applying Lemma I.2 and Lemma I.3 , we have the following, \displaystyle{\mathbb{E}}_{t-\tau}\|\omega_{t}^{k}-\omega_{t-\tau}^{k}\|^{2} \displaystyle\leq \displaystyle\tau\sum_{j=t-\tau}^{t-1}{\mathbb{E}}_{t-\tau}\|\omega_{j+1}^{k}-\omega_{j}^{k}\|^{2} \displaystyle\leq \displaystyle\tau^{2}C_{5}(\tau^{2})\frac{\beta^{4}}{L^{2}}+\tau^{2}C_{6}(\tau^{2})\frac{\zeta^{2}\beta^{2}}{L^{2}}+8\tau^{2}\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+\tau^{2}C_{7}\frac{\beta^{2}}{L}+\tau^{2}C_{8}(\tau^{2})\gamma^{2}\beta^{2}+2\tau^{2}\beta^{2}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}, (83) and \displaystyle{\mathbb{E}}_{t-\tau}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|^{2} \displaystyle\leq \displaystyle\tau^{2}C_{16}(\tau^{2})\frac{\zeta^{2}\beta^{2}}{L^{2}}+\tau^{2}C_{17}(\tau^{2})\frac{\zeta^{4}}{L^{2}}+\tau^{2}C_{18}\frac{\zeta^{2}}{L^{2}}\frac{1}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2}+\tau^{2}C_{19}(\tau^{2})\zeta^{2}\beta^{2}+\tau^{2}C_{20}\frac{\zeta^{2}}{K} \displaystyle+\tau^{2}C_{21}\zeta^{2}m^{2}\rho^{2\tau}+\tau^{2}C_{22}(\tau^{2})\gamma^{2}\zeta^{2}+\tau^{2}C_{23}\frac{\zeta^{2}}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}+\tau^{2}C_{24}\zeta^{4}. (84) For I_{1} , by Lemma I.6 , we have \displaystyle{\mathbb{E}}_{t-\tau}[I_{1}]={\mathbb{E}}_{t-\tau}\langle m_{t-\tau},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t-\tau,\perp}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big](\omega_{t-\tau}^{k})^{\top}\rangle \displaystyle= \displaystyle\langle m_{t-\tau},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t-\tau,\perp}{\mathbb{E}}_{t-\tau}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big](\omega_{t-\tau}^{k})^{\top}\rangle \displaystyle\leq \displaystyle 4U_{r}U_{\omega}\zeta m\rho^{\tau}. (85) Similarly, for I_{2} , by Lemma I.6 , we have \displaystyle{\mathbb{E}}_{t-\tau}[I_{2}]={\mathbb{E}}_{t-\tau}\langle m_{t-\tau},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t-\tau,\perp}(\widetilde{A}_{t,L}^{k}-A_{L}^{k}){\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}(\omega_{t-\tau}^{k})^{\top}\rangle \displaystyle\leq \displaystyle\langle m_{t-\tau},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{\mathbf{P}}_{t-\tau,\perp}{\mathbb{E}}_{t-\tau}\left[\widetilde{A}_{t,L}^{k}-A_{L}^{k}\right]{\mathbf{B}}_{t-\tau}\omega_{t-\tau}^{k}(\omega_{t-\tau}^{k})^{\top}\rangle \displaystyle\leq \displaystyle 4U_{\omega}^{2}\frac{\zeta}{L}m\rho^{\tau}. (86) Putting Eq. ( 81 ), Eq. ( 82 ), Eq. ( E ), Eq. ( E ), Eq. ( E ) and Eq. ( E ) back to Eq. ( E ), taking conditional expectation, and using \beta=c\zeta , we get \displaystyle{\mathbb{E}}_{t-\tau}\langle m_{t},\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{\perp}^{*\top}{{\mathbf{P}}_{t,\perp}}\xi_{t,L}^{k}(\omega_{t}^{k})^{\top}\rangle \displaystyle\leq \displaystyle\left(\frac{1}{2}+U_{\omega}^{2}\right)\left(16U_{r}^{2}+32U_{r}^{2}\frac{m\rho}{1-\rho}\right)\frac{\zeta^{2}}{L} \displaystyle+\left(\frac{1}{2}+U_{\omega}\right)\bigg(\tau^{2}C_{5}(\tau^{2})c^{4}\frac{\zeta^{4}}{L^{2}}+\tau^{2}C_{6}(\tau^{2})c^{2}\frac{\zeta^{4}}{L^{2}}+8\tau^{2}c^{2}\frac{\zeta^{2}}{L^{2}}\frac{1}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|^{2} \displaystyle\qquad\qquad+\tau^{2}C_{7}c^{2}\frac{\zeta^{2}}{L}+\tau^{2}C_{8}(\tau^{2})c^{2}\gamma^{2}\zeta^{2}+2\tau^{2}c^{2}\zeta^{2}\frac{1}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|^{2}\bigg) \displaystyle+\left(U_{\omega}+\frac{3}{2}U_{\omega}^{4}\right)\left(16+32\frac{m\rho}{1-\rho}\right)\frac{\zeta^{2}}{L} \displaystyle+8\bigg(\tau^{2}C_{16}(\tau^{2})c^{2}\frac{\zeta^{4}}{L^{2}}+\tau^{2}C_{17}(\tau^{2})\frac{\zeta^{4}}{L^{2}}+\tau^{2}C_{18}\frac{\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|^{2}+\tau^{2}C_{19}(\tau^{2})c^{2}\zeta^{4}+\tau^{2}C_{20}\frac{\zeta^{2}}{K} \displaystyle\qquad\qquad+\tau^{2}C_{21}\zeta^{2}m^{2}\rho^{2\tau}+\tau^{2}C_{22}(\tau^{2})\gamma^{2}\zeta^{2}+\tau^{2}C_{23}\frac{\zeta^{2}}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|^{2}+\tau^{2}C_{24}\zeta^{4}\bigg) \displaystyle+2U_{r}U_{\omega}\zeta m\rho^{\tau}+2U_{\omega}^{2}\frac{\zeta}{L}m\rho^{\tau} \displaystyle\leq \displaystyle C_{25}(\tau^{2})\frac{\zeta^{2}}{L}+C_{26}(\tau^{4})\frac{\zeta^{4}}{L^{2}}+C_{27}(\tau^{4})\zeta^{4}+C_{28}(\tau^{4})\gamma^{2}\zeta^{2} \displaystyle+C_{29}(\tau^{2})\frac{\zeta^{2}}{KL}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|^{2}+C_{30}(\tau^{2})\frac{\zeta^{2}}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|^{2} \displaystyle+8\tau^{2}C_{20}\frac{\zeta^{2}}{K}+8\tau^{2}C_{21}\zeta^{2}m^{2}\rho^{2\tau}+2U_{r}U_{\omega}\zeta m\rho^{\tau}+2U_{\omega}^{2}\frac{\zeta}{L}m\rho^{\tau}. (87) where \displaystyle C_{25}(\tau^{2})=\Bigg[\left(\frac{1}{2}+U_{\omega}^{2}\right)\left(16U_{r}^{2}+32U_{r}^{2}\frac{m\rho}{1-\rho}\right)+\left(U_{\omega}+\frac{3}{2}U_{\omega}^{4}\right)\left(16+32\frac{m\rho}{1-\rho}\right)+\left(\frac{1}{2}+U_{\omega}\right)\tau^{2}C_{7}c^{2}\Bigg], \displaystyle C_{26}(\tau^{4})=\Bigg[\left(\frac{1}{2}+U_{\omega}\right)\left(\tau^{2}C_{5}(\tau^{2})c^{4}+\tau^{2}C_{6}(\tau^{2})c^{2}\right)+8\left(\tau^{2}C_{16}(\tau^{2})c^{2}+\tau^{2}C_{17}(\tau^{2})\right)\Bigg], \displaystyle C_{27}(\tau^{4})=\Bigg[8\tau^{2}C_{19}(\tau^{2})c^{2}+8\tau^{2}C_{24}\Bigg],C_{28}(\tau^{4})=\Bigg[\left(\frac{1}{2}+U_{\omega}\right)\tau^{2}C_{8}(\tau^{2})c^{2}+8\tau^{2}C_{22}(\tau^{2})\Bigg], \displaystyle C_{29}(\tau^{2})=\Bigg[8\tau^{2}c^{2}\left(\frac{1}{2}+U_{\omega}\right)+8\tau^{2}C_{18}\Bigg],C_{30}(\tau^{2})=\Bigg[2\tau^{2}c^{2}\left(\frac{1}{2}+U_{\omega}\right)+8\tau^{2}C_{23}\Bigg]. Putting the above Eq. ( E ) and Eq. ( C.2 ) back to Eq. ( E ) and taking expectation, we get \displaystyle{\mathbb{E}}_{t-\tau}\|m_{t+1}\|_{F}^{2}\leq(1-\frac{\lambda\alpha\zeta}{r}+c_{1}^{2}\zeta+c_{2}^{2}\zeta){\mathbb{E}}_{t-\tau}\|m_{t}\|_{F}^{2} \displaystyle+\left(\frac{32U_{\omega}^{2}\zeta^{2}}{L^{2}}+4C_{29}(\tau^{2})\frac{\zeta^{2}}{L}+\frac{144U_{\omega}^{2}}{c_{2}^{2}}\frac{\zeta}{L^{2}}\right)\frac{1}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2} \displaystyle+\left(4C_{30}(\tau^{2})\zeta^{2}+8U_{\omega}^{2}\zeta^{2}+\frac{4U_{\omega}^{2}}{c_{1}^{2}}\zeta\right)\frac{1}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2} \displaystyle+4C_{25}(\tau^{2})\frac{\zeta^{2}}{L}+4C_{26}(\tau^{4})\frac{\zeta^{4}}{L^{2}}+\left(4C_{27}(\tau^{4})+8\tau^{2}\frac{U_{\delta,L}^{4}}{L^{4}}c^{2}\right)\zeta^{4}+4C_{28}(\tau^{4})\gamma^{2}\zeta^{2} \displaystyle\qquad\qquad+\left(32\tau^{2}C_{20}+8\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{2}\right)\frac{\zeta^{2}}{K}+\left(32\tau^{2}C_{21}+16U_{\omega}^{2}\frac{U_{\delta,L}^{2}}{L^{2}}\right)\zeta^{2}m^{2}\rho^{2\tau}+8U_{r}U_{\omega}\zeta m\rho^{\tau}+8U_{\omega}^{2}\frac{\zeta}{L}m\rho^{\tau}. (88) Recall M_{t}={\mathbb{E}}_{t-\tau}\|m_{t}\|_{F}^{2};\bar{X}_{t}=\frac{1}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-\tau}\|x_{t}^{k}\|^{2};\bar{Y}_{t}=\frac{1}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-\tau}|y_{t}^{k}|^{2}, and choose \tau=\lceil 2\log_{\rho}(\zeta)\rceil , we get, \displaystyle M_{t+1}\leq(1-\frac{\lambda\alpha\zeta}{r}+c_{1}^{2}\zeta+c_{2}^{2}\zeta)M_{t} \displaystyle\qquad+\left({\mathcal{C}}_{M,4}(\tau^{2})\zeta^{2}+\frac{144U_{\omega}^{2}}{c_{2}^{2}}\frac{\zeta}{L^{2}}+4\lambda\zeta\right)\bar{X}_{t}+\left({\mathcal{C}}_{M,5}(\tau^{2})\zeta^{2}+\frac{4U_{\omega}^{2}}{c_{1}^{2}}\zeta\right)\bar{Y}_{t} \displaystyle\qquad+4C_{25}(\tau^{2})\frac{\zeta^{2}}{L}+{\mathcal{C}}_{M,1}\zeta^{4}+4C_{28}(\tau^{4})\gamma^{2}\zeta^{2}+{\mathcal{C}}_{M,2}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{C}}_{M,3}\zeta^{6}+{\mathcal{C}}_{M,6}\zeta^{3}, (89) where \displaystyle{\mathcal{C}}_{M,1}=\left(4C_{27}(\tau^{4})+8\tau^{2}\frac{U_{\delta,L}^{4}}{L^{4}}c^{2}+4C_{26}(\tau^{4})\frac{1}{L^{2}}\right),{\mathcal{C}}_{M,2}(\tau^{2})=\left(32\tau^{2}C_{20}+8\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{2}\right), \displaystyle{\mathcal{C}}_{M,3}=\left(32\tau^{2}C_{21}+16U_{\omega}^{2}\frac{U_{\delta,L}^{2}}{L^{2}}\right)m^{2},{\mathcal{C}}_{M,4}(\tau^{2})=\left[\frac{32U_{\omega}^{2}}{L^{2}}+\frac{4C_{29}(\tau^{2})}{L}\right],{\mathcal{C}}_{M,5}(\tau^{2})=\left[4C_{30}(\tau^{2})+8U_{\omega}^{2}\right], \displaystyle{\mathcal{C}}_{M,6}=\left[8U_{r}U_{\omega}m+8U_{\omega}^{2}m\frac{1}{L}\right]. Appendix F Proof of Lemma 5.4 By Eq. ( 11 ) and the update of \eta^{k} in Algorithm 1 , we have \displaystyle y_{t+1}^{k}= \displaystyle\eta_{t+1}^{k}-J^{k}=\eta_{t}^{k}+\gamma\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{t,\ell}^{k}-\eta_{t}^{k}\right)-J^{k}=y_{t}^{k}+\gamma\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{t,\ell}^{k}-\eta_{t}^{k}\right) \displaystyle= \displaystyle y_{t}^{k}+\gamma\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{t,\ell}^{k}-J^{k}+J^{k}-\eta_{t}^{k}\right) \displaystyle= \displaystyle y_{t}^{k}+\gamma\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{t,\ell}^{k}-J^{k}-y_{t}^{k}\right) \displaystyle= \displaystyle(1-\gamma)y_{t}^{k}+\gamma\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{t,\ell}^{k}-J^{k}\right) \displaystyle= \displaystyle(1-\gamma)^{t+1}y_{0}^{k}+\gamma\sum_{i=0}^{t}(1-\gamma)^{t-i}\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{i,\ell}^{k}-J^{k}\right). Then, \displaystyle(y_{t}^{k})^{2}= \displaystyle(1-\gamma)^{2t}(y_{0}^{k})^{2}+\gamma^{2}\left(\sum_{i=0}^{t-1}(1-\gamma)^{t-1-i}\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{i,\ell}^{k}-J^{k}\right)\right)^{2} \displaystyle+2(1-\gamma)^{t}y_{0}^{k}\gamma\sum_{i=0}^{t-1}(1-\gamma)^{t-1-i}\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{i,\ell}^{k}-J^{k}\right). Taking expectation, we get \displaystyle{\mathbb{E}}(y_{t}^{k})^{2}= \displaystyle\underbrace{(1-\gamma)^{2t}(y_{0}^{k})^{2}}_{I_{1}}+\underbrace{\gamma^{2}{\mathbb{E}}\left(\sum_{i=0}^{t-1}(1-\gamma)^{t-1-i}\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{i,\ell}^{k}-J^{k}\right)\right)^{2}}_{I_{2}} \displaystyle+\underbrace{2(1-\gamma)^{t}y_{0}^{k}\gamma{\mathbb{E}}\sum_{i=0}^{t-1}(1-\gamma)^{t-1-i}\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{i,\ell}^{k}-J^{k}\right)}_{I_{3}}. (90) For I_{1} in Eq. ( F ), using the fact that \ln(1-\gamma)\leq-\gamma for \gamma\in(0,1) , we have \displaystyle I_{1}=(1-\gamma)^{2t}(y_{0}^{k})^{2}\leq(y_{0}^{k})^{2}e^{-2\gamma t}. For I_{3} in Eq. ( F ), we have, \displaystyle{\mathbb{E}}\sum_{i=0}^{t-1}(1-\gamma)^{t-1-i}\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{i,\ell}^{k}-J^{k}\right) \displaystyle= \displaystyle{\mathbb{E}}\sum_{i=0}^{\tau-1}(1-\gamma)^{t-1-i}\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{i,\ell}^{k}-J^{k}\right)+{\mathbb{E}}\sum_{i=\tau}^{t-1}(1-\gamma)^{t-1-i}\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{i,\ell}^{k}-J^{k}\right) \displaystyle\leq \displaystyle 2U_{r}\frac{(1-\gamma)^{t-\tau}(1-(1-\gamma)^{\tau})}{1-(1-\gamma)}+\sum_{i=\tau}^{t-1}(1-\gamma)^{t-1-i}{\mathbb{E}}\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{i,\ell}^{k}-J^{k}\right) \displaystyle\leq \displaystyle 2U_{r}\frac{(1-\gamma)^{t-\tau}}{\gamma}+2U_{r}\sum_{i=\tau}^{t-1}(1-\gamma)^{t-1-i}\sup_{s\in{\mathcal{S}}}d_{TV}(P_{0:i}^{k}(\cdot,\cdot,\cdot,...,\cdot\mid s_{0}^{k}=s),\mu^{k}\otimes(\pi\otimes P^{k})^{L-1}\otimes\pi) \displaystyle\leq \displaystyle 2U_{r}\frac{(1-\gamma)^{t-\tau}}{\gamma}+2U_{r}\sum_{i=\tau}^{t-1}(1-\gamma)^{t-1-i}m\rho^{i} \displaystyle\leq \displaystyle 2U_{r}\frac{(1-\gamma)^{t-\tau}}{\gamma}+\frac{2U_{r}m\rho^{\tau}}{\gamma}. where the first inequality follows because \left|\frac{1}{L}\sum_{\ell=0}^{L-1}r_{i,\ell}^{k}-J^{k}\right|\leq 2U_{r} , and the third inequality follows from Assumption 2 . In the second inequality, we slightly abuse notation to use P_{0:i}^{k}(\cdot,\cdot,\cdot,...,\cdot\mid s_{0}^{k}=s) to denote the L -block state-action pair at time t . It holds because the mixing of the L -block is no slower than the original. Thus, \displaystyle I_{3}\leq \displaystyle 2(1-\gamma)^{t}|y_{0}^{k}|\left(2U_{r}(1-\gamma)^{t-\tau}+2U_{r}m\rho^{\tau}\right) \displaystyle\leq \displaystyle 4|y_{0}^{k}|U_{r}(1-\gamma)^{2t-\tau}+4U_{r}(1-\gamma)^{t}|y_{0}^{k}|m\rho^{\tau}. Lastly, for I_{2} in Eq. ( F ), we have \displaystyle{\mathbb{E}}\left(\sum_{i=0}^{t-1}(1-\gamma)^{t-1-i}\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{i,\ell}^{k}-J^{k}\right)\right)^{2} \displaystyle= \displaystyle{\mathbb{E}}\sum_{i=0}^{t-1}(1-\gamma)^{2t-2-2i}\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{i,\ell}^{k}-J^{k}\right)^{2} \displaystyle+2{\mathbb{E}}\sum_{i=0}^{t-1}\sum_{j=i+1}^{t-1}(1-\gamma)^{t-1-i}(1-\gamma)^{t-1-j}\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{i,\ell}^{k}-J^{k}\right)\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{j,\ell}^{k}-J^{k}\right) \displaystyle\leq \displaystyle\frac{1-(1-\gamma)^{2t}}{1-(1-\gamma)^{2}}4U_{r}^{2}+\underbrace{2{\mathbb{E}}\sum_{i=0}^{t-\tau-1}\sum_{j=i+\tau}^{t-1}(1-\gamma)^{t-1-i}(1-\gamma)^{t-1-j}\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{i,\ell}^{k}-J^{k}\right)\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{j,\ell}^{k}-J^{k}\right)}_{I_{21}} \displaystyle+\underbrace{2{\mathbb{E}}\sum_{i=0}^{t-\tau-1}\sum_{j=i+1}^{i+\tau-1}(1-\gamma)^{t-1-i}(1-\gamma)^{t-1-j}\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{i,\ell}^{k}-J^{k}\right)\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{j,\ell}^{k}-J^{k}\right)}_{I_{22}} \displaystyle+\underbrace{2{\mathbb{E}}\sum_{i=t-\tau}^{t-1}\sum_{j=i+1}^{t-1}(1-\gamma)^{t-1-i}(1-\gamma)^{t-1-j}\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{i,\ell}^{k}-J^{k}\right)\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{j,\ell}^{k}-J^{k}\right)}_{I_{23}}. (91) Recall v_{0:t}^{k} is the Markovian trajectory at agent k : \displaystyle v_{0:t}^{k}=(s_{0}^{k},a_{0}^{k},s_{1}^{k},a_{1}^{k},...,s_{t-\tau+1}^{k},a_{t-\tau+1}^{k},...,s_{t}^{k},a_{t}^{k},s_{t+1}^{k}). (92) For I_{21} in Eq. ( F ), we have \displaystyle I_{21}= \displaystyle 2\sum_{i=0}^{t-\tau-1}\sum_{j=i+\tau}^{t-1}(1-\gamma)^{t-1-i}(1-\gamma)^{t-1-j}{\mathbb{E}}\left[{\mathbb{E}}\left[\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{i,\ell}^{k}-J^{k}\right)\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{j,\ell}^{k}-J^{k}\right)|v_{0:i}^{k}\right]\right] \displaystyle\leq \displaystyle 2\sum_{i=0}^{t-\tau-1}\sum_{j=i+\tau}^{t-1}(1-\gamma)^{t-1-i}(1-\gamma)^{t-1-j}2U_{r}{\mathbb{E}}\left[\left|{\mathbb{E}}\left[\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{j,\ell}^{k}-J^{k}\right)|v_{0:i}^{k}\right]\right|\right] \displaystyle\leq \displaystyle 2\sum_{i=0}^{t-\tau-1}\sum_{j=i+\tau}^{t-1}(1-\gamma)^{t-1-i}(1-\gamma)^{t-1-j}4U^{2}_{r}m\rho^{\tau} \displaystyle\leq \displaystyle 8U^{2}_{r}m\rho^{\tau}\sum_{i=0}^{t-\tau-1}\sum_{j=i+\tau}^{t-1}(1-\gamma)^{t-1-i}(1-\gamma)^{t-1-j} \displaystyle\leq \displaystyle 8U^{2}_{r}m\rho^{\tau}\sum_{i=0}^{t-\tau-1}(1-\gamma)^{t-1-i}\frac{1-(1-\gamma)^{t-i-\tau}}{\gamma} \displaystyle\leq \displaystyle 8U^{2}_{r}m\rho^{\tau}\frac{(1-\gamma)^{\tau}}{\gamma^{2}}. Noting that \displaystyle\sum_{i=0}^{t-\tau-1}\sum_{j=i+1}^{i+\tau-1}(1-\gamma)^{t-1-i}(1-\gamma)^{t-1-j} \displaystyle= \displaystyle\sum_{i=0}^{t-\tau-1}(1-\gamma)^{t-1-i}\left(\sum_{j=i+1}^{i+\tau-1}(1-\gamma)^{t-1-j}\right) \displaystyle= \displaystyle\sum_{i=0}^{t-\tau-1}(1-\gamma)^{t-1-i}\frac{(1-\gamma)^{t-i-\tau}(1-(1-\gamma)^{\tau})}{\gamma} \displaystyle= \displaystyle\frac{(1-(1-\gamma)^{\tau})}{\gamma}\sum_{i=0}^{t-\tau-1}(1-\gamma)^{2t-1-2i-\tau}. Since (1-\gamma)^{\tau}\geq 1-\tau\gamma , we have \displaystyle\frac{(1-(1-\gamma)^{\tau})}{\gamma}\leq\frac{1-(1-\tau\gamma)}{\gamma}=\tau. In addition, \displaystyle\sum_{i=0}^{t-\tau-1}(1-\gamma)^{2t-1-2i-\tau}\leq\frac{(1-\gamma)^{\tau+1}}{1-(1-\gamma)^{2}}\leq\frac{(1-\gamma)^{\tau}}{\gamma}. Thus, I_{22} in Eq. ( F ) can be bounded as \displaystyle I_{22}\leq 8U_{r}^{2}\tau\frac{(1-\gamma)^{\tau}}{\gamma}. For I_{23} in Eq. ( F ), we have \displaystyle I_{23} \displaystyle\leq 2\sum_{i=t-\tau}^{t-1}\sum_{j=i+1}^{t-1}4U_{r}^{2}\leq 8U_{r}^{2}\frac{\tau(\tau-1)}{2}. Now, we can bound I_{2} in Eq. ( F ) as \displaystyle I_{2}\leq\gamma^{2}\left(\frac{1-(1-\gamma)^{2t}}{1-(1-\gamma)^{2}}4U_{r}^{2}+8U^{2}_{r}m\rho^{\tau}\frac{(1-\gamma)^{\tau}}{\gamma^{2}}+8U_{r}^{2}\tau\frac{(1-\gamma)^{\tau}}{\gamma}+8U_{r}^{2}\frac{\tau(\tau-1)}{2}\right) \displaystyle\leq \displaystyle\frac{\gamma}{2-\gamma}4U_{r}^{2}+8U_{r}m\rho^{\tau}(1-\gamma)^{\tau}+8U_{r}^{2}\tau\gamma(1-\gamma)^{\tau}+8U_{r}^{2}\frac{\tau(\tau-1)}{2}\gamma^{2}. Putting I_{1},I_{2} and I_{3} together, and choosing \gamma=\frac{\log(T)}{2T} , we get \displaystyle{\mathbb{E}}(y_{T}^{k})^{2}\leq(y_{0}^{k})^{2}e^{-2\gamma T}+\frac{\gamma}{2-\gamma}4U_{r}^{2}+8U^{2}_{r}m\rho^{\tau}(1-\gamma)^{\tau}+8U_{r}^{2}\tau\gamma(1-\gamma)^{\tau} \displaystyle+8U_{r}^{2}\frac{\tau(\tau-1)}{2}\gamma^{2}+4|y_{0}^{k}|U_{r}e^{\gamma\tau}e^{-\gamma 2T}+4U_{r}(1-\gamma)^{T}|y_{0}^{k}|m\rho^{\tau} \displaystyle\leq 4U_{r}^{2}\frac{1}{T}+\frac{4U_{r}^{2}\log(T)}{2T}+8U_{r}m\rho^{\tau}+8U_{r}^{2}\tau\frac{\log(T)}{2T} \displaystyle+8U_{r}^{2}\frac{\tau(\tau-1)}{2}\frac{\log^{2}(T)}{4T^{2}}+8U_{r}^{2}e^{\gamma\tau}\frac{1}{T}+8U_{r}\frac{1}{\sqrt{T}}m\rho^{\tau}. When \tau\leq\min\{\frac{2T}{\log(T)},\lceil\log_{\rho}(\gamma)\rceil,\lceil 2\log_{\rho}(\zeta)\rceil\} , it holds that e^{\gamma\tau}\leq 3 and that 8U_{r}^{2}\frac{\tau(\tau-1)}{2}\frac{\log^{2}(T)}{4T^{2}}\leq\frac{2\tau U_{r}^{2}\log(T)}{T} , and \rho^{\tau}\leq\gamma . Thus, the upper bound of {\mathbb{E}}(y_{T}^{k})^{2} can be simplified as \displaystyle{\mathbb{E}}(y_{T}^{k})^{2}\leq\frac{28U_{r}^{2}}{T}+\frac{8U_{r}^{2}\tau\log(T)}{T}+16U_{r}m\rho^{\tau}\leq\frac{28U_{r}^{2}}{T}+\frac{8U_{r}^{2}\tau\log(T)}{T}+\frac{8U_{r}m\log(T)}{T}, (93) which trivially implies that \displaystyle\bar{Y}_{T}=\frac{1}{K}\sum_{k=1}^{K}Y_{T}^{k}\leq\frac{28U_{r}^{2}}{T}+\frac{8U_{r}^{2}\tau\log(T)}{T}+\frac{8U_{r}m\log(T)}{T}. Appendix G Proof of Theorem 1 For ease of expostion, let \bar{Y}_{T}=\frac{1}{K}\sum_{k=1}^{K}Y_{T}^{k} . From Lemma 5.4 (Eq. ( 93 )), Lemma D.1 , Lemma 5.2 , and Lemma 5.3 (Eq. ( E )), we get, for T\geq 2\tau , \displaystyle\bar{Y}_{T}\leq\frac{28U_{r}^{2}}{T}+\frac{8U_{r}^{2}\tau\log(T)}{T}+\frac{8U_{r}m\log(T)}{T}, \displaystyle\bar{X}_{T+1}\leq\Bigg(1-2\lambda c\zeta+\Big(c_{3}^{2}\frac{c}{L}+c_{4}^{2}c+c_{5}^{2}\frac{1}{L}+c_{6}^{2}+4U_{\omega}^{4}\frac{1}{c_{5}^{2}L}\Big)\zeta+{\mathcal{C}}_{X,1}(\tau^{2})\zeta^{2}+{\mathcal{C}}_{X,2}(\tau^{2})\zeta^{3}\Bigg)\bar{X}_{T} \displaystyle\qquad+\left(\frac{36U_{\omega}^{2}}{c_{3}^{2}}\frac{c\zeta}{L}+6U_{\omega}U_{\delta}c\zeta\right)M_{T} \displaystyle\qquad+\Bigg(\Big[\frac{1}{c_{4}^{2}}c+\frac{U_{\omega}^{4}}{c_{6}^{2}}\Big]\zeta+{\mathcal{C}}_{X,3}(\tau^{2})\zeta^{2}+{\mathcal{C}}_{X,4}(\tau^{2})\zeta^{3}\Bigg)\bar{Y}_{T} \displaystyle\qquad+{\mathcal{C}}_{X,5}(\tau^{2})\frac{\zeta^{2}}{L}+{\mathcal{C}}_{X,6}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{C}}_{X,7}\zeta^{3}+{\mathcal{C}}_{X,8}\zeta^{4}+{\mathcal{C}}_{X,9}\gamma^{2}\zeta^{2}+{\mathcal{C}}_{X,10}\zeta^{5}+{\mathcal{C}}_{X,11}\gamma^{2}\zeta^{3}+{\mathcal{C}}_{X,12}\zeta^{7}+{\mathcal{C}}_{X,13}\zeta^{6}, \displaystyle M_{T+1}\leq(1-\frac{\lambda\alpha\zeta}{r}+c_{1}^{2}\zeta+c_{2}^{2}\zeta)M_{T} \displaystyle\qquad+\left({\mathcal{C}}_{M,4}(\tau^{2})\zeta^{2}+\frac{16U_{\omega}^{2}}{c_{2}^{2}}\frac{\zeta}{L^{2}}\right)\bar{X}_{T}+\left({\mathcal{C}}_{M,5}(\tau^{2})\zeta^{2}+\frac{4U_{\omega}^{2}}{c_{1}^{2}}\zeta\right)\bar{Y}_{T} \displaystyle\qquad+4C_{25}(\tau^{2})\frac{\zeta^{2}}{L}+{\mathcal{C}}_{M,1}\zeta^{4}+4C_{28}(\tau^{4})\gamma^{2}\zeta^{2}+{\mathcal{C}}_{M,2}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{C}}_{M,3}\zeta^{6}+{\mathcal{C}}_{M,6}\zeta^{3}. Note that \bar{Y}_{T}={\mathcal{O}}(\frac{\log(T)}{T}) , and we collect all terms of order \zeta^{3} and higher into {\mathcal{O}}(\zeta^{3}) , then the system can be simplified to, \displaystyle\bar{X}_{T+1}\leq\Bigg(1-2\lambda c\zeta+\Big(c_{3}^{2}\frac{c}{L}+c_{4}^{2}c+c_{5}^{2}\frac{1}{L}+c_{6}^{2}+4U_{\omega}^{4}\frac{1}{c_{5}^{2}L}\Big)\zeta+{\mathcal{C}}_{X,1}(\tau^{2})\zeta^{2}+{\mathcal{C}}_{X,2}(\tau^{2})\zeta^{3}\Bigg)\bar{X}_{T} \displaystyle\qquad+\left(\frac{36U_{\omega}^{2}}{c_{3}^{2}}\frac{c\zeta}{L}+6U_{\omega}U_{\delta}c\zeta\right)M_{T}+{\mathcal{C}}_{X,5}(\tau^{2})\frac{\zeta^{2}}{L}+{\mathcal{C}}_{X,6}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{O}}(\frac{\zeta\log(T)}{T})+{\mathcal{O}}(\zeta^{3}), \displaystyle M_{T+1}\leq(1-\frac{\lambda\alpha\zeta}{r}+c_{1}^{2}\zeta+c_{2}^{2}\zeta)M_{T} \displaystyle\qquad+\left({\mathcal{C}}_{M,4}(\tau^{2})\zeta^{2}+\frac{16U_{\omega}^{2}}{c_{2}^{2}}\frac{\zeta}{L^{2}}\right)+4C_{25}(\tau^{2})\frac{\zeta^{2}}{L}+{\mathcal{C}}_{M,2}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{O}}(\zeta^{3})+{\mathcal{O}}(\frac{\zeta\log(T)}{T}). Choosing c_{3}=\sqrt{\frac{\lambda L}{30}},c_{4}=\sqrt{\frac{\lambda}{30}},c_{5}=1,c_{6}=\min\{\sqrt{\frac{1}{L}},\sqrt{\frac{4U_{\omega}^{4}}{L}}\} , and \displaystyle c\geq\max\{\frac{30}{\lambda L},\frac{120U_{\omega}^{4}}{\lambda L}\}, (94) and requiring \zeta\leq\frac{\lambda c}{6{\mathcal{C}}_{X,1}(\tau^{2})},\zeta^{2}\leq\frac{\lambda c}{6{\mathcal{C}}_{X,2}(\tau^{2})} , we get \displaystyle\bar{X}_{T+1}\leq\left(1-\frac{3}{2}\lambda c\zeta\right)\bar{X}_{T}+\left(\frac{36U_{\omega}^{2}}{c_{3}^{2}}\frac{c\zeta}{L}+6U_{\omega}U_{\delta}c\zeta\right)M_{T}+{\mathcal{C}}_{X,5}(\tau^{2})\frac{\zeta^{2}}{L}+{\mathcal{C}}_{X,6}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{O}}(\frac{\zeta\log(T)}{T})+{\mathcal{O}}(\zeta^{3}). We choose c_{1}=c_{2}=\sqrt{\frac{\lambda\alpha}{4r}} , which requires {\mathcal{C}}_{M,4}(\tau^{2})\zeta\leq\frac{16U_{\omega}^{2}}{c_{2}^{2}L^{2}} , we have \displaystyle M_{T+1}\leq\left(1-\frac{\lambda\alpha\zeta}{2r}\right)M_{T}+\frac{32U_{\omega}^{2}}{c_{2}^{2}}\frac{\zeta}{L^{2}}\bar{X}_{T}+4C_{25}(\tau^{2})\frac{\zeta^{2}}{L}+{\mathcal{C}}_{M,2}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{O}}(\zeta^{3})+{\mathcal{O}}(\frac{\zeta\log(T)}{T}). Substituting c_{2} and c_{3} , we get, \displaystyle\bar{X}_{T+1}\leq\left(1-\frac{3}{2}\lambda c\zeta\right)\bar{X}_{T}+\left(\frac{1080U_{\omega}^{2}}{\lambda L^{2}}c\zeta+6U_{\omega}U_{\delta}c\zeta\right)M_{T}+{\mathcal{C}}_{X,5}(\tau^{2})\frac{\zeta^{2}}{L}+{\mathcal{C}}_{X,6}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{O}}(\zeta\frac{\log(T)}{T})+{\mathcal{O}}(\zeta^{3}). \displaystyle M_{T+1}\leq\left(1-\frac{\lambda\alpha\zeta}{2r}\right)M_{T}+\frac{128rU_{\omega}^{2}}{\lambda\alpha}\frac{\zeta}{L^{2}}\bar{X}_{T}+4C_{25}(\tau^{2})\frac{\zeta^{2}}{L}+{\mathcal{C}}_{M,2}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{O}}(\zeta^{3})+{\mathcal{O}}(\zeta\frac{\log(T)}{T}). Defining the weighted Lyapunov function as {\mathcal{V}}_{T+1}=\bar{X}_{T+1}+\kappa M_{T+1} , we have, \displaystyle{\mathcal{V}}_{T+1}\leq \displaystyle\left(1-\frac{3}{2}\lambda c\zeta+\kappa\frac{128rU_{\omega}^{2}}{\lambda\alpha}\frac{\zeta}{L^{2}}\right)\bar{X}_{T}+{\mathcal{C}}_{X,5}(\tau^{2})\frac{\zeta^{2}}{L}+{\mathcal{C}}_{X,6}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{O}}(\zeta\frac{\log(T)}{T})+{\mathcal{O}}(\zeta^{3}) \displaystyle+\kappa\left(\left(1-\frac{\lambda\alpha\zeta}{2r}+\frac{1}{\kappa}\left(\frac{1080U_{\omega}^{2}}{\lambda L^{2}}c\zeta+6U_{\omega}U_{\delta}c\zeta\right)\right)M_{T}+4C_{25}(\tau^{2})\frac{\zeta^{2}}{L}+{\mathcal{C}}_{M,2}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{O}}(\zeta^{3})+{\mathcal{O}}(\zeta\frac{\log(T)}{T})\right) For sufficiently large L , c satisfies Eq. ( 94 ) and the following condition \displaystyle\kappa\frac{256rU_{\omega}^{2}}{\lambda^{2}\alpha L^{2}}\leq c\leq\kappa\frac{\lambda^{2}\alpha L^{2}}{4r\left(1080U_{\omega}^{2}+6U_{\omega}U_{\delta}\lambda L^{2}\right)} (95) such that \displaystyle\frac{128rU_{\omega}^{2}}{\lambda\alpha L^{2}}\kappa\leq\frac{1}{2}\lambda c, \displaystyle\frac{1}{\kappa}\left(\frac{1080U_{\omega}^{2}}{\lambda L^{2}}c+6U_{\omega}U_{\delta}c\right)\leq\frac{\lambda\alpha}{4r}, then, \displaystyle{\mathcal{V}}_{T+1}\leq \displaystyle\left(1-\lambda c\zeta\right)\bar{X}_{T}+{\mathcal{C}}_{X,5}(\tau^{2})\frac{\zeta^{2}}{L}+{\mathcal{C}}_{X,6}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{O}}(\zeta\frac{\log(T)}{T})+{\mathcal{O}}(\zeta^{3}) \displaystyle+\kappa\left(\left(1-\frac{\lambda\alpha\zeta}{4r}\right)M_{T}+4C_{25}(\tau^{2})\frac{\zeta^{2}}{L}+{\mathcal{C}}_{M,2}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{O}}(\zeta^{3})+{\mathcal{O}}(\zeta\frac{\log(T)}{T})\right) (96) Choosing \kappa\geq\frac{2(1080U_{\omega}^{2}+6U_{\omega}U_{\delta}\lambda L^{2})}{\lambda^{2}L^{2}}=\Theta(1) , the upper bound of c in Eq. ( 95 ) satisfies, \displaystyle\kappa\frac{\lambda^{2}\alpha L^{2}}{4r\left(1080U_{\omega}^{2}+6U_{\omega}U_{\delta}\lambda L^{2}\right)}\geq\frac{\alpha}{2r}, leaving a room to further constrain c to be greater than \frac{\alpha}{4r} and choose L=K , such that Eq. ( G ) can be written as, \displaystyle{\mathcal{V}}_{T+1}\leq \displaystyle\left(1-\frac{\lambda\alpha\zeta}{4r}\right)\bar{X}_{T}+{\mathcal{C}}_{X,5}(\tau^{2})\frac{\zeta^{2}}{L}+{\mathcal{C}}_{X,6}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{O}}(\zeta\frac{\log(T)}{T})+{\mathcal{O}}(\zeta^{3}) \displaystyle+\kappa\left(1-\frac{\lambda\alpha\zeta}{4r}\right)M_{T}+4\kappa C_{25}(\tau^{2})\frac{\zeta^{2}}{L}+\kappa{\mathcal{C}}_{M,2}(\tau^{2})\frac{\zeta^{2}}{K}+\kappa{\mathcal{O}}(\zeta^{3})+\kappa{\mathcal{O}}(\zeta\frac{\log(T)}{T}) \displaystyle\leq \displaystyle\left(1-\frac{\lambda\alpha\zeta}{4r}\right){\mathcal{V}}_{T}+{\mathcal{C}}_{V}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{O}}(\zeta\frac{\log(T)}{T})+{\mathcal{O}}(\zeta^{3}), (97) where \displaystyle{\mathcal{C}}_{V}(\tau^{2})=\left({\mathcal{C}}_{X,5}(\tau^{2})+{\mathcal{C}}_{X,6}(\tau^{2})+4\kappa C_{25}(\tau^{2})+\kappa{\mathcal{C}}_{M,2}(\tau^{2})\right). Unrolling Eq. ( G ), we get \displaystyle{\mathcal{V}}_{T+1}\leq \displaystyle\left(1-\frac{\lambda\alpha\zeta}{4r}\right){\mathcal{V}}_{T}+{\mathcal{C}}_{V}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{O}}(\zeta\frac{\log(T)}{T})+{\mathcal{O}}(\zeta^{3}) \displaystyle\leq \displaystyle\left(1-\frac{\lambda\alpha\zeta}{4r}\right)^{T-2\tau+1}{\mathcal{V}}_{2\tau}+\sum_{i=0}^{T-2\tau}\left(1-\frac{\lambda\alpha\zeta}{4r}\right)^{i}\left[{\mathcal{C}}_{V}(\tau^{2})\frac{\zeta^{2}}{K}+{\mathcal{O}}(\zeta\frac{\log(T)}{T})+{\mathcal{O}}(\zeta^{3})\right] \displaystyle\leq \displaystyle\exp\left\{-\frac{\lambda\alpha\zeta(T-2\tau+1)}{4r}\right\}+\frac{4r}{\lambda\alpha}\left[{\mathcal{C}}_{V}(\tau^{2})\frac{\zeta}{K}+{\mathcal{O}}(\frac{\log(T)}{T})+{\mathcal{O}}(\zeta^{2})\right]. Choosing \zeta=\frac{4r}{\sqrt{T}\lambda\alpha} , we have, \displaystyle{\mathcal{V}}_{T+1}\leq\exp\left\{-\sqrt{T}/2\right\}+\frac{16r^{2}}{\lambda^{2}\alpha^{2}}{\mathcal{C}}_{V}(\tau^{2})\frac{1}{\sqrt{T}K}+{\mathcal{O}}(\frac{\log(T)}{T}). Lastly, collecting the sufficient conditions on c and K , we can always choose \displaystyle\max\left\{\frac{30}{\lambda K},\frac{120U_{\omega}^{4}}{\lambda K}\right\}\leq\frac{\alpha}{2r};\quad\frac{\alpha}{4r}\leq c\leq\frac{\alpha}{2r}. Appendix H Supporting Propositions and Lemmas H.1 Boundedness Lemma H.1 . For any t\geq 0,L\geq 0,k\in[K] , it holds that \eta_{t}^{k}\in[-U_{r},U_{r}] . Proof. We prove Lemma H.1 by induction. The algorithm initializes \eta_{0}^{k}=0 . Suppose that \eta_{t^{\prime}}^{k}\in[-U_{r},U_{r}] for 0\leq t^{\prime}\leq t-1 and for any k\in[K] . We have \displaystyle\eta_{t}^{k}=\eta_{t-1}^{k}+\gamma\left(\frac{1}{L}\sum_{\ell=0}^{L-1}r_{t-1,\ell}^{k}-\eta^{k}_{t-1}\right)=(1-\gamma)\eta_{t-1}^{k}+\frac{\gamma}{L}\sum_{\ell=0}^{L-1}r_{t-1,\ell}^{k}\leq(1-\gamma)U_{r}+\frac{\gamma}{L}L\cdot U_{r}=U_{r}. Similarly, it can be shown that \eta_{t}^{k}\geq-U_{r} . ∎ Lemma H.2 . Suppose that Assumption 3 holds. Define U_{\delta,L}:=2LU_{r}+2U_{\omega}. For any t\geq 0,L\geq 0,k\in[K] , it holds that \displaystyle|\delta^{k}_{t,L}|\leq U_{\delta,L},~~~~\text{and}~~~\|{\mathbf{b}}_{t,L}^{k}\|\leq U_{\delta,L}. (98) Proof. From Eq. ( 6 ), \displaystyle|\delta^{k}_{t,L}| \displaystyle=|\sum_{\ell=0}^{L-1}(r_{t,\ell}^{k}-\eta_{t}^{k})+(\phi(s_{t,L}^{k})-\phi(s_{t,0}^{k}))^{\top}\mathbf{B}_{t}\omega_{t}^{k}| \displaystyle\leq|\sum_{\ell=0}^{L-1}(r_{t,\ell}^{k}-\eta_{t}^{k})|+|(\phi(s_{t,L}^{k})-\phi(s_{t,0}^{k}))^{\top}\mathbf{B}_{t}\omega_{t}^{k}| \displaystyle\leq 2LU_{r}+2U_{\omega}, where the last inequality holds due to Lemma H.1 and \|\phi(s^{k}_{t+1})-\phi(s^{k}_{t})\|\leq 2 . Recall from Eq. ( B ) that \displaystyle{\mathbf{b}}_{t,L}^{k}=\sum_{\ell=0}^{L-1}\left((r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\right)\phi(s_{t,0}^{k}). So, \displaystyle\|{\mathbf{b}}_{t,L}^{k}\| \displaystyle=\|\sum_{\ell=0}^{L-1}\left((r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\right)\phi(s_{t,0}^{k})\| \displaystyle\leq\|\sum_{\ell=0}^{L-1}(r_{t,\ell}^{k}-J^{k})\|+\|(\phi(s_{t,L}^{k})-\phi(s_{t,0}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\| \displaystyle\leq 2LU_{r}+2U_{\omega}=U_{\delta,L}. ∎ Appendix I Other Supporting Results Proof of Proposition 1 . It follows from Eq. ( 6 ) that for any t and L : \displaystyle\delta_{t,L}^{k}\phi(s_{t,0}^{k})=\left(\sum_{\ell=0}^{L-1}(r_{t,\ell}^{k}-\eta_{t}^{k})+(\phi(s_{t,L}^{k})-\phi(s_{t,0}^{k}))^{\top}\mathbf{B}_{t}\omega_{t}^{k}\right)\phi(s_{t,0}^{k}) \displaystyle= \displaystyle\sum_{\ell=0}^{L-1}\left((r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}_{t}\omega_{t}^{k}\right)\phi(s_{t,0}^{k})+\left(\sum_{\ell=0}^{L-1}(J^{k}-\eta_{t}^{k})\right)\phi(s_{t,0}^{k}) \displaystyle= \displaystyle\underbrace{\sum_{\ell=0}^{L-1}\phi(s_{t,0}^{k})(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}}_{\widetilde{A}_{t,L}^{k}}x_{t}^{k}\quad\quad~~~\text{(recall that $x_{t}^{k}=\mathbf{B}_{t}\omega_{t}^{k}-\mathbf{B}^{*}\omega^{k,*}$)} \displaystyle+\underbrace{\sum_{\ell=0}^{L-1}\left((r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\right)\phi(s_{t,0}^{k})}_{{\mathbf{b}}_{t,L}^{k}}+Ly_{t}^{k}\phi(s_{t,0}^{k}) \displaystyle= \displaystyle\widetilde{A}_{t,L}^{k}x_{t}^{k}+{\mathbf{b}}_{t,L}^{k}+Ly_{t}^{k}\phi(s_{t,0}^{k}), proving Eq. ( 17 ). In the remaining proof, we focus on proving the second decomposition in Eq. ( 18 ). We have, \displaystyle\delta_{t,L}^{k}\phi(s_{t,0}^{k})=\left(\sum_{\ell=0}^{L-1}(r_{t,\ell}^{k}-\eta_{t}^{k})+(\phi(s_{t,L}^{k})-\phi(s_{t,0}^{k}))^{\top}\mathbf{B}_{t}\omega_{t}^{k}\right)\phi(s_{t,0}^{k}) \displaystyle= \displaystyle\sum_{\ell=0}^{L-1}\left((r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}_{t}\omega_{t}^{k}\right)\phi(s_{t,0}^{k})+\left(\sum_{\ell=0}^{L-1}(J^{k}-\eta_{t}^{k})\right)\phi(s_{t,0}^{k}) \displaystyle= \displaystyle\sum_{\ell=0}^{L-1}\left((r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}_{t}\omega_{t}^{k}\right)\phi(s_{t,0}^{k}) \displaystyle-\sum_{\ell=0}^{L-1}{\mathbb{E}}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left((R(s^{(\ell)},a^{(\ell)})-J^{k})+(\phi(s^{(\ell+1)})-\phi(s^{(\ell)}))^{\top}\mathbf{B}_{t}\omega_{t}^{k}\right)\phi(s^{(0)}) \displaystyle+\sum_{\ell=0}^{L-1}{\mathbb{E}}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left((R(s^{(\ell)},a^{(\ell)})-J^{k})+(\phi(s^{(\ell+1)})-\phi(s^{(\ell)}))^{\top}\mathbf{B}_{t}\omega_{t}^{k}\right)\phi(s^{(0)}) \displaystyle+L(J^{k}-\eta_{t}^{k})\phi(s_{t,0}^{k}). Under steady-state distribution, by the Bellman equation, we have \displaystyle{\mathbb{E}}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left((R(s^{(\ell)},a^{(\ell)})-J^{k})+(\phi(s^{(\ell+1)})-\phi(s^{(\ell)}))^{\top}z^{k,*}\right)\phi(s^{(0)})=0. (99) Then, \displaystyle\sum_{\ell=0}^{L-1}{\mathbb{E}}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left((R(s^{(\ell)},a^{(\ell)})-J^{k})+(\phi(s^{(\ell+1)})-\phi(s^{(\ell)}))^{\top}\mathbf{B}_{t}\omega_{t}^{k}\right)\phi(s^{(0)}) \displaystyle= \displaystyle\sum_{\ell=0}^{L-1}{\mathbb{E}}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left((R(s^{(\ell)},a^{(\ell)})-J^{k})+(\phi(s^{(\ell+1)})-\phi(s^{(\ell)}))^{\top}\mathbf{B}_{t}\omega_{t}^{k}\right)\phi(s^{(0)}) \displaystyle-\sum_{\ell=0}^{L-1}{\mathbb{E}}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left((R(s^{(\ell)},a^{(\ell)})-J^{k})+(\phi(s^{(\ell+1)})-\phi(s^{(\ell)}))^{\top}z^{k,*}\right)\phi(s^{(0)}) \displaystyle= \displaystyle\sum_{\ell=0}^{L-1}{\mathbb{E}}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}(\phi(s^{(\ell+1)})-\phi(s^{(\ell)}))^{\top}x_{t}^{k}\phi(s^{(0)}) \displaystyle= \displaystyle{\mathbb{E}}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left[(\phi(s^{(L)})-\phi(s^{(0)}))^{\top}x_{t}^{k}\phi(s^{(0)})\right] \displaystyle= \displaystyle\underbrace{{\mathbb{E}}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left[\phi(s^{(0)})(\phi(s^{(L)})-\phi(s^{(0)}))^{\top}\right]}_{A_{L}^{k}}x_{t}^{k} \displaystyle= \displaystyle A_{L}^{k}x_{t}^{k}. Then, \displaystyle\delta_{t,L}^{k}\phi(s_{t,0}^{k})=\xi_{t,L}^{k}+Ly_{t}^{k}\phi(s_{t,0}^{k})+A_{L}^{k}x_{t}^{k}, where \displaystyle\xi_{t,L}^{k}= \displaystyle\sum_{\ell=0}^{L-1}\left((r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}_{t}\omega_{t}^{k}\right)\phi(s_{t,0}^{k}) \displaystyle-{\mathbb{E}}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left((r(s^{(\ell)},a^{(\ell)})-J^{k})+(\phi(s^{(\ell+1)})-\phi(s^{(\ell)}))^{\top}\mathbf{B}_{t}\omega_{t}^{k}\right)\phi(s^{(0)}). ∎ Lemma I.1 . Fix \tau\in{\mathbb{N}} . When \frac{\zeta U_{\delta,L}U_{\omega}}{L}\leq 1/2 , for any t\geq\tau>0 and k\in[K] , it holds that \displaystyle\|\omega_{t}^{k}-\omega_{t-\tau}^{k}\|\leq\tau\frac{U_{\delta,L}}{L}\beta,~~\text{and }~\|\mathbf{B}_{t}-\mathbf{B}_{t-\tau}\|\leq 2\tau\frac{U_{\delta,L}}{L}U_{\omega}\zeta+4\tau\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{2}\zeta^{2}. Proof. For any t\geq\tau>0 and k\in[K] , we have \displaystyle\|\omega_{t}^{k}-\omega_{t-\tau}^{k}\|\leq\sum_{i=t-\tau}^{t-1}\|\omega_{i+1}^{k}-\omega_{i}^{k}\| \displaystyle= \displaystyle\sum_{i=t-\tau}^{t-1}\|\Pi_{U_{\omega}}\left(\omega^{k}_{i}+\beta\frac{1}{L}\delta_{i,L}^{k}\mathbf{B}_{i}^{\top}\phi(s_{i,0}^{k})\right)-\Pi_{U_{\omega}}(\omega_{i}^{k})\| \displaystyle\leq \displaystyle\sum_{i=t-\tau}^{t-1}\|\beta\frac{1}{L}\delta_{i,L}^{k}\mathbf{B}_{i}^{\top}\phi(s_{i,0}^{k})\| \displaystyle\leq \displaystyle\tau\frac{U_{\delta,L}}{L}\beta, where the last inequality follows from Lemma H.2 . In addition, \displaystyle\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|\leq\sum_{i=t-\tau}^{t-1}\|{\mathbf{B}}_{i+1}-{\mathbf{B}}_{i}\| \displaystyle= \displaystyle\sum_{i=t-\tau}^{t-1}\|(\mathbf{B}_{i}+\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{i,\perp}{\mathbf{B}}_{i,\perp}^{\top}\delta_{i,L}^{k}\phi(s_{i,0}^{k})(\omega_{i}^{k})^{\top}){\mathbf{R}}_{i+1}^{-1}-{\mathbf{B}}_{i}\| \displaystyle= \displaystyle\sum_{i=t-\tau}^{t-1}\|(\mathbf{B}_{i}+\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{i,\perp}{\mathbf{B}}_{i,\perp}^{\top}\delta_{i,L}^{k}\phi(s_{i,0}^{k})(\omega_{i}^{k})^{\top}){\mathbf{R}}_{i+1}^{-1}-{\mathbf{B}}_{i}{\mathbf{R}}_{i+1}^{-1}+{\mathbf{B}}_{i}{\mathbf{R}}_{i+1}^{-1}-{\mathbf{B}}_{i}\| \displaystyle\leq \displaystyle\sum_{i=t-\tau}^{t-1}\|\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{i,\perp}{\mathbf{B}}_{i,\perp}^{\top}\delta_{i,L}^{k}\phi(s_{i,0}^{k})(\omega_{i}^{k})^{\top}\|\|{\mathbf{R}}_{i+1}^{-1}\|+\sum_{i=t-\tau}^{t-1}\|{\mathbf{R}}_{i+1}^{-1}-{\mathbf{I}}\|\|{\mathbf{B}}_{i}\| \displaystyle\leq \displaystyle\zeta\frac{U_{\delta,L}}{L}U_{\omega}\sum_{i=t-\tau}^{t-1}\|{\mathbf{R}}_{i+1}^{-1}\|+\sum_{i=t-\tau}^{t-1}\|{\mathbf{R}}_{i+1}^{-1}-{\mathbf{I}}\| \displaystyle\leq \displaystyle\zeta\frac{U_{\delta,L}}{L}U_{\omega}\sum_{i=t-\tau}^{t-1}\frac{1}{1-2\|{\mathbf{Q}}_{i}\|_{F}^{2}}+\sum_{i=t-\tau}^{t-1}4\|{\mathbf{Q}}_{i}\|_{F}^{2}, where the last inequality follows from Lemma C.1 , and {\mathbf{Q}}_{i} is defined in Eq. ( 19 ) as \displaystyle{\mathbf{Q}}_{i}=\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{B}}_{i,\perp}{\mathbf{B}}_{i,\perp}^{\top}\delta_{i,L}^{k}\phi(s_{i,0}^{k})(\omega_{i}^{k})^{\top}. We have \displaystyle\|{\mathbf{Q}}_{i}\|_{F} \displaystyle\leq\frac{\zeta}{KL}\sum_{k=1}^{K}\|{\mathbf{P}}_{i,\perp}\delta_{i,L}^{k}\phi(s_{i}^{k})(\omega_{i}^{k})^{\top}\|_{F} \displaystyle\leq\frac{\zeta}{KL}\sum_{k=1}^{K}|\delta_{i,L}^{k}|\|{\mathbf{P}}_{i,\perp}\|\|\phi(s_{i}^{k})\|\|\omega_{i}^{k}\| \displaystyle\leq\zeta\frac{U_{\delta,L}}{L}U_{\omega}. (100) When \zeta\frac{U_{\delta,L}}{L}U_{\omega}\leq 1/2 , 1-2\|{\mathbf{Q}}_{i}\|_{F}^{2}\geq 1/2 . Therefore, we have \displaystyle\|{\mathbf{B}}_{t}-{\mathbf{B}}_{t-\tau}\|\leq \displaystyle 2\tau\zeta\frac{U_{\delta,L}}{L}U_{\omega}+4\tau\zeta^{2}\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{2}. ∎ Lemma I.2 . Fix \tau\in{\mathbb{N}} . For any k\in[K] and any j\geq t-\tau>\tau , where t\geq 2\tau , it holds that \displaystyle{\mathbb{E}}_{t-2\tau}\|\omega_{j+1}^{k}-\omega_{j}^{k}\| \displaystyle\leq C_{1}(\tau)\frac{\beta^{2}}{L}+C_{2}(\tau)\frac{\zeta\beta}{L}+C_{3}(\tau)\gamma\beta+C_{4}\frac{\beta}{\sqrt{L}}+2\frac{\beta}{L}{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|+\beta{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|, \displaystyle{\mathbb{E}}_{t-2\tau}\|\omega_{j+1}^{k}-\omega_{j}^{k}\|^{2} \displaystyle\leq C_{5}(\tau^{2})\frac{\beta^{4}}{L^{2}}+C_{6}(\tau^{2})\frac{\zeta^{2}\beta^{2}}{L^{2}}+8\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|^{2}+C_{7}\frac{\beta^{2}}{L}+C_{8}(\tau^{2})\gamma^{2}\beta^{2}+2\beta^{2}{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|^{2}, where \displaystyle C_{1}(\tau)=2\tau\frac{U_{\delta,L}}{L},C_{2}(\tau)=8\tau\frac{U_{\delta,L}}{L}U_{\omega}^{2},C_{3}(\tau)=2\tau U_{r},C_{4}=\left(U_{\delta}+U_{\delta}\sqrt{\frac{2m\rho}{1-\rho}}\right), \displaystyle C_{5}(\tau^{2})=48\tau^{2}\frac{U_{\delta,L}^{2}}{L^{2}},C_{6}(\tau^{2})=768\tau^{2}U_{\omega}^{3}\frac{U_{\delta,L}^{2}}{L^{2}},C_{7}=3U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho}),C_{8}(\tau^{2})=24\tau^{2}U_{r}^{2}. (101) Proof. For any j , one has \displaystyle\|\omega_{j+1}^{k}-\omega_{j}^{k}\|=\|\Pi_{U_{\omega}}(\omega_{j}^{k}+\frac{\beta}{L}\delta_{j,L}^{k}{\mathbf{B}}_{j}^{\top}\phi(s_{j,0}^{k}))-\Pi_{U_{\omega}}(\omega_{j}^{k})\| \displaystyle\leq \displaystyle\|\frac{\beta}{L}{\mathbf{B}}_{j}^{\top}\delta_{j,L}^{k}\phi(s_{j,0}^{k})\|\qquad\qquad\text{(by non-expansive property of projection)} \displaystyle= \displaystyle\frac{\beta}{L}\|{\mathbf{B}}_{j}^{\top}\left(\widetilde{A}_{j,L}^{k}x_{j}^{k}+{\mathbf{b}}_{j,L}^{k}+Ly_{j}^{k}\phi(s_{j,0}^{k})\right)\|~~~~~~\text{(by Proposition \ref{prop: key intermediate})} \displaystyle\leq \displaystyle\frac{\beta}{L}\|{\mathbf{B}}_{j}^{\top}{\widetilde{A}_{j,L}^{k}}x_{j}^{k}\|+\frac{\beta}{L}\|{\mathbf{B}}_{j}^{\top}{\mathbf{b}}_{j,L}^{k}\|+\frac{\beta}{L}\|{\mathbf{B}}_{j}^{\top}Ly_{j}^{k}\phi(s_{j,0}^{k})\| \displaystyle\leq \displaystyle{\frac{2\beta}{L}}\|x_{j}^{k}\|+\frac{\beta}{L}\|{\mathbf{b}}_{j,L}^{k}\|+\beta|y_{j}^{k}|. (102) By Lemma I.5 , and by the tower property and the notation defined in Section B , \displaystyle{\mathbb{E}}_{t-2\tau}\|{\mathbf{b}}_{j,L}^{k}\| \displaystyle\leq\left(U_{\delta}+U_{\delta}\sqrt{\frac{2m\rho}{1-\rho}}\right)\sqrt{L}, where U_{\delta}=2U_{r}+2U_{\omega} as defined in Lemma I.5 . Then, taking expectation up to t-2\tau , Eq. ( I ) can be written as, \displaystyle{\mathbb{E}}_{t-2\tau}\|\omega_{j+1}^{k}-\omega_{j}^{k}\|\leq{\frac{2\beta}{L}}{\mathbb{E}}_{t-2\tau}\|x_{j}^{k}\|+\frac{\beta}{\sqrt{L}}\left(U_{\delta}+U_{\delta}\sqrt{\frac{2m\rho}{1-\rho}}\right)+\beta{\mathbb{E}}_{t-2\tau}|y_{j}^{k}|. (103) By Lemma I.4 , Eq. ( 103 ) can be written as, \displaystyle{\mathbb{E}}_{t-2\tau}\|\omega_{j+1}^{k}-\omega_{j}^{k}\|\leq 2\frac{\beta}{L}{\mathbb{E}}_{t-2\tau}\|x_{j}^{k}\|+\frac{\beta}{\sqrt{L}}\left(U_{\delta}+U_{\delta}\sqrt{\frac{2m\rho}{1-\rho}}\right)+\beta{\mathbb{E}}_{t-2\tau}|y_{j}^{k}| \displaystyle\leq \displaystyle 2\frac{\beta}{L}\left(\tau\frac{U_{\delta,L}}{L}\beta+4\tau\frac{U_{\delta,L}}{L}U_{\omega}^{2}\zeta+{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|\right)+\frac{\beta}{\sqrt{L}}\left(U_{\delta}+U_{\delta}\sqrt{\frac{2m\rho}{1-\rho}}\right) \displaystyle+\beta\left(2\tau U_{r}\gamma+{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|\right) \displaystyle= \displaystyle\underbrace{2\tau\frac{U_{\delta,L}}{L}}_{:=C_{1}(\tau)}{\frac{\beta^{2}}{L}}+\underbrace{8\tau\frac{U_{\delta,L}}{L}U_{\omega}^{2}}_{:=C_{2}(\tau)}\frac{\zeta\beta}{L}+\underbrace{2\tau U_{r}}_{:=C_{3}(\tau)}\gamma\beta+\frac{\beta}{\sqrt{L}}\underbrace{\left(U_{\delta}+U_{\delta}\sqrt{\frac{2m\rho}{1-\rho}}\right)}_{:=C_{4}}+2\frac{\beta}{L}{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|+\beta{\mathbb{E}}_{t-2\tau}|y_{t}^{k}| \displaystyle\leq \displaystyle C_{1}(\tau)\frac{\beta^{2}}{L}+C_{2}(\tau)\frac{\zeta\beta}{L}+C_{3}(\tau)\gamma\beta+C_{4}\frac{\beta}{\sqrt{L}}+2\frac{\beta}{L}{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|+\beta{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|. Similarly, \displaystyle{\mathbb{E}}_{t-2\tau}\|\omega_{j+1}^{k}-\omega_{j}^{k}\|^{2}={\mathbb{E}}_{t-2\tau}\|\Pi_{U_{\omega}}(\omega_{j}^{k}+\frac{\beta}{L}\delta_{j,L}^{k}{\mathbf{B}}_{j}^{\top}\phi(s_{j,0}^{k}))-\Pi_{U_{\omega}}(\omega_{j}^{k})\|^{2} \displaystyle\leq \displaystyle{\mathbb{E}}_{t-2\tau}\|\frac{\beta}{L}{\mathbf{B}}_{j}^{\top}\delta_{j,L}^{k}\phi(s_{j,0}^{k})\|^{2} \displaystyle= \displaystyle\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-2\tau}\|{\mathbf{B}}_{j}^{\top}\left(\widetilde{A}_{j,L}^{k}x_{j}^{k}+{\mathbf{b}}_{j,L}^{k}+Ly_{j}^{k}\phi(s_{j,0}^{k})\right)\|^{2}\quad\quad~~~~~~\text{(by Proposition \ref{prop: key intermediate})} \displaystyle\leq \displaystyle 3\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-2\tau}\|{\mathbf{B}}_{j}^{\top}{\widetilde{A}_{j,L}^{k}}x_{j}^{k}\|^{2}+3\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-2\tau}\|{\mathbf{B}}_{j}^{\top}{\mathbf{b}}_{j,L}^{k}\|^{2}+3\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-2\tau}\|{\mathbf{B}}_{j}^{\top}Ly_{j}^{k}\phi(s_{j,0}^{k})\|^{2} \displaystyle\leq \displaystyle{\frac{12\beta^{2}}{L^{2}}}{\mathbb{E}}_{t-2\tau}\|x_{j}^{k}\|^{2}+3\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-2\tau}\|{\mathbf{b}}_{j,L}^{k}\|^{2}+3\beta^{2}{\mathbb{E}}_{t-2\tau}|y_{j}^{k}|^{2}. (104) By Lemma I.5 and the tower property, we get, \displaystyle{\mathbb{E}}_{t-2\tau}\|{\mathbf{b}}_{t,L}^{k}\|^{2}\leq U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})L. By Lemma I.4 , Eq. ( I ) can be written as, \displaystyle{\mathbb{E}}_{t-2\tau}\|\omega_{j+1}^{k}-\omega_{j}^{k}\|^{2}\leq 3\left({\frac{4\beta^{2}}{L^{2}}}{\mathbb{E}}_{t-2\tau}\|x_{j}^{k}\|^{2}+U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})\frac{\beta^{2}}{L}+\beta^{2}{\mathbb{E}}_{t-2\tau}|y_{j}^{k}|^{2}\right) \displaystyle\leq \displaystyle 48\tau^{2}\frac{U_{\delta,L}^{2}}{L^{4}}\beta^{4}+768\tau^{2}U_{\omega}^{3}\frac{U_{\delta,L}^{2}}{L^{4}}\zeta^{2}\beta^{2}+24\frac{\beta^{2}}{L^{2}}{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|^{2} \displaystyle+3U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})\frac{\beta^{2}}{L}+24\tau^{2}U_{r}^{2}\gamma^{2}\beta^{2}+6\beta^{2}{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|^{2}. ∎ Lemma I.3 . Fix \tau>0 . Choosing \zeta so that \zeta\frac{U_{\delta,L}}{L}U_{\omega}\leq 1/2 and U_{\omega}\zeta\leq 1/2 , for any j\geq t-\tau\geq\tau , we have \displaystyle{\mathbb{E}}_{t-2\tau}[\|\mathbf{B}_{j+1}-\mathbf{B}_{j}\|]\leq \displaystyle{C_{9}(\tau)}\frac{\beta\zeta}{L}+{C_{10}(\tau)}\frac{\zeta^{2}}{L}+4U_{\omega}\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|+{C_{11}(\tau)}\zeta\beta+{C_{12}}\frac{\zeta}{\sqrt{K}} \displaystyle+{C_{13}}\zeta m\rho^{\tau}+{C_{14}(\tau)}\gamma\zeta+2U_{\omega}\frac{\zeta}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|+{C_{15}}\zeta^{2}, \displaystyle{\mathbb{E}}_{t-2\tau}\|\mathbf{B}_{j+1}-\mathbf{B}_{j}\|^{2}\leq \displaystyle C_{16}(\tau^{2})\frac{\zeta^{2}\beta^{2}}{L^{2}}+C_{17}(\tau^{2})\frac{\zeta^{4}}{L^{2}}+C_{18}\frac{\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|^{2}+C_{19}(\tau^{2})\zeta^{2}\beta^{2}+C_{20}\frac{\zeta^{2}}{K} \displaystyle+C_{21}\zeta^{2}m^{2}\rho^{2\tau}+C_{22}(\tau^{2})\gamma^{2}\zeta^{2}+C_{23}\frac{\zeta^{2}}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|^{2}+C_{24}\zeta^{4}, where \displaystyle C_{9}(\tau)=4U_{\omega}\tau\frac{U_{\delta,L}}{L},C_{10}(\tau)=16\tau\frac{U_{\delta,L}}{L}U_{\omega}^{3},C_{11}(\tau)=2\frac{U_{\delta,L}^{2}}{L^{2}}\tau,C_{12}=2U_{\omega}\frac{U_{\delta,L}}{L}, \displaystyle C_{13}=4\frac{U_{\delta,L}}{L}U_{\omega},C_{14}(\tau)=4U_{\omega}\tau U_{r},C_{15}=4\frac{U_{\delta,L}^{2}}{L}U_{\omega}^{2}, \displaystyle C_{16}(\tau^{2})=384\tau^{2}U_{\omega}^{2}\frac{U_{\delta,L}^{2}}{L^{2}},C_{17}(\tau^{2})=6144\tau^{2}U_{\omega}^{5}\frac{U_{\delta,L}^{2}}{L^{2}},C_{18}=192U_{\omega}^{2},C_{19}=\frac{48U_{\delta,L}^{4}}{L^{4}}\tau^{2},C_{20}=\frac{48U_{\omega}^{2}U_{\delta,L}^{2}}{L^{2}}, \displaystyle C_{21}=\frac{48U_{\omega}^{2}U_{\delta,L}^{2}}{L^{2}},C_{22}(\tau^{2})=192U_{\omega}^{2}\tau^{2}U_{r}^{2},C_{23}=48U_{\omega}^{2},C_{24}=32\frac{U_{\delta,L}^{4}}{L^{4}}U_{\omega}^{4}. Proof. Fix \tau>0 . For any j\geq t-\tau , the one-step perturbation of {\mathbf{B}}_{j} can be bounded as \displaystyle\|{\mathbf{B}}_{j+1}-{\mathbf{B}}_{j}\| \displaystyle= \displaystyle\|(\mathbf{B}_{j}+\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{P}}_{j,\perp}\delta_{j,L}^{k}\phi(s_{j,0}^{k})(\omega_{j}^{k})^{\top}){\mathbf{R}}_{j+1}^{-1}-{\mathbf{B}}_{j}\| \displaystyle= \displaystyle\|(\mathbf{B}_{j}+\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{P}}_{j,\perp}\delta_{j,L}^{k}\phi(s_{j,0}^{k})(\omega_{j}^{k})^{\top}){\mathbf{R}}_{j+1}^{-1}-{\mathbf{B}}_{j}{\mathbf{R}}_{j+1}^{-1}+{\mathbf{B}}_{j}{\mathbf{R}}_{j+1}^{-1}-{\mathbf{B}}_{j}\| \displaystyle\leq \displaystyle\left\|\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{P}}_{j,\perp}\delta_{j,L}^{k}\phi(s_{j,0}^{k})(\omega_{j}^{k})^{\top}\right\|\left\|{\mathbf{R}}_{j+1}^{-1}\right\|+\|{\mathbf{R}}_{j+1}^{-1}-{\mathbf{I}}\|\|{\mathbf{B}}_{j}\| \displaystyle\leq \displaystyle\left\|\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{P}}_{j,\perp}\delta_{j,L}^{k}\phi(s_{j,0}^{k})(\omega_{j}^{k})^{\top}\right\|\frac{1}{1-2\|{\mathbf{Q}}_{j}\|_{F}^{2}}+4\|{\mathbf{Q}}_{j}\|_{F}^{2}, (105) where the last inequality follows from Lemma C.1 and the fact that \|{\mathbf{B}}_{j}\|\leq 1 . From Eq. ( 19 ), we know \|{\mathbf{Q}}_{j}\|_{F}\leq\frac{U_{\delta,L}}{L}U_{\omega}\zeta. When \frac{U_{\delta,L}}{L}U_{\omega}\zeta\leq 1/2 , it holds that \frac{1}{1-2\|{\mathbf{Q}}_{j}\|_{F}^{2}}\leq 2 . Thus, it remains to bound \left\|\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{P}}_{j,\perp}\delta_{j,L}^{k}\phi(s_{j,0}^{k})(\omega_{j}^{k})^{\top}\right\| , for which we have \displaystyle\left\|\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{P}}_{j,\perp}\delta_{j,L}^{k}\phi(s_{j,0}^{k})(\omega_{j}^{k})^{\top}\right\| \displaystyle= \displaystyle\frac{\zeta}{KL}\left\|\sum_{k=1}^{K}{\mathbf{P}}_{j,\perp}\delta_{j,L}^{k}\phi(s_{j,0}^{k})(\omega_{j}^{k})^{\top}\right\| \displaystyle\leq \displaystyle\frac{\zeta}{KL}\left\|\sum_{k=1}^{K}\delta_{j,L}^{k}\phi(s_{j,0}^{k})(\omega_{j}^{k})^{\top}\right\| \displaystyle\leq \displaystyle\frac{\zeta}{KL}\left\|\sum_{k=1}^{K}\left({\widetilde{A}_{j,L}^{k}}x_{j}^{k}+{\mathbf{b}}_{j,L}^{k}+Ly_{j}^{k}\phi(s_{j,0}^{k})\right)(\omega_{j}^{k})^{\top}\right\| \displaystyle\leq \displaystyle\frac{\zeta}{KL}\left\|\sum_{k=1}^{K}{\widetilde{A}_{j,L}^{k}}x_{j}^{k}(\omega_{j}^{k})^{\top}\right\|+\frac{\zeta}{KL}\left\|\sum_{k=1}^{K}{\mathbf{b}}_{j,L}^{k}(\omega_{j}^{k})^{\top}\right\|+\frac{\zeta}{KL}\left\|\sum_{k=1}^{K}Ly_{j}^{k}\phi(s_{j,0}^{k})(\omega_{j}^{k})^{\top}\right\| \displaystyle\leq \displaystyle 2U_{\omega}\frac{\zeta}{KL}\sum_{k=1}^{K}\|x_{j}^{k}\|+\underbrace{\frac{\zeta}{KL}\left\|\sum_{k=1}^{K}{\mathbf{b}}_{j,L}^{k}(\omega_{j}^{k}-\omega_{j-\tau}^{k})^{\top}\right\|}_{I_{1}}+\underbrace{\frac{\zeta}{KL}\left\|\sum_{k=1}^{K}{\mathbf{b}}_{j,L}^{k}(\omega_{j-\tau}^{k})^{\top}\right\|}_{I_{2}}+U_{\omega}\frac{\zeta}{K}\sum_{k=1}^{K}|y_{j}^{k}|. (106) For I_{1} in Eq. ( I ), \displaystyle\frac{\zeta}{KL}\left\|\sum_{k=1}^{K}{\mathbf{b}}_{j,L}^{k}(\omega_{j}^{k}-\omega_{j-\tau}^{k})^{\top}\right\|\leq\frac{U_{\delta,L}^{2}}{L^{2}}\tau\zeta\beta.~~~~~(\text{ by Lemma \ref{lmm: crude parameter bound}}) (107) For I_{2} in Eq. ( I ), taking conditional expectation with respect to the Markovian trajectories up to time j-\tau , we have \displaystyle\frac{\zeta}{KL}{\mathbb{E}}_{j-\tau}\left\|\sum_{k=1}^{K}{\mathbf{b}}_{j,L}^{k}(\omega_{j-\tau}^{k})^{\top}\right\| \displaystyle\leq \displaystyle\frac{\zeta}{KL}\left({\mathbb{E}}_{j-\tau}\left\|\sum_{k=1}^{K}{\mathbf{b}}_{j,L}^{k}(\omega_{j-\tau}^{k})^{\top}\right\|^{2}\right)^{\frac{1}{2}}~~~~(\text{by Jensen's inequality for expectation}) \displaystyle= \displaystyle\frac{\zeta}{KL}\left({\mathbb{E}}_{j-\tau}\sum_{k=1}^{K}\left\|{\mathbf{b}}_{j,L}^{k}(\omega_{j-\tau}^{k})^{\top}\right\|^{2}+2\sum_{k=1}^{K}\sum_{k^{\prime}=k+1}^{K}{\mathbb{E}}_{j-\tau}\left\langle{\mathbf{b}}_{j,L}^{k}(\omega_{j-\tau}^{k})^{\top},{\mathbf{b}}_{j,L}^{k^{\prime}}(\omega_{j-\tau}^{k^{\prime}})^{\top}\right\rangle\right)^{\frac{1}{2}} \displaystyle\overset{(a)}{\leq} \displaystyle\frac{\zeta}{KL}\left(KU_{\omega}^{2}U_{\delta,L}^{2}+2\sum_{k=1}^{K}\sum_{k^{\prime}=k+1}^{K}{\mathbb{E}}_{j-\tau}\left[{\mathbf{b}}_{j,L}^{k}(\omega_{j-\tau}^{k})^{\top}\right]{\mathbb{E}}_{j-\tau}\left[{\mathbf{b}}_{j,L}^{k^{\prime}}(\omega_{j-\tau}^{k^{\prime}})^{\top}\right]\right)^{\frac{1}{2}} \displaystyle\leq \displaystyle\frac{\zeta}{KL}\left(KU_{\omega}^{2}U_{\delta,L}^{2}+K^{2}U_{\delta,L}^{2}U_{\omega}^{2}m^{2}\rho^{2\tau}\right)^{\frac{1}{2}} \displaystyle\leq \displaystyle U_{\omega}\frac{U_{\delta,L}}{L}\frac{\zeta}{\sqrt{K}}+\frac{U_{\delta,L}}{L}U_{\omega}\zeta m\rho^{\tau}. (108) where inequality (a) follows from Lemma H.2 and conditional independence. Plugging Eq. ( 107 ) and Eq. ( I ) back to Eq. ( I ) and Eq. ( I ), we get, under expectation, \displaystyle{\mathbb{E}}_{j-\tau}[\|\mathbf{B}_{j+1}-\mathbf{B}_{j}\|] \displaystyle={\mathbb{E}}[\|\mathbf{B}_{j+1}-\mathbf{B}_{j}\|\mid v_{0:j-\tau}] \displaystyle\leq 4U_{\omega}\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbb{E}}_{j-\tau}\|x_{j}^{k}\|+2\frac{U_{\delta,L}^{2}}{L^{2}}\tau\zeta\beta+2U_{\omega}\frac{U_{\delta,L}}{L}\frac{\zeta}{\sqrt{K}} \displaystyle\quad+4\frac{U_{\delta,L}}{L}U_{\omega}\zeta m\rho^{\tau}+2U_{\omega}\frac{\zeta}{K}\sum_{k=1}^{K}{\mathbb{E}}_{j-\tau}|y_{j}^{k}|+4\frac{U_{\delta,L}^{2}}{L}U_{\omega}^{2}\zeta^{2}. (109) Finally, \displaystyle{\mathbb{E}}_{t-2\tau}[\|\mathbf{B}_{j+1}-\mathbf{B}_{j}\|]={\mathbb{E}}[\|\mathbf{B}_{j+1}-\mathbf{B}_{j}\|\mid v_{0:t-2\tau}] \displaystyle= \displaystyle{\mathbb{E}}[~{\mathbb{E}}[\|\mathbf{B}_{j+1}-\mathbf{B}_{j}\|\mid v_{0:j-\tau}]\mid v_{0:t-2\tau}]\quad\quad~~~~~\text{(by the tower property of conditional expecation)} \displaystyle\leq \displaystyle 4U_{\omega}\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}[{\mathbb{E}}_{j-\tau}\|x_{j}^{k}\|]+2\frac{U_{\delta,L}^{2}}{L^{2}}\tau\zeta\beta+2U_{\omega}\frac{U_{\delta,L}}{L}\frac{\zeta}{\sqrt{K}}\quad~~~~~\text{(by applying Eq.\,(\ref{eqn: one step B before substituting jth}))} \displaystyle+4\frac{U_{\delta,L}}{L}U_{\omega}\zeta m\rho^{\tau}+2U_{\omega}\frac{\zeta}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}[{\mathbb{E}}_{j-\tau}|y_{j}^{k}|]+4\frac{U_{\delta,L}^{2}}{L}U_{\omega}^{2}\zeta^{2} \displaystyle= \displaystyle 4U_{\omega}\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}[\|x_{j}^{k}\|]+2\frac{U_{\delta,L}^{2}}{L^{2}}\tau\zeta\beta+2U_{\omega}\frac{U_{\delta,L}}{L}\frac{\zeta}{\sqrt{K}} \displaystyle+4\frac{U_{\delta,L}}{L}U_{\omega}\zeta m\rho^{\tau}+2U_{\omega}\frac{\zeta}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}[|y_{j}^{k}|]+4\frac{U_{\delta,L}^{2}}{L}U_{\omega}^{2}\zeta^{2} \displaystyle\leq \displaystyle 4U_{\omega}\frac{\zeta}{L}\left(\tau\frac{U_{\delta,L}}{L}\beta+4\tau\frac{U_{\delta,L}}{L}U_{\omega}^{2}\zeta+\frac{1}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|\right)+2\frac{U_{\delta,L}^{2}}{L^{2}}\tau\zeta\beta+2U_{\omega}\frac{U_{\delta,L}}{L}\frac{\zeta}{\sqrt{K}} \displaystyle+4\frac{U_{\delta,L}}{L}U_{\omega}\zeta m\rho^{\tau}+2U_{\omega}\frac{\zeta}{K}\sum_{k=1}^{K}\left(2\tau U_{r}\gamma+{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|\right)+4\frac{U_{\delta,L}^{2}}{L}U_{\omega}^{2}\zeta^{2}\quad\quad~~~~~\text{(applying Lemma \ref{lmm: jth to tth error bound})} \displaystyle= \displaystyle\underbrace{4U_{\omega}\tau\frac{U_{\delta,L}}{L}}_{:=C_{9}(\tau)}\frac{\beta\zeta}{L}+\underbrace{16\tau\frac{U_{\delta,L}}{L}U_{\omega}^{3}}_{:=C_{10}(\tau)}\frac{\zeta^{2}}{L}+4U_{\omega}\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|+\underbrace{2\frac{U_{\delta,L}^{2}}{L^{2}}\tau}_{:=C_{11}(\tau)}\zeta\beta+\underbrace{2U_{\omega}\frac{U_{\delta,L}}{L}}_{:=C_{12}}\frac{\zeta}{\sqrt{K}} \displaystyle+\underbrace{4\frac{U_{\delta,L}}{L}U_{\omega}}_{:=C_{13}}\zeta m\rho^{\tau}+\underbrace{4U_{\omega}\tau U_{r}}_{:=C_{14}(\tau)}\gamma\zeta+2U_{\omega}\frac{\zeta}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|+\underbrace{4\frac{U_{\delta,L}^{2}}{L}U_{\omega}^{2}}_{:=C_{15}}\zeta^{2} \displaystyle= \displaystyle{C_{9}(\tau)}\frac{\beta\zeta}{L}+{C_{10}(\tau)}\frac{\zeta^{2}}{L}+4U_{\omega}\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|+{C_{11}(\tau)}\zeta\beta+{C_{12}}\frac{\zeta}{\sqrt{K}} \displaystyle+{C_{13}}\zeta m\rho^{\tau}+{C_{14}(\tau)}\gamma\zeta+2U_{\omega}\frac{\zeta}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|+{C_{15}}\zeta^{2}. Analogously, \displaystyle\|{\mathbf{B}}_{j+1}-{\mathbf{B}}_{j}\|^{2} \displaystyle\leq \displaystyle 2\left\|\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{P}}_{j,\perp}\delta_{j,L}^{k}\phi(s_{j,0}^{k})(\omega_{j}^{k})^{\top}\right\|^{2}\left(\frac{1}{1-2\|{\mathbf{Q}}_{j}\|_{F}^{2}}\right)^{2}+32\|{\mathbf{Q}}_{j}\|_{F}^{4} \displaystyle\leq \displaystyle 8\left\|\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{P}}_{j,\perp}\delta_{j,L}^{k}\phi(s_{j,0}^{k})(\omega_{j}^{k})^{\top}\right\|^{2}+32\|{\mathbf{Q}}_{j}\|_{F}^{4}. (110) Similar to Eq. ( I ), the first term can be decomposed and bounded as, \displaystyle 8\left\|\frac{\zeta}{KL}\sum_{k=1}^{K}{\mathbf{P}}_{j,\perp}\delta_{j,L}^{k}\phi(s_{j,0}^{k})(\omega_{j}^{k})^{\top}\right\|^{2} \displaystyle\leq \displaystyle\frac{24\zeta^{2}}{K^{2}L^{2}}\left\|\sum_{k=1}^{K}{\widetilde{A}_{j,L}^{k}}x_{j}^{k}(\omega_{j}^{k})^{\top}\right\|^{2}+\frac{24\zeta^{2}}{K^{2}L^{2}}\left\|\sum_{k=1}^{K}{\mathbf{b}}_{j,L}^{k}(\omega_{j}^{k})^{\top}\right\|^{2}+\frac{24\zeta^{2}}{K^{2}L^{2}}\left\|\sum_{k=1}^{K}Ly_{j}^{k}\phi(s_{j,0}^{k})(\omega_{j}^{k})^{\top}\right\|^{2} \displaystyle\leq \displaystyle 96U_{\omega}^{2}\frac{\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}\|x_{j}^{k}\|^{2}+\underbrace{\frac{48\zeta^{2}}{K^{2}L^{2}}\left\|\sum_{k=1}^{K}{\mathbf{b}}_{j,L}^{k}(\omega_{j}^{k}-\omega_{j-\tau}^{k})^{\top}\right\|^{2}}_{I_{3}}+\underbrace{\frac{48\zeta^{2}}{K^{2}L^{2}}\left\|\sum_{k=1}^{K}{\mathbf{b}}_{j,L}^{k}(\omega_{j-\tau}^{k})^{\top}\right\|^{2}}_{I_{4}}+U_{\omega}^{2}\frac{24\zeta^{2}}{K}\sum_{k=1}^{K}|y_{j}^{k}|^{2}. (111) For I_{3} in Eq. ( I ), by Lemma H.2 and Lemma I.1 , we have \displaystyle\frac{48\zeta^{2}}{K^{2}L^{2}}\left\|\sum_{k=1}^{K}{\mathbf{b}}_{j,L}^{k}(\omega_{j}^{k}-\omega_{j-\tau}^{k})^{\top}\right\|^{2}\leq\frac{48U_{\delta,L}^{4}}{L^{4}}\tau^{2}\zeta^{2}\beta^{2}. (112) For I_{4} in Eq. ( I ), we take the conditional expectation with respect to the Markovian trajectories up to time j-\tau . Similar to Eq. ( I ), we obtain \displaystyle\frac{48\zeta^{2}}{K^{2}L^{2}}{\mathbb{E}}_{j-\tau}\left\|\sum_{k=1}^{K}{\mathbf{b}}_{j,L}^{k}(\omega_{j-\tau}^{k})^{\top}\right\|^{2} \displaystyle\leq\frac{48\zeta^{2}}{K^{2}L^{2}}\left(KU_{\omega}^{2}U_{\delta,L}^{2}+K^{2}U_{\delta,L}^{2}U_{\omega}^{2}m^{2}\rho^{2\tau}\right) \displaystyle\leq\frac{48U_{\omega}^{2}U_{\delta,L}^{2}}{L^{2}}\frac{\zeta^{2}}{K}+\frac{48U_{\omega}^{2}U_{\delta,L}^{2}}{L^{2}}\zeta^{2}m^{2}\rho^{2\tau}. (113) Plugging Eq. ( 112 ) and Eq. ( I ) back to Eq. ( I ) and Eq. ( I ), and applying the tower property, we get \displaystyle{\mathbb{E}}_{t-2\tau}\|{\mathbf{B}}_{j+1}-{\mathbf{B}}_{j}\|^{2}={\mathbb{E}}_{t-2\tau}\left[{\mathbb{E}}_{j-\tau}\|{\mathbf{B}}_{j+1}-{\mathbf{B}}_{j}\|^{2}\right] \displaystyle\leq \displaystyle 96U_{\omega}^{2}\frac{\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}\|x_{j}^{k}\|^{2}+\frac{48U_{\delta,L}^{4}}{L^{4}}\tau^{2}\zeta^{2}\beta^{2}+\frac{48U_{\omega}^{2}U_{\delta,L}^{2}}{L^{2}}\frac{\zeta^{2}}{K} \displaystyle+\frac{48U_{\omega}^{2}U_{\delta,L}^{2}}{L^{2}}\zeta^{2}m^{2}\rho^{2\tau}+U_{\omega}^{2}\frac{24\zeta^{2}}{K^{2}}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}|y_{j}^{k}|^{2}+32(\frac{U_{\delta,L}}{L}U_{\omega}\zeta)^{4} \displaystyle\leq \displaystyle 96U_{\omega}^{2}\frac{\zeta^{2}}{L^{2}}\left(4\tau^{2}\frac{U_{\delta,L}^{2}}{L^{2}}\beta^{2}+64\tau^{2}U_{\omega}^{3}\frac{U_{\delta,L}^{2}}{L^{2}}\zeta^{2}+2\frac{1}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|^{2}\right)+\frac{48U_{\delta,L}^{4}}{L^{4}}\tau^{2}\zeta^{2}\beta^{2}+\frac{48U_{\omega}^{2}U_{\delta,L}^{2}}{L^{2}}\frac{\zeta^{2}}{K} \displaystyle+\frac{48U_{\omega}^{2}U_{\delta,L}^{2}}{L^{2}}\zeta^{2}m^{2}\rho^{2\tau}+U_{\omega}^{2}\frac{24\zeta^{2}}{K^{2}}\sum_{k=1}^{K}\left(8\tau^{2}U_{r}^{2}\gamma^{2}+2{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|^{2}\right)+32(\frac{U_{\delta,L}}{L}U_{\omega}\zeta)^{4}~~~~~\text{(applying Lemma \ref{lmm: jth to tth error bound})} \displaystyle\leq \displaystyle 384\tau^{2}U_{\omega}^{2}\frac{U_{\delta,L}^{2}}{L^{2}}\frac{\zeta^{2}\beta^{2}}{L^{2}}+6144\tau^{2}U_{\omega}^{5}\frac{U_{\delta,L}^{2}}{L^{2}}\frac{\zeta 4}{L^{2}}+192U_{\omega}^{2}\frac{\zeta^{2}}{KL^{2}}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|^{2}+\frac{48U_{\delta,L}^{4}}{L^{4}}\tau^{2}\zeta^{2}\beta^{2}+\frac{48U_{\omega}^{2}U_{\delta,L}^{2}}{L^{2}}\frac{\zeta^{2}}{K} \displaystyle+\frac{48U_{\omega}^{2}U_{\delta,L}^{2}}{L^{2}}\zeta^{2}m^{2}\rho^{2\tau}+192U_{\omega}^{2}\tau^{2}U_{r}^{2}\gamma^{2}\zeta^{2}+48U_{\omega}^{2}\frac{\zeta^{2}}{K}\sum_{k=1}^{K}{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|^{2}+32\frac{U_{\delta,L}^{4}}{L^{4}}U_{\omega}^{4}\zeta^{4}. ∎ Lemma I.4 . For 0\leq t-\tau\leq j\leq t-1 , we have for any k\in[K] , \displaystyle{\mathbb{E}}_{t-2\tau}\|x_{j}^{k}\|\leq\tau\frac{U_{\delta,L}}{L}\beta+4\tau\frac{U_{\delta,L}}{L}U_{\omega}^{2}\zeta+{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|, \displaystyle{\mathbb{E}}_{t-2\tau}\|x_{j}^{k}\|^{2}\leq 4\tau^{2}\frac{U_{\delta,L}^{2}}{L^{2}}\beta^{2}+64\tau^{2}U_{\omega}^{3}\frac{U_{\delta,L}^{2}}{L^{2}}\zeta^{2}+2{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|^{2}, \displaystyle{\mathbb{E}}_{t-2\tau}|y_{j}^{k}|\leq 2\tau U_{r}\gamma+{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|, \displaystyle{\mathbb{E}}_{t-2\tau}|y_{j}^{k}|^{2}\leq 8\tau^{2}U_{r}^{2}\gamma^{2}+2{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|^{2}, when 4\tau\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{3}\zeta\leq 2\tau\frac{U_{\delta,L}}{L}U_{\omega}^{2} , and 128\tau^{2}U_{\omega}^{5}\frac{U_{\delta,L}^{2}}{L^{2}}\zeta^{2}\leq 32\tau^{2}U_{\omega}^{3}\frac{U_{\delta,L}^{2}}{L^{2}} . Lemma I.4 aligns the error terms indexed by j to a common reference time t , which is required for a consistent Lyapunov analysis. The bounds follow from the triangle inequality applied along the trajectory from j to t ; since j\geq t-\tau , the difference between x_{j}^{k} (resp., y_{j}^{k} ) and x_{t}^{k} (resp., y_{t}^{k} ) accumulates at most \tau one-step perturbations. Each perturbation is upper bounded using the update dynamics, yielding deviations that scale linearly in \tau , while {\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\| and {\mathbb{E}}_{t-2\tau}|y_{t}^{k}| serve as anchored reference terms. Proof. For any t-1\geq j\geq t-\tau , we have \displaystyle\|x_{j}^{k}\|^{2}\leq 2\|x_{j}^{k}-x_{t}^{k}\|^{2}+2\|x_{t}^{k}\|^{2} \displaystyle= \displaystyle 2\|{\mathbf{B}}_{t}\omega_{t}^{k}-{\mathbf{B}}_{j}\omega_{j}^{k}\|^{2}+2\|x_{t}^{k}\|^{2} \displaystyle\leq \displaystyle 4\|{\mathbf{B}}_{t}\omega_{t}^{k}-{\mathbf{B}}_{t}\omega_{j}^{k}\|^{2}+4\|{\mathbf{B}}_{t}\omega_{j}^{k}-{\mathbf{B}}_{j}\omega_{j}^{k}\|^{2}+2\|x_{t}^{k}\|^{2} \displaystyle\leq \displaystyle 4\|\omega_{t}^{k}-\omega_{j}^{k}\|^{2}+4U_{\omega}\|{\mathbf{B}}_{t}-{\mathbf{B}}_{j}\|^{2}+2\|x_{t}^{k}\|^{2} \displaystyle\leq \displaystyle 4\tau^{2}\frac{U_{\delta,L}^{2}}{L^{2}}\beta^{2}+4\tau^{2}U_{\omega}(2\frac{U_{\delta,L}}{L}U_{\omega}\zeta+4\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{2}\zeta^{2})^{2}+2\|x_{t}^{k}\|^{2}~~~~~(\text{by Lemma \ref{lmm: crude parameter bound}}). Then, taking expectation, we get \displaystyle{\mathbb{E}}_{t-2\tau}\|x_{j}^{k}\|^{2}\leq 4\tau^{2}\frac{U_{\delta,L}^{2}}{L^{2}}\beta^{2}+32\tau^{2}U_{\omega}^{3}\frac{U_{\delta,L}^{2}}{L^{2}}\zeta^{2}+128\tau^{2}U_{\omega}^{5}\frac{U_{\delta,L}^{2}}{L^{2}}\zeta^{4}+2{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|^{2} \displaystyle\leq \displaystyle 4\tau^{2}\frac{U_{\delta,L}^{2}}{L^{2}}\beta^{2}+64\tau^{2}U_{\omega}^{3}\frac{U_{\delta,L}^{2}}{L^{2}}\zeta^{2}+2{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|^{2}. The last inequality holds since 128\tau^{2}U_{\omega}^{5}\frac{U_{\delta,L}^{2}}{L^{2}}\zeta^{2}\leq 32\tau^{2}U_{\omega}^{3}\frac{U_{\delta,L}^{2}}{L^{2}} . Analogously, we get, \displaystyle{\mathbb{E}}_{t-2\tau}\|x_{j}^{k}\|\leq\tau\frac{U_{\delta,L}}{L}\beta+2\tau\frac{U_{\delta,L}}{L}U_{\omega}^{2}\zeta+4\tau\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{3}\zeta^{2}+{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\| \displaystyle\leq \displaystyle\tau\frac{U_{\delta,L}}{L}\beta+4\tau\frac{U_{\delta,L}}{L}U_{\omega}^{2}\zeta+{\mathbb{E}}_{t-2\tau}\|x_{t}^{k}\|. The last inequality holds since 4\tau\frac{U_{\delta,L}^{2}}{L^{2}}U_{\omega}^{3}\zeta\leq 2\tau\frac{U_{\delta,L}}{L}U_{\omega}^{2} . Similarly, for |y_{j}^{k}| , we have, \displaystyle|y_{j}^{k}|^{2}\leq 2|y_{j}^{k}-y_{t}^{k}|^{2}+2|y_{t}^{k}|^{2} \displaystyle\leq \displaystyle 2\tau\sum_{i=j}^{t-1}|y_{i+1}^{k}-y_{i}^{k}|^{2}+2|y_{t}^{k}|^{2} \displaystyle= \displaystyle 2\tau\sum_{i=j}^{t-1}|\eta_{i+1}^{k}-\eta_{i}^{k}|^{2}+|y_{t}^{k}| \displaystyle= \displaystyle 2\tau\gamma^{2}\sum_{i=j}^{t-1}\left|\frac{1}{L}\sum_{\ell=0}^{L-1}r_{t,\ell}^{k}-\eta^{k}_{t}\right|^{2}+2|y_{t}^{k}|^{2} \displaystyle\leq \displaystyle 8\tau^{2}U_{r}^{2}\gamma^{2}+2|y_{t}^{k}|^{2}. Taking expectation, we get \displaystyle{\mathbb{E}}_{t-2\tau}|y_{j}^{k}|^{2}\leq 8\tau^{2}U_{r}^{2}\gamma^{2}+2{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|^{2}. Analogously, we get, \displaystyle{\mathbb{E}}_{t-2\tau}|y_{j}^{k}|\leq 2\tau U_{r}\gamma+{\mathbb{E}}_{t-2\tau}|y_{t}^{k}|. ∎ Lemma I.5 . For k\in[K] , t\geq 0 , L>0 , and any state s_{t,0}^{k} , we have \displaystyle{\mathbb{E}}\left[\|{\mathbf{b}}_{t,L}^{k}\|\right]\leq\left(U_{\delta}+U_{\delta}\sqrt{\frac{2m\rho}{1-\rho}}\right)\sqrt{L}, (114) \displaystyle{\mathbb{E}}\|{\mathbf{b}}_{t,L}^{k}\|^{2}\leq U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})L, (115) \displaystyle{\mathbb{E}}\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\|\leq\sqrt{16U_{r}^{2}+32U_{r}^{2}\frac{m\rho}{1-\rho}}\sqrt{L}, (116) \displaystyle{\mathbb{E}}\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\|^{2}\leq\left(16U_{r}^{2}+32U_{r}^{2}\frac{m\rho}{1-\rho}\right)L, (117) \displaystyle{\mathbb{E}}\left\|{\widetilde{A}_{t,L}^{k}-A_{L}^{k}}\right\|\leq\sqrt{16+32\frac{m\rho}{1-\rho}}\sqrt{L}, (118) \displaystyle{\mathbb{E}}\left\|{\widetilde{A}_{t,L}^{k}-A_{L}^{k}}\right\|^{2}\leq\left(16+32\frac{m\rho}{1-\rho}\right)L, (119) where U_{\delta}:=2U_{r}+2U_{\omega} , and the expectation is taken over the Markovian sampling trajectories. Proof. We know that V(s)=\phi(s)^{\top}z^{k,*}={\mathbb{E}}_{\pi\otimes P^{k}}[R(s,a)-J^{k}+V(s^{\prime})] , which implies for any s , \displaystyle{\mathbb{E}}_{s\sim u^{k},(a,s^{\prime})\sim\pi\otimes P^{k}}[R(s,a)-J^{k}+(\phi(s^{\prime})-\phi(s))^{\top}z^{k,*}]=0. (120) Then, for \|{\mathbf{b}}_{t,L}^{k}\| , taking expectation, we have \displaystyle{\mathbb{E}}\|{\mathbf{b}}_{t,L}^{k}\|\leq\sqrt{{\mathbb{E}}\|{\mathbf{b}}_{t,L}^{k}\|^{2}}~~~~(\text{by Jensen's inequality}) \displaystyle= \displaystyle\sqrt{{\mathbb{E}}\left\|\sum_{\ell=0}^{L-1}\left[(r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\right]\phi(s_{t,0}^{k})\right\|^{2}} \displaystyle\leq \displaystyle\Bigg(\sum_{\ell=0}^{L-1}{\mathbb{E}}\left\|\left[(r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\right]\phi(s_{t,0}^{k})\right\|^{2} \displaystyle+2\sum_{\ell=0}^{L-1}\sum_{\ell^{\prime}=\ell+1}^{L-1}{\mathbb{E}}\bigg\langle\left[(r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\right]\phi(s_{t,0}^{k}), \displaystyle\qquad\qquad\qquad\qquad\left[(r_{t,\ell^{\prime}}^{k}-J^{k})+(\phi(s_{t,\ell^{\prime}+1}^{k})-\phi(s_{t,\ell^{\prime}}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\right]\phi(s_{t,0}^{k})\bigg\rangle\Bigg)^{\frac{1}{2}} \displaystyle= \displaystyle\Bigg(\sum_{\ell=0}^{L-1}{\mathbb{E}}\left\|\left[(r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\right]\phi(s_{t,0}^{k})\right\|^{2} \displaystyle+2\sum_{\ell=0}^{L-1}\sum_{\ell^{\prime}=\ell+1}^{L-1}{\mathbb{E}}\left[{\mathbb{E}}\left[\bigg\langle\left[(r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\right]\phi(s_{t,0}^{k}),\right.\right. \displaystyle\qquad\qquad\qquad\qquad\left.\left.\left[(r_{t,\ell^{\prime}}^{k}-J^{k})+(\phi(s_{t,\ell^{\prime}+1}^{k})-\phi(s_{t,\ell^{\prime}}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\right]\phi(s_{t,0}^{k})\bigg\rangle\mid v_{0:{(j,\ell)}}\right]\right]\Bigg)^{\frac{1}{2}} \displaystyle\leq \displaystyle\Bigg(U_{\delta}^{2}L+2\sum_{\ell=0}^{L-1}\sum_{\ell^{\prime}=\ell+1}^{L-1}{\mathbb{E}}\left[\langle\left[(r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\right]\phi(s_{t,0}^{k}),\right. \displaystyle\qquad\qquad\qquad\qquad\left.{\mathbb{E}}[\left[(r_{t,\ell^{\prime}}^{k}-J^{k})+(\phi(s_{t,\ell^{\prime}+1}^{k})-\phi(s_{t,\ell^{\prime}}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\right]\phi(s_{t,0}^{k})\mid v_{0:{(j,\ell)}}]\right]\rangle\Bigg)^{\frac{1}{2}} \displaystyle\leq \displaystyle\Bigg(U_{\delta}^{2}L+2\sum_{\ell=0}^{L-1}\sum_{\ell^{\prime}=\ell+1}^{L-1}{\mathbb{E}}[\left\|{\left[(r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\right]\phi(s_{t,0}^{k})}\right\| \displaystyle\qquad\qquad\qquad\qquad\|{\mathbb{E}}[\left[(r_{t,\ell^{\prime}}^{k}-J^{k})+(\phi(s_{t,\ell^{\prime}+1}^{k})-\phi(s_{t,\ell^{\prime}}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\right]\phi(s_{t,0}^{k})\mid v_{0:{(t,\ell)}}]\|]\Bigg)^{\frac{1}{2}}(\text{by Cauchy-Schwarz}) \displaystyle\overset{(a)}{\leq} \displaystyle\sqrt{U_{\delta}^{2}L+2\sum_{\ell=0}^{L-1}\sum_{\ell^{\prime}=\ell+1}^{L-1}U_{\delta}\left\|{\mathbb{E}}\left[\left[(r_{t,\ell^{\prime}}^{k}-J^{k})+(\phi(s_{t,\ell^{\prime}+1}^{k})-\phi(s_{t,\ell^{\prime}}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\right]\phi(s_{t,0}^{k})\mid v_{0:{(t,\ell)}}\right]\right\|} \displaystyle\leq \displaystyle\sqrt{U_{\delta}^{2}L+2\sum_{\ell=0}^{L-1}\sum_{\ell^{\prime}=\ell+1}^{L-1}U_{\delta}^{2}m\rho^{\ell^{\prime}-\ell}}~~~~~~~~~~(\text{by the ergodicity Assumption}) \displaystyle\leq \displaystyle U_{\delta}\sqrt{L}+\sqrt{2U_{\delta}^{2}m\frac{\rho L}{1-\rho}} \displaystyle= \displaystyle\left(U_{\delta}+U_{\delta}\sqrt{\frac{2m\rho}{1-\rho}}\right)\sqrt{L}, where inequality (a) holds because \left\|{\left[(r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\right]\phi(s_{t,0}^{k})}\right\|\leq U_{\delta} . Similarly, for Eq. ( 115 ), we have \displaystyle{\mathbb{E}}\|{\mathbf{b}}_{t,L}^{k}\|^{2}\leq U_{\delta}^{2}L+2U_{\delta}^{2}\frac{m\rho}{1-\rho}L=U_{\delta}^{2}(1+2\frac{m\rho}{1-\rho})L. Now we prove Eq. ( 117 ). For ease of exposition, we omit the explicit specification of the underlying distributions in the expectation when it is clear from the context. \displaystyle{\mathbb{E}}\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\|^{2} \displaystyle= \displaystyle{\mathbb{E}}\left\|{\sum_{\ell=0}^{L-1}(r_{t,\ell}^{k}-J^{k})\phi(s_{t,0}^{k})-{\mathbb{E}}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left[\sum_{\ell=0}^{L-1}(r(s^{(\ell)},a^{(\ell)})-J^{k})\phi(s^{(0)})\right]}\right\|^{2} \displaystyle\leq \displaystyle\sum_{\ell=0}^{L-1}{\mathbb{E}}\left\|{\left[(r_{t,\ell}^{k}-J^{k})\phi(s_{t,0}^{k})-{\mathbb{E}}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left[(r(s^{(\ell)},a^{(\ell)})-J^{k})\phi(s^{(0)})\right]\right]}\right\|^{2} \displaystyle+2\sum_{\ell=0}^{L-1}\sum_{\ell^{\prime}=\ell+1}^{L-1}{\mathbb{E}}\big\langle(r_{t,\ell}^{k}-J^{k})\phi(s_{t,0}^{k})-{\mathbb{E}}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left[(r(s^{(\ell)},a^{(\ell)})-J^{k})\phi(s^{(0)})\right], \displaystyle\qquad\qquad\qquad(r_{t,\ell^{\prime}}^{k}-J^{k})\phi(s_{t,0}^{k})-{\mathbb{E}}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left[(r(s^{(\ell^{\prime})},a^{(\ell^{\prime})})-J^{k})\phi(s^{(0)})\right]\big\rangle \displaystyle\overset{(a)}{\leq} \displaystyle 16U_{r}^{2}L+32U_{r}^{2}\sum_{\ell=0}^{L-1}\sum_{\ell^{\prime}=\ell+1}^{L-1}m\rho^{\ell^{\prime}-\ell} \displaystyle\leq \displaystyle\left(16U_{r}^{2}+32U_{r}^{2}\frac{m\rho}{1-\rho}\right)L. (a) follows analogously to the proof of Eq. ( 114 ). For Eq. ( 116 ), by Jensen’s inequality, we easily get \displaystyle{\mathbb{E}}\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\|\leq\sqrt{{\mathbb{E}}\|\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\|^{2}}\leq\sqrt{16U_{r}^{2}+32U_{r}^{2}\frac{m\rho}{1-\rho}}\sqrt{L}. Similarly, for Eq. ( 119 ), we have, \displaystyle{\mathbb{E}}\left\|{\widetilde{A}_{t,L}^{k}-A_{L}^{k}}\right\|^{2} \displaystyle= \displaystyle{\mathbb{E}}\left\|{\sum_{\ell=0}^{L-1}\phi(s_{t,0}^{k})(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}-\mathbb{E}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}[\phi(s^{(0)})(\phi(s^{(\ell+1)})-\phi(s^{(\ell)}))^{\top}}\right\|^{2} \displaystyle= \displaystyle\sum_{\ell=0}^{L-1}{\mathbb{E}}\left\|{\phi(s_{t,0}^{k})(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}-\mathbb{E}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}[\phi(s^{(0)})(\phi(s^{(\ell+1)})-\phi(s^{(\ell)}))^{\top}}\right\|^{2} \displaystyle+2\sum_{\ell=0}^{L-1}\sum_{\ell^{\prime}=0}^{L-1}{\mathbb{E}}\left\langle\phi(s_{t,0}^{k})(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}-\mathbb{E}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}[\phi(s^{(0)})(\phi(s^{(\ell+1)})-\phi(s^{(\ell)}))^{\top},\right. \displaystyle\qquad\qquad\qquad\left.\phi(s_{t,0}^{k})(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}-\mathbb{E}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}[\phi(s^{(0)})(\phi(s^{(\ell+1)})-\phi(s^{(\ell)}))^{\top}\right\rangle \displaystyle\leq \displaystyle 16L+32\sum_{\ell=0}^{L-1}\sum_{\ell^{\prime}=\ell+1}^{L-1}m\rho^{\ell^{\prime}-\ell} \displaystyle\leq \displaystyle\left(16+32\frac{m\rho}{1-\rho}\right)L. By Jensen’s inequality, Eq. ( 118 ) can be proved, \displaystyle{\mathbb{E}}\left\|{\widetilde{A}_{t,L}^{k}-A_{L}^{k}}\right\|\leq\sqrt{{\mathbb{E}}\left\|{\widetilde{A}_{t,L}^{k}-A_{L}^{k}}\right\|^{2}}\leq\sqrt{16+32\frac{m\rho}{1-\rho}}\sqrt{L}. ∎ Lemma I.6 . Suppose that Assumption 2 . Fix \tau>0 . For any t\geq\tau , L>0 , and k\in[K] , it holds that \displaystyle\|{\mathbb{E}}_{t-\tau}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big]\|\leq 4LU_{r}m\rho^{\tau} \displaystyle\|{\mathbb{E}}_{t-\tau}\big[\widetilde{A}_{t,L}^{k}-\bar{A}_{L}^{k}\big]\|\leq 4m\rho^{\tau} \displaystyle\|{\mathbb{E}}_{t-\tau}[{\mathbf{b}}_{t,L}^{k}]\|\leq 2U_{\delta,L}m\rho^{\tau}. Proof. Recall from Eq. ( B ) that \displaystyle{\mathbf{b}}_{t,L}^{k}=\sum_{\ell=0}^{L-1}\big((r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\big)\phi(s_{t,0}^{k}), and the average-reward Bellman equation \mathbb{E}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left[\sum_{\ell=0}^{L-1}(r(a^{(\ell)},s^{(\ell)})-J^{k})+(\phi(s^{(\ell+1)})-\phi(s^{(\ell)}))^{\top}B^{*}w^{k,*}\phi(s^{(0)})\right]=0. Thus, we have \displaystyle\|{\mathbb{E}}_{t-\tau}[{\mathbf{b}}_{t,L}^{k}]\|= \displaystyle\left|{\mathbb{E}}_{t-\tau}\left[\sum_{\ell=0}^{L-1}\big((r_{t,\ell}^{k}-J^{k})+(\phi(s_{t,\ell+1}^{k})-\phi(s_{t,\ell}^{k}))^{\top}\mathbf{B}^{*}\omega^{k,*}\big)\phi(s_{t,0}^{k})\right]\right. \displaystyle\quad\left.-\mathbb{E}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left[\sum_{\ell=0}^{L-1}(r(a^{(\ell)},s^{(\ell)})-J^{k})+(\phi(s^{(\ell+1)})-\phi(s^{(\ell)}))^{\top}B^{*}w^{k,*}\phi(s^{(0)})\right]\right| \displaystyle\leq \displaystyle 2U_{\delta,L}\cdot d_{TV}(\mathbb{P}_{0:\tau}^{k}(s_{t,0}^{k},a_{t,1}^{k},s_{t,1}^{k},\cdots,a_{t,L-1}^{k},s_{t,L}^{k}\mid s_{t-\tau}^{k}=s),\mu^{k}\otimes(\pi\otimes P^{k})^{L-1}) \displaystyle\leq \displaystyle 2U_{\delta,L}m\rho^{\tau}, where the notation \mathbb{P}_{0:\tau}^{k} is defined in Assumption 2 , the first inequality follows from Lemma H.2 , and the last inequality follows from Assumption 2 and the fact that L -block of a Markov chain mixes at the same rate as the original chain. Recall from Eq. ( 14 ) that \displaystyle\widetilde{b}_{t,L}^{k}=\sum_{\ell=0}^{L-1}(r_{t,\ell}^{k}-J^{k})\phi(s_{t,0}^{k});\quad\bar{b}_{L}^{k}={\mathbb{E}}_{s^{(0)}\sim\mu^{k},(a^{(\ell)},s^{(\ell)})\sim\pi\otimes P^{k}~\forall\ell\in[0,L-1]}\left[\sum_{\ell=0}^{L-1}(r(s^{(\ell)},a^{(\ell)})-J^{k})\phi(s^{(0)})\right]. Note that \left|\widetilde{b}_{t,L}^{k}\right|\leq 2LU_{r}. Similar to the above derivation, we have \displaystyle\|{\mathbb{E}}_{t-\tau}\big[\widetilde{b}_{t,L}^{k}-\bar{b}_{L}^{k}\big]\| \displaystyle\leq 2LU_{r}2\cdot d_{TV}(\mathbb{P}_{0:\tau}^{k}(s_{t,0}^{k},a_{t,1}^{k},s_{t,1}^{k},\cdots,a_{t,L-1}^{k},s_{t,L}^{k}\mid s_{t-\tau}^{k}=s),\mu^{k}\otimes(\pi\otimes P^{k})^{L-1}) \displaystyle\leq 4LU_{r}m\rho^{\tau}. Recall from Eq. ( B ) that \widetilde{A}_{t,L}^{k}=\phi(s_{t,0}^{k})(\phi(s_{t,L}^{k})-\phi(s_{t,0}^{k}))^{\top} . Since \|\widetilde{A}_{t,L}^{k}\|\leq 2 , we have \displaystyle\|{\mathbb{E}}_{t-\tau}\big[\widetilde{A}_{t,L}^{k}-\bar{A}_{L}^{k}\big]\|\leq 4m\rho^{\tau}. ∎ I.1 Proof of Eq. ( 31 ) Proof. Recall that {\mathcal{R}} is the optimal rotation matrix that aligns {\mathbf{B}}_{t+1} and {\mathbf{B}}^{*} in the sense that {\mathcal{R}}=\text{argmin}_{{\mathcal{R}}^{\top}{\mathcal{R}}={\mathbf{I}}}\|{\mathbf{B}}_{t+1}{\mathcal{R}}-{\mathbf{B}}^{*}\|_{F} . That is, {\mathcal{R}}={\mathbf{U}}{\mathbf{V}}^{\top} , where {\mathbf{U}}\Sigma{\mathbf{V}}^{\top}={\mathbf{B}}_{t+1}{\mathbf{B}}^{*} is the SVD. We have \displaystyle{\mathbf{I}}-({\mathbf{B}}_{t+1}{\mathcal{R}})^{\top}{\mathbf{B}}^{*} \displaystyle= \displaystyle{\mathbf{I}}-{\mathcal{R}}^{\top}{\mathbf{B}}_{t+1}^{\top}{\mathbf{B}}^{*} \displaystyle= \displaystyle{\mathbf{I}}-({\mathbf{U}}{\mathbf{V}}^{\top})^{\top}{\mathbf{U}}\Sigma{\mathbf{V}}^{\top} \displaystyle= \displaystyle{\mathbf{I}}-{\mathbf{V}}\Sigma{\mathbf{V}}^{\top}. For any given matrix {\mathbf{A}} , let \sigma_{i}({\mathbf{A}}) denote its i -th largest singular value. Since multiplying an orthogonal matrix does not change the singular values, the singular values of {\mathbf{I}}-({\mathbf{B}}_{t+1}{\mathcal{R}})^{\top}{\mathbf{B}}^{*} can be bounded as follows, \displaystyle\sigma_{i}({\mathbf{I}}-{\mathbf{V}}\Sigma{\mathbf{V}}^{\top})=\sigma_{i}({\mathbf{V}}({\mathbf{I}}-\Sigma){\mathbf{V}}^{\top})=\sigma_{i}({\mathbf{I}}-\Sigma)=1-\cos(\theta_{i}), where \cos(\theta_{i}) are the singular values of the matrix ({\mathbf{B}}_{t+1}{\mathcal{R}})^{\top}{\mathbf{B}}^{*} , and \theta_{i} are the principal angles between the two subspaces of interest. Then \displaystyle\sigma_{i}({\mathbf{I}}-({\mathbf{B}}_{t+1}{\mathcal{R}})^{\top}{\mathbf{B}}^{*})=1-\cos(\theta_{i})\leq 1-\cos^{2}(\theta_{i})=\sin^{2}(\theta_{i}). Recall \sigma_{i}({\mathbf{B}}_{\perp}^{*\top}{\mathbf{B}}_{t+1})=\sin(\theta_{i}) . All the singular values of {\mathbf{I}}-({\mathbf{B}}_{t+1}{\mathcal{R}})^{\top}{\mathbf{B}}^{*} are smaller than the largest singular value of {\mathbf{B}}_{\perp}^{*\top}{\mathbf{B}}_{t+1} squared, i.e., \displaystyle\sigma_{i}({\mathbf{I}}-({\mathbf{B}}_{t+1}{\mathcal{R}})^{\top}{\mathbf{B}}^{*})\leq\|{\mathbf{B}}_{\perp}^{*\top}{\mathbf{B}}_{t+1}\|^{2}, which implies \displaystyle\|{\mathbf{I}}-({\mathbf{B}}_{t+1}{\mathcal{R}})^{\top}{\mathbf{B}}^{*}\|\leq\|{\mathbf{B}}_{\perp}^{*\top}{\mathbf{B}}_{t+1}\|^{2}. ∎ Appendix J More Experiment Results Figure 4 : Policy Learning Performance Comparison on Acrobot Figure 5 : Policy Learning Performance Comparison on Cartpole