Accelerated Extragradient-Type Methods: Generalization and Unified Convergence Analysis Quoc Tran-Dinh, Nghia Nguyen-Trung Abstract This paper provides a comprehensive study for two types of extragradient methods: anchored extragradient-type and Nesterov’s accelerated extragradient-type for solving [non]linear inclusions (and, in particular, equations), primarily under the Lipschitz continuity and the co-hypomonotonicity assumptions. It consists of the following main contributions. First, we unify and generalize a class of anchored extragradient methods for monotone inclusions to a wider range of schemes encompassing existing algorithms as special cases. We establish 𝒪 ⁡ ( 1 / k ) \mathcal{O}\big(1/k\big) last-iterate convergence rates on the residual norm of the underlying mapping for this general framework and then specialize it to obtain convergence guarantees for specific instances, where k k denotes the iteration counter. Second, we extend our approach to a class of anchored Tseng’s forward-backward-forward splitting (FBFS) methods to obtain a broader class of algorithms for solving co-hypomonotone inclusions, covering a class of nonmonotone problems. Again, we prove 𝒪 ⁡ ( 1 / k ) \mathcal{O}\big(1/k\big) last-iterate convergence rates for this general scheme and specialize it to obtain convergence results for existing and new variants. Third, we develop a new class of moving anchor FBFS algorithms that achieves both 𝒪 ⁡ ( 1 / k ) \mathcal{O}\big(1/k\big) and o ⁡ ( 1 / k ) o\big(1/k\big) convergence rates, and the convergence of iterate sequences. Fourth, we generalize and unify Nesterov’s accelerated extra-gradient method to a new class of algorithms that covers existing schemes as special instances while generating new variants. For these schemes, we can prove 𝒪 ⁡ ( 1 / k ) \mathcal{O}\left(1/k\right) last-iterate convergence rates for the residual norm under co-hypomonotonicity. Fifth, we propose another novel class of Nesterov’s accelerated extragradient methods to solve inclusions. Interestingly, these algorithms achieve both 𝒪 ⁡ ( 1 / k ) \mathcal{O}\left(1/k\right) and o ⁡ ( 1 / k ) o\left(1/k\right) last-iterate convergence rates, and also the convergence of iterate sequences under co-hypomonotonicity and Lipschitz continuity. Finally, we provide a set of numerical experiments encompassing different scenarios to validate our results. ∎ MSC 90C25 90C06 90-08 1 Introduction Overview. The generalized equation , also known as the nonlinear inclusion , serves as a versatile mathematical tool for modeling a wide range of problems in computational mathematics and related disciplines. It encompasses optimization problems (both constrained and unconstrained), minimax optimization, variational inequalities, complementarity problems, game theory, and fixed-point problems, see, e.g., [ 9 , 17 , 28 , 61 , 64 , 66 , 68 ] . This model has found direct applications in diverse fields such as operations research, economics, uncertainty quantification, and transportation, see, e.g., [ 12 , 29 , 35 , 28 , 38 ] . In recent years, there has been a surge of interest in minimax problems, a special case of generalized equations, driven by their applications in machine learning and robust optimization. This is particularly evident in the areas of generative adversarial networks (GANs), adversarial training, and distributionally robust optimization, see, e.g., [ 2 , 12 , 31 , 41 , 45 , 63 ] . Furthermore, minimax models have emerged as valuable tools in online learning and reinforcement learning, see, e.g., [ 2 , 7 , 13 , 31 , 36 , 41 , 43 , 45 , 63 , 84 ] . The growing prominence of these applications has spurred renewed research efforts in nonlinear inclusions, fixed-point problems, and operator equations. 1.1 Problem statements, equivalent forms, and special cases \mathrm{(a)} Problem statement. In this paper, we consider the following generalized equation (also known as a [composite] [non]linear inclusion ): \textrm{Find $x^{\star}\in\mathrm{dom}(\Phi)$ such that:}\quad 0\in\Phi x^{\star}\equiv Fx^{\star}+Tx^{\star}, (NI) where F:\mathbb{R}^{p}\to\mathbb{R}^{p} is a single-valued operator, T:\mathbb{R}^{p}\rightrightarrows 2^{\mathbb{R}^{p}} is a set-valued (or multivalued) mapping from \mathbb{R}^{p} to 2^{\mathbb{R}^{p}} (the set of all subsets of \mathbb{R}^{p} ), \Phi:=F+T , and \mathrm{dom}(\Phi):=\mathrm{dom}(F)\cap\mathrm{dom}(T) is the domain of \Phi , which is the intersection of the domains of F and T . We denote \mathrm{zer}(\Phi) the solution set of ( NI ). In this paper, we work with finite-dimensional Euclidean spaces \mathbb{R}^{p} and \mathbb{R}^{n} . Nevertheless, we believe that most results presented in this paper can be extended to Hilbert spaces. This work is also a continuation of our recent paper [ 81 ] on non-accelerated extragradient-type methods. \mathrm{(b)} Special cases. Problem ( NI ) covers the following special cases. \mathrm{(i)} If T=0 , then ( NI ) reduces to a [non]linear equation : \textrm{Find $x^{\star}\in\mathrm{dom}(F)$ such that:}\quad Fx^{\star}=0. (NE) Though ( NE ) is a special case of ( NI ), in many cases, ( NI ) can be reformulated equivalently to ( NE ) via, e.g., Moreau-Yosida’s approximation or forward-backward splitting residual operator, see, e.g., [ 80 ] . \mathrm{(ii)} Note that ( NE ) is equivalent to the following fixed-point problem: \textrm{Find $x^{\star}\in\mathrm{dom}(G)$ such that:}\quad x^{\star}=Gx^{\star}, (FixP) where G:\mathbb{R}^{p}\to\mathbb{R}^{p} is a given operator. Clearly, by defining Fx:=x-Gx , then ( FixP ) is equivalent to ( NE ). Alternatively, we can rewrite ( NI ) equivalently to Kakutani’s fixed-point problem: Find x^{\star} such that x^{\star}\in\Psi{x^{\star}} in the context of multivalued mappings, where \Psi:=\mathbb{I}-\Phi . \mathrm{(iii)} If T:=\partial{g} , the subdifferential of a proper, closed, and convex function g:\mathbb{R}^{p}\to\mathbb{R}\cup\{+\infty\} , then ( NI ) reduces a mixed variational inequality problem (MVIP): Find x^{\star}\in\mathbb{R}^{p} such that: \langle Fx^{\star},x-x^{\star}\rangle+g(x)-g(x^{\star})\geq 0,\ \forall x\in\mathbb{R}^{p}. (MVIP) In particular, if T=\mathcal{N}_{\mathcal{X}} , the normal cone of a nonempty, closed, and convex set \mathcal{X} in \mathbb{R}^{p} (i.e. g=\delta_{\mathcal{X}} , the indicator of \mathcal{X} ), then ( MVIP ) reduces to the classical (Stampacchia’s) variational inequality problem (VIP): \textrm{Find $x^{\star}\in\mathcal{X}$ such that:}\quad\langle Fx^{\star},x-x^{\star}\rangle\geq 0,\ \textrm{for all}\ x\in\mathcal{X}. (VIP) While ( VIP ) can be viewed as a primal VIP (or a strong VIP), its dual (or weak) form can be written as \textrm{Find $x^{\star}\in\mathcal{X}$ such that:}\quad\langle Fx,x-x^{\star}\rangle\geq 0,\ \textrm{for all}\ x\in\mathcal{X}, (DVIP) which is known as Minty’s variational inequality problem. If F is monotone (see the definition in Section 2 ) then both problems ( VIP ) and ( MVIP ) are equivalent, i.e. their solution sets are identical, see [ 28 , 38 ] . \mathrm{(iv)} One important special case of ( NI ) or ( VIP ) is the optimality condition of the following general minimax problem: \min_{u\in\mathbb{R}^{m}}\max_{v\in\mathbb{R}^{n}}\Big\{\mathcal{L}(u,v):=\varphi(u)+\mathcal{H}(u,v)-\psi(v)\Big\}, (1) where \varphi:\mathbb{R}^{m}\to\mathbb{R}\cup\{+\infty\} and \psi:\mathbb{R}^{n}\to\mathbb{R}\cup\{+\infty\} are often proper, closed, and convex functions, and \mathcal{H}:\mathbb{R}^{m}\times\mathbb{R}^{n}\to\mathbb{R} is a joint objective function, often assumed to be differentiable, but not necessarily convex-concave. If we denote x:=[u,v] as the concatenation of u and v , and define T:=[\partial{\varphi},\partial{\psi}] and F:=[\nabla_{u}{\mathcal{H}}(u,v),-\nabla_{v}{\mathcal{H}}(u,v)] , then the optimality condition of ( 1 ) is exactly covered by ( NI ) as a special case. Since ( NI ) covers the above problems as special cases or equivalent forms, our methods developed in this paper can be customized to solve these problems. 1.2 Our goals and contributions Before discussing related work, let us state our goals and contributions in this paper. We also compare our work with the most related and recent works. \mathrm{(a)} Our goals. Our first goal is to unify and generalize both Halpern’s and Nesterov’s accelerated extragradient-type methods, covering existing algorithms as special cases. Our second goal is to develop new classes of accelerated extragradient algorithms to solve ( NI ) with “better” theoretical guarantees. We establish convergence rates for these generalized methods under standard or weaker assumptions. Unlike classical methods using gap functions or a distance to the solution set \mathrm{zer}(\Phi) , our convergence metric is the residual norm ( cf. ( 2 )), which is applicable to both monotone and nonmonotone problems, and requires mild assumptions. We also prove the convergence of iterate sequences to a solution of ( NI ). Our approach allows for the generation of new variants through flexible choices for a general search direction, denoted by u^{k} . \mathrm{(b)} Our contributions. Our contributions are detailed below, and Table 1 offers a brief overview of our results and their distinction from prior results. \mathrm{1.} Generalized Anchored EG. We generalize the extra-anchored gradient method to a broader class of algorithms for solving ( NI ). Under a “monotonicity” assumption on F and a 3 -cyclically monotone assumption of T , we prove \mathcal{O}\big(1/k\big) -last-iterate sublinear convergence rates for this generalized scheme, where k denotes the iteration counter. This framework subsumes several common variants and known schemes in the literature as special instances (see Table 1 ), while allowing us to derive new variants. \mathrm{2.} Generalized Fast EG. We generalize the fast EG method in [ 40 ] for solving ( NI ) to a broader class of schemes and establish its \mathcal{O}\big(1/k\big) -sublinear last-iterate convergence rate under a “co-hypomonotonicity” assumption of \Phi and the Lipschitz continuity of F . This generalization encompasses the fast EG method in [ 40 ] and the past-fast EG in [ 19 ] as special instances. \mathrm{3.} Generalized Moving Anchor EG. We propose a class of generalized moving anchor EG methods for solving ( NI ). This class covers the methods in [ 1 , 88 ] as special cases and again allows us to derive new variants, including a new variant of the optimistic gradient scheme. We establish both \mathcal{O}\big(1/k\big) and o\big(1/k\big) last-iterate convergence rates under co-hypomonotonicity of \Phi and Lipschitz continuity of F . We also prove the convergence of iterate sequences to a solution of ( NI ). \mathrm{4.} Generalized Nesterov’s Accelerated EG. We study a generalized form of Nesterov’s accelerated EG method, inspired by the connection to Halpern’s methods in [ 78 ] . We provide a unified \mathcal{O}\left(1/k\right) -last-iterate convergence rate for this generalized method, which covers Tseng’s accelerated variant and Nesterov’s accelerated forward-reflected-backward method (also known as an accelerated optimistic gradient scheme). \mathrm{5.} Generalized Nesterov’s Accelerated EG with Better Guarantees. We develop a new class of Nesterov’s accelerated EG methods using different correction terms to solve ( NI ). These methods appear to have better convergence guarantees than the previous one, and encompass [ 15 , 71 ] as special cases but under the co-hypomonotone setting. We prove both \mathcal{O}\left(1/k\right) and o\left(1/k\right) convergence rates through flexible parameter updates. In addition, we also establish the convergence of iterate sequences to a solution of ( NI ), which have not been proven in the previous methods. Table 1: Summary of existing results and our methods (marked by blue color) ​​​ ​​Methods Choice of u^{k} ​​Assumptions ​​Add. Ass. ​​Rates ISC ​​​ ​​References For solving ( NE ) EAG u^{k}:=Fx^{k} F is mono None \mathcal{O}\big(1/k\big) No [ 87 ] EAG-mv u^{k}:=Fx^{k} F is chm None \mathcal{O}\big(1/k\big) No [ 1 ] FEG u^{k}:=Fx^{k} F is chm None \mathcal{O}\big(1/k\big) No [ 40 ] GEAG \mathrm{lin}(Fx^{k},Fx^{k-1},Fy^{k-1}) F is mono None \mathcal{O}\big(1/k\big) No Ours NesEG u^{k}:=Fx^{k} F is mono None \mathcal{O}\big(1/k\big) Yes [ 15 , 78 ] ​​​ PEAG / APEG u^{k}:=Fy^{k-1} F is mono None \mathcal{O}\big(1/k\big) No [ 79 , 78 ] GFEG \mathrm{lin}(Fx^{k},Fx^{k-1},Fy^{k-1}) F is chm None \mathcal{O}\big(1/k\big) No Ours GAEG \mathrm{lin}(Fx^{k},Fx^{k-1},Fy^{k-1}) F is chm None \mathcal{O}\big(1/k\big) No Ours GAEG + \mathrm{lin}(Fx^{k},Fx^{k-1},Fy^{k-1}) F is chm None ​​​​​​​ \mathcal{O}\big(1/k\big) , o\left(1/k\right) Yes Ours For solving ( NI ), ( MVIP ), and ( VIP ) EAG u^{k}:=Fx^{k} F is mono T:=\mathcal{N}_{\mathcal{X}} \mathcal{O}\big(1/k\big) No [ 18 ] SEG u^{k}:=Fx^{k} \Phi is chm None ​​​​​ \mathcal{O}\big(1/k\big),{\color[rgb]{1,0,0}{o\left(1/k\right)}} Yes [ 88 ] ​​ FEG / AEG u^{k}:=Fx^{k} \Phi is chm None \mathcal{O}\big(1/k\big) No [ 18 , 77 ] ​​​ PEAG / APEG u^{k}:=Fy^{k-1} \Phi is chm None \mathcal{O}\big(1/k\big) No [ 19 , 77 ] GEAG \mathrm{lin}(Fx^{k},Fx^{k-1},Fy^{k-1}) F is mono T is 3 -cm \mathcal{O}\big(1/k\big) No Ours GFEG \mathrm{lin}(Fx^{k},Fy^{k-1},{\color[rgb]{0,1,0}v^{k-1}}) \Phi is chm None \mathcal{O}\big(1/k\big) No Ours GFEG-mv \mathrm{lin}(Fx^{k},Fy^{k-1},{\color[rgb]{0,1,0}v^{k-1}}) \Phi is chm None ​​​​​​ \mathcal{O}\big(1/k\big) , o\left(1/k\right) Yes Ours NesEG u^{k}:=Fx^{k} F is mono T:=\mathcal{N}_{\mathcal{X}} \mathcal{O}\big(1/k\big) Yes [ 71 , 78 ] GAEG \mathrm{lin}(Fx^{k},Fy^{k-1},{\color[rgb]{0,1,0}v^{k-1}}) F is chm None \mathcal{O}\big(1/k\big) No Ours GAEG + \mathrm{lin}(Fx^{k},Fy^{k-1},{\color[rgb]{0,1,0}v^{k-1}}) F is chm None ​​​​​​ \mathcal{O}\big(1/k\big) , o\left(1/k\right) Yes Ours Abbreviations: EAG = extra-anchored gradient; FEG = fast extragradient; EAG-mv = moving anchor fast extragradient; PEAG = past extra-anchored gradient; SEG = symplectic extragradient; AEG = Nesterov’s accelerated extragradient; APEG = Nesterov’s accelerated past extragradient; and the prefix G is for Generalized (i.e. our methods). In addition, v^{k-1} is an appropriate direction depending on the method; mono = monotone; chm = co-hypomonotone; and 3-cm = 3 -cyclically monotone. Note that “Add. Ass.” means “Additional Assumptions”; and “ ISC ” abbreviates for “iterate sequence convergence”. \mathrm{(b)} Comparison. We believe that our results in this paper are new and significant. Indeed, let us compare them with the most related works here. \bullet First, one key novelty lies in our search directions, denoted by u^{k} , as illustrated in Figure 1 . While classical extragradient methods and their accelerated variants specifically employ two search directions: Fx^{k} and Fy^{k} evaluated at two distinct sequences \{x^{k}\} and \{y^{k}\} , past-extragradient and optimistic gradient-type schemes reduce the number of F evaluations by replacing Fx^{k} by Fy^{k-1} , enabling one to eliminate the sequence \{x^{k}\} . Our GEG-type methods utilize u^{k} and Fy^{k} , where u^{k} is a flexible linear combination of Fx^{k} , Fy^{k-1} , and potentially a prior computed quantity v^{k-1} . This flexibility expands the spectrum of our methods and offers a potential for improving practical performance by searching for the “best” direction u^{k} among various linear combinations. Figure 1: The search directions in existing methods (EG- and Past-EG-type) and our schemes (GEG-type): with u^{k}:=\alpha_{1}Fx^{k}+\alpha_{2}Fy^{k-1}+\alpha_{3}v^{k-1} for \alpha_{1}=1 , \alpha_{2}=0.5 , and \alpha_{3}=-0.5 . \bullet Second, our generalized EAG method in Section 3 is new because it employs a general direction u^{k} as described above and accommodates 3 -cyclically monotone operators T (covering normal cone and subdifferential mappings as special cases, but not necessarily maximally monotone). Our method subsumes several existing methods, including those in [ 18 , 79 , 87 ] , and can generate new variants by flexibly choosing u^{k} to satisfy the condition ( 6 ). However, its applicability is currently limited to monotone F and 3 -cyclically monotone T , while extensions to non-monotone cases remain open. Similarly, our generalized anchored FBFS method in Section 4 is novel and general, encompassing the algorithms from [ 19 , 40 , 77 ] as special cases. It also allows one to generate new variants through flexible choices of u^{k} under the condition ( 21 ). Nevertheless, the convergence of iterates is still unknown. \bullet Third, our moving-anchor FBFS method in Section 5 is more general and different from the one in [ 1 ] and a very recent work in [ 88 ] due to the anchor point update. Note that the moving anchor trick is very similar to classical restarting techniques in optimization [ 59 ] and in fixed-point iterations [ 44 ] . While [ 1 ] only achieves \mathcal{O}\big(1/k\big) convergence rates under different settings, [ 88 ] was able to prove a sharper rate, i.e. o\big(1/k\big) , and the convergence of the iterates. We achieve all of these convergence results for a much more general class of methods compared to [ 1 , 88 ] , covering new variants. \bullet Fourth, our generalized Nesterov’s accelerated method in Section 6 extends the first algorithm in [ 77 ] to a broader class of algorithms. This includes novel variants, such as the case where u^{k}:=Fy^{k-1} . Notably, when u^{k}:=Fy^{k-1} , our method covers those in [ 15 , 71 ] as special cases, while relaxing monotone assumption of \Phi to a co-hypomonotone one. It also allows T to be a general (e.g., maximally monotone) mapping instead of a normal cone as in [ 71 ] . However, we can only prove \mathcal{O}\left(1/k\right) rates of the residual norm \|Fx^{k}+\xi^{k}\| . \bullet Finally, our algorithm in Section 7 appears to be novel both in terms of its algorithmic form and its convergence analysis. It differs from existing methods, even in special cases such as solving ( NE ) under monotonicity assumptions. We establish both \mathcal{O}\left(1/k\right) and o\left(1/k\right) last-iterate convergence rates under co-hypomonotonicity of \Phi and Lipschitz continuity of F . Our method subsumes those in [ 15 , 71 ] as special cases and allows for the creation of new variants by the selection of u^{k} . Furthermore, our analysis is more intricate compared to the analyses in [ 15 , 71 ] and [ 88 ] due to the generalization of our scheme and the co-hypomonotonicity of \Phi . 1.3 Related work Theory and numerical methods for ( NI ) and its special cases have been extensively studied for many decades, as evidenced in monographs such as [ 9 , 28 , 52 , 67 ] and the references therein. In our recent work [ 81 ] , we studied several non-accelerated extragradient-type methods. This paper focuses on both anchored (aka Halpern’s) and Nesterov’s accelerated counterparts of the extragradient method, which are expected to achieve faster convergence rates. Therefore, we only review related work along this line. \mathrm{(a)} From non-accelerated to accelerated methods. For the EG method and its variants employing constant stepsizes, as investigated in our recent work [ 81 ] , their convergence rate on the residual norm \|Fx^{k}+\xi^{k}\| for \xi^{k}\in Tx^{k} is \mathcal{O}\big(1/\sqrt{k}\big) . This rate is unimprovable for constant parameters, as demonstrated in [ 30 ] . To enhance the convergence rates of EG-type methods from \mathcal{O}\big(1/\sqrt{k}\big) to \mathcal{O}\big(1/k\big) , it is essential to utilize variable parameters, including stepsizes, or [dual] averaging techniques [ 54 , 58 ] . For variable parameters, there are at least two distinct approaches to developing accelerated methods for solving ( NE ) and ( NI ). The first approach relies on Halpern’s fixed-point iteration [ 34 ] , while the second leverages Nesterov’s accelerated techniques. Although these approaches were developed independently, they exhibit a connection as discussed in [ 78 ] . \mathrm{(b)} Halpern’s fixed-point iteration-type methods. The Halpern fixed-point iteration, a classical method for approximating fixed points of nonexpansive operators (equivalently, finding roots of co-coercive operators), can be viewed as a blend of fixed-point iteration and adaptive Tikhonov regularization, as discussed in [ 14 ] . Alternatively, it is also known as an anchored method revolves around a fixed initial point (i.e. anchor point). While extensively studied in fixed-point theory, the first demonstration of its \mathcal{O}\left(1/k\right) last-iterate convergence rate was achieved by Sabach & Shtern in [ 70 ] and subsequently by Lieder in [ 42 ] . Notably, this \mathcal{O}\left(1/k\right) rate on the residual norm is shown to be unimprovable in [ 42 ] through a constructive example. In the last few years, we have seen significant advancements of Halpern’s fixed-point method, as evidenced by many works such as [ 26 , 40 , 79 , 87 ] . Diakonikolas et al. [ 26 ] effectively leveraged Halpern’s technique for solving both equations and variational inequalities. A seminal contribution by Yoon and Ryu [ 87 ] extended Halpern’s iteration to the EG method for solving ( NE ), resulting in the “extra-anchored gradient” (EAG) method. Remarkably, EAG maintains the \mathcal{O}\left(1/k\right) last-iterate convergence rate while only requiring monotonicity and Lipschitz continuity of F . Building upon EAG, Lee and Kim [ 40 ] extended its applicability to the co-hypomonotone setting of ( NE ), achieving the same convergence rates. Tran-Dinh and Luo [ 79 ] further expanded this line of research by applying the EAG framework to the past-extragradient method introduced by Popov [ 62 ] , leading to the “past extra-anchored gradient” (PEAG) method with the same \mathcal{O}\left(1/k\right) last-iterate convergence rates (up to a constant factor). Recent works, such as [ 18 ] and [ 19 ] , have significantly extended these results in [ 40 , 79 , 87 ] to develop methods for solving ( VIP ) and ( NI ), all while preserving the desirable \mathcal{O}\left(1/k\right) last-iterate convergence rates for \|Fx^{k}+\xi^{k}\| . Unlike Nesterov’s accelerated methods, both o\big(1/k\big) convergence rates and the convergence of iterates remain open in these works. We suspect that Halpern-type methods suffer from a key limitation: parameter selection, typically \tau_{k}=\frac{1}{k+\nu} for some \nu>1 (e.g., \nu=2 ), restricts their flexibility and can potentially hinder their performance. In addition, similar to original Nesterov’s accelerated methods, this choice of \tau_{k} may not allow us to prove faster convergence rates (e.g., o\left(1/k\right) rates) as well as the convergence of iterates as discussed in [ 21 ] . Moreover, the fixed anchor point contributes to all the iterates, potentially affects the overall performance. Recent work by Yuan and Zhang [ 88 ] addresses these issues by adapting existing Halpern’s accelerated schemes [ 18 , 40 , 87 ] and the moving anchor EG method in [ 1 ] with a new parameter choice and a forward update for the anchor point as in [ 1 ] . This approach, derived from a symplectic discretization of an associated ODE, achieves both \mathcal{O}\left(1/k\right) and o\left(1/k\right) last-iterate convergence rates on \|Fx^{k}+\xi^{k}\| under co-hypomonotonicity while enabling a more flexible parameter choice. However, the focus of [ 88 ] remains on the extragradient method, leaving extensions to past-extragradient and other schemes as an open research question. In this paper, we will also address this open question. \mathrm{(c)} Nesterov’s accelerated-type methods. Alternatively, Nesterov’s accelerated method [ 55 ] stands as a significant breakthrough in convex optimization over the past few decades. Though initially introduced in 1983, its widespread recognition surged with seminal works by Nesterov [ 57 ] and Beck and Teboulle [ 11 ] . This powerful technique has since been extensively explored and applied across diverse fields, encompassing proximal-point, coordinate gradient, stochastic gradient, primal-dual, operator splitting, conditional gradient, Newton-type, and high-order methods. While the majority of research on accelerated methods has focused on convex optimization, recent years have witnessed a surge of interest in extending these techniques to the realm of monotone equations and inclusions. Early contributions in this direction include the works of Attouch et al. [ 3 , 6 ] , Bot et al. [ 15 , 16 ] , Kim and Fessler [ 37 ] , and Maingé [ 46 , 47 ] , among many others. In this paper, we go beyond the monotonicity in prior works to tackle a class of co-hypomonotone operators, possibly covering nonmonotone applications. Developing accelerated methods for inclusions of the form ( NI ) encounters significant challenges compared to convex optimization, necessitating a fundamental shift in the construction of an appropriate Lyapunov function, serving as a key metric for convergence analysis. Existing research often leverages proximal-point methods, which were extended to accelerated schemes in [ 32 ] . An alternative approach to designing accelerated methods involves employing the “performance estimation problem” technique pioneered in [ 27 ] and further explored in [ 75 , 76 ] . This technique has found successful applications in various works, including [ 33 , 37 , 69 ] . Notably, Nesterov’s accelerated methods for diverse problem classes have been demonstrated to exhibit “optimal” convergence rates, meaning their upper bounds on convergence rates (or “oracle” complexity) align with the corresponding lower bounds in a specific sense, as evidenced in [ 56 , 86 ] , making them theoretically unimprovable. \mathrm{(d)} Other connections. It has been demonstrated that Halpern’s and Nesterov’s accelerated methods can be interpreted as discretizations of corresponding dynamical systems, a perspective frequently observed in classical gradient methods [ 73 , 74 , 88 ] . This viewpoint has spurred extensive research into new variants and extensions, as shown in works such as [ 5 , 15 , 16 , 72 , 85 ] . Furthermore, connections between Nesterov’s method and other algorithms have been explored. For instance, [ 4 ] establishes an equivalence between Nesterov’s accelerated methods and Ravine’s methods, originally proposed in 1961. Recent studies, such as [ 60 , 78 ] , have revealed relationships between Nesterov’s accelerated schemes and Halpern’s fixed-point iterations [ 34 ] , demonstrating their equivalence under specific conditions. Building upon this understanding, [ 77 , 78 ] have leveraged this perspective to develop novel Nesterov’s accelerated variants for solving ( NE ) and ( NI ), including extragradient methods. 1.4 Paper organization Our paper is organized in such a way that each section from Section 3 to Section 7 presents one class of methods and its convergence theory and special instances. Each section is relatively independent, and mostly does not depend on other sections. It starts with the algorithm framework, special instances, key technical lemmas for convergence analysis, convergence theorems, and the convergence of special cases. Technical lemmas show the flow of our analysis. For clarity of presentation, we defer technical proofs to the appendix. More specifically, the remainder of this paper is structured as follows. 1. Section 2 provides a foundation by reviewing essential concepts and relevant prior results. One important concept is the “co-hypomonotonicity”, see [ 10 ] . 2. Section 3 introduces a novel class of anchored extragradient methods designed to address monotone inclusions ( NI ), along with their accompanying convergence guarantees and special instances. 3. Building upon this, Section 4 generalizes and explores a class of anchored Tseng’s forward-backward-forward splitting schemes for solving ( NI ). This section encompasses a unified convergence analysis of the proposed method, deriving convergence rates for specific instances. 4. Section 5 proposes a class of moving anchor FBFS schemes for solving ( NI ) and establishes its convergence guarantees under the co-hypomonotonicity. 5. Section 6 delves into a generalized class of Nesterov’s accelerated schemes and analyzes their convergence rates. 6. Distinct from Sections 5 and 6 , Section 7 introduces a novel generalized Nesterov’s accelerated EG framework. Notably, we establish both \mathcal{O}\left(1/k\right) and o\left(1/k\right) convergence rates for our method and its specific instances, while also demonstrating the convergence of iterates to a solution of ( NI ). 7. Finally, Section 8 presents a set of numerical experiments to illustrate the theoretical aspects and practical ability of our proposed methods. Although the technical proofs in this paper are both innovative and substantially distinct from prior works, we have elected to relocate all of them to the appendix. This decision was made primarily to enhance the overall presentation and readability of the main body of the paper. The proofs are organized in each corresponding section as outlined in the main text of this paper. 2 Background and Preliminary Results We recall several concepts which will be used in this paper. These concepts and properties are well-known and can be found, e.g., in [ 9 , 28 , 64 , 65 , 68 ] . 2.1 Basic Concepts, Monotonicity, and Lipschitz Continuity We work with finite dimensional Euclidean spaces \mathbb{R}^{p} and \mathbb{R}^{n} equipped with standard inner product \langle\cdot,\cdot\rangle and Euclidean norm \|\cdot\| . For a multivalued mapping T:\mathbb{R}^{p}\rightrightarrows 2^{\mathbb{R}^{p}} , \mathrm{dom}(T)=\left\{x\in\mathbb{R}^{p}:Tx\not=\emptyset\right\} denotes its domain, \mathrm{ran}(T):=\bigcup_{x\in\mathrm{dom}(T)}Tx is its range, and \mathrm{gra}(T)=\left\{(x,y)\in\mathbb{R}^{p}\times\mathbb{R}^{p}:y\in Tx\right\} stands for its graph, where 2^{\mathbb{R}^{p}} is the set of all subsets of \mathbb{R}^{p} . The inverse mapping of T is defined as T^{-1}y:=\{x\in\mathbb{R}^{p}:y\in Tx\} . We say that T is closed if \mathrm{gra}(T) is closed. For a proper, closed, and convex function f:\mathbb{R}^{p}\to\mathbb{R}\cup\{+\infty\} , \mathrm{dom}(f):=\{x\in\mathbb{R}^{p}:f(x)<+\infty\} denotes the domain of f , \partial{f} denotes the subdifferential of f , and \nabla{f} stands for the [sub]gradient of f . \mathrm{(a)} Monotonicity and comonotonicity. For a multivalued mapping T:\mathbb{R}^{p}\rightrightarrows 2^{\mathbb{R}^{p}} and \rho\in\mathbb{R} , we say that T is \rho -comonotone (see [ 10 ] ) if \begin{array}[]{ll}\langle u-v,x-y\rangle\geq\rho\|u-v\|^{2},\quad\forall(x,u),(y,v)\in\mathrm{gra}(T).\end{array} If T is single-valued, then this condition reduces to \langle Tx-Ty,x-y\rangle\geq\rho\|Tx-Ty\|^{2} for all x,y\in\mathrm{dom}(T) . • If \rho=0 , then we say that T is monotone. • If \rho>0 , then T is called \rho -co-coercive. In particular, if \rho=1 , then T is firmly nonexpansive. • If \rho<0 , then T is called |\rho| -co-hypomonotone, see [ 10 , 23 ] . Clearly, T is |\rho| -co-hypomonotone iff T^{-1} is |\rho| -hypomonotone, see [ 10 ] . Note that a co-hyopomonotone operator can also be nonmonotone. A simple example is Tx=\mathbf{Q}x+\mathbf{q} for a given invertible symmetric, but not necessarily positive definite matrix \mathbf{Q} , and \mathbf{q}\in\mathbb{R}^{p} . In this case, we have \rho=-\lambda_{\min}(\mathbf{Q}^{-1})>0 . We say that T is maximally \rho -comonotone if \mathrm{gra}(T) is not properly contained in the graph of any other \rho -comonotone operator. If \rho=0 , then we say that T is maximally monotone. Note that T is maximally monotone, then \eta T is also maximally monotone for any \eta>0 , and if T and F are maximally monotone, and \mathrm{dom}(T)\cap\mathrm{int}\left(\mathrm{dom}(F)\right)\not=\emptyset , then F+T is maximally monotone. For a proper, closed, and convex function f:\mathbb{R}^{p}\to\mathbb{R}\cup\{+\infty\} , the subdifferential \partial{f} of f is maximally monotone. \mathrm{(b)} Cyclic monotonicity. We say that a mapping T is m -cyclically monotone ( m\geq 2 ) if \sum_{i=1}^{m}\langle u^{i},x^{i}-x^{i+1}\rangle\geq 0 for all (x^{i},u^{i})\in\mathrm{gra}(T) and x_{1}=x_{m+1} (see [ 9 ] ). We say that T is cyclically monotone if it is m -cyclically monotone for every m\geq 2 . If T is m -cyclically monotone, then it is also \hat{m} -cyclically monotone for any 2\leq\hat{m}\leq m . Since a 2 -cyclically monotone operator T is monotone, any m -cyclically monotone operator T is 2 -cyclically monotone, and thus is also monotone. An m -cyclically monotone operator T is called maximally m -cyclically monotone if \mathrm{gra}(T) is not properly contained into the graph of any other m -cyclically monotone operator. As proven in ( 9 , Theorem 22.18) that T is maximally cyclically monotone iff T=\partial{f} , the subdifferential of a proper, closed, and convex function f . On the one hand, there exist maximally monotone operators (e.g., rotation operators) that are not 3 -cyclically monotone, see ( 9 , Example 22.15) . On the other hand, as indicated in ( 8 , Example 2.16) , there exist maximally 3 -cyclically monotone operators that are not maximally monotone. These classes of operators are intersected with each other, and one is not a subclass of the other. \mathrm{(c)} Lipschitz continuity and contraction. A multivalued mapping T is called L -Lipschitz continuous if \sup\left\{\|u-v\|:u\in Tx,\ v\in Ty\right\}\leq L\|x-y\| for all x,y\in\mathrm{dom}(T) , where L\geq 0 is the Lipschitz constant. If T is single-valued, then this condition reduces to \|Tx-Ty\|\leq L\|x-y\| for all x,y\in\mathrm{dom}(T) . If L=1 , then we say that T is nonexpansive, while if L\in[0,1) , then we say that T is L -contractive, and L is its contraction factor. If T is \rho -co-coercive with \rho>0 , then T is also L -Lipschitz continuous with the Lipschitz constant L:=\frac{1}{\rho} . However, the reverse statement is not true in general. For a continuously differentiable function f:\mathbb{R}^{p}\to\mathbb{R} , we say that f is L -smooth if its gradient \nabla{f} is L -Lipschitz continuous on \mathrm{dom}(f) . If f is convex and L -smooth, then \nabla{f} is \frac{1}{L} -co-coercive and vice versa, see, e.g., [ 56 ] . \mathrm{(d)} Normal cone. Given a nonempty, closed, and convex set \mathcal{X} in \mathbb{R}^{p} , the normal cone of \mathcal{X} is defined as \mathcal{N}_{\mathcal{X}}(x):=\{w\in\mathbb{R}^{p}:\langle w,x-y\rangle\geq 0,\ \forall y\in\mathcal{X}\} if x\in\mathcal{X} and \mathcal{N}_{\mathcal{X}}(x)=\emptyset , otherwise. If f:=\delta_{\mathcal{X}} , the indicator of a convex set \mathcal{X} , then we have \partial{f}=\mathcal{N}_{\mathcal{X}} . Moreover, J_{\partial{f}} reduces to the projection onto \mathcal{X} . \mathrm{(e)} Resolvent and proximal operators. Given a multivalued operator T , the operator J_{T}x:=\left\{y\in\mathbb{R}^{p}:x\in y+Ty\right\} is called the resolvent of T , denoted by J_{T}x=(\mathbb{I}+T)^{-1}x , where \mathbb{I} is the identity mapping. If T is monotone, then evaluating J_{T} requires solving a strongly monotone inclusion 0\in y-x+Ty . Hence, J_{T} is well-defined and single-valued. If T=\partial{f} , the subdifferential of proper, closed, and convex function f , then J_{T} reduces to the proximal operator of f , denoted by \mathrm{prox}_{f} , which can be computed as \mathrm{prox}_{f}(x):=\mathrm{arg}\min_{y}\{f(y)+(1/2)\|y-x\|^{2}\} . In particular, if T=\mathcal{N}_{\mathcal{X}} , the normal cone of a closed and convex set \mathcal{X} , then J_{T} is the projection onto \mathcal{X} , denoted by \mathrm{proj}_{\mathcal{X}} . If T is maximally monotone, then \mathrm{ran}(\mathbb{I}+T)=\mathbb{R}^{p} (by Minty’s theorem) and T is firmly nonexpansive (and thus nonexpansive). 2.2 Exact Solutions and Approximate Solutions There are different metrics to characterize exact and approximate solutions of ( NI ). The most obvious one is the residual norm of \Phi , which is defined as r(x):=\min_{\xi\in Tx}\|Fx+\xi\|,\quad x\in\mathrm{dom}(\Phi).\vskip-2.15277pt (2) Clearly, r(x^{\star})=0 iff x^{\star}\in\mathrm{zer}(\Phi) , a solution of ( NI ). If T=0 , then r(x)=\|Fx\| . Given a tolerance \epsilon>0 , if r(\hat{x})\leq\epsilon , then we say that \hat{x} is an \epsilon -approximate solution to ( NI ). The algorithms presented in this paper use this metric as the main tool to characterize approximate solutions to ( NI ). Other metrics often used for monotone ( VIP ), a special case of ( NI ), are gap and restricted gap functions [ 28 , 38 , 58 ] , which are respectively defined as \mathbb{G}(x):=\max_{y\in\mathcal{X}}\langle Fy,y-x\rangle\quad\text{and}\quad\mathbb{G}_{\mathcal{B}}(x):=\max_{y\in\mathcal{X}\cap\mathcal{B}}\langle Fy,y-x\rangle,\vskip-2.15277pt (3) where \mathcal{B} is a given nonempty, closed, and bounded convex set. Note that, under the monotonicity of F , \mathbb{G}(x)\geq 0 for all x\in\mathcal{X} , and \mathbb{G}(x^{\star})=0 iff x^{\star} is a solution to ( VIP ). Therefore, to characterize an \epsilon -approximate solution \tilde{x} to ( VIP ), we can impose a condition \mathbb{G}(\tilde{x})\leq\epsilon . For the restricted gap function \mathbb{G}_{\mathcal{B}} , if x^{\star} is a solution of ( VIP ) and x^{\star}\in\mathcal{B} , then \mathbb{G}_{\mathcal{B}}(x^{\star})=0 . Conversely, if \mathbb{G}_{\mathcal{B}}(x^{\star})=0 and x^{\star}\in\mathrm{int}\left(\mathcal{B}\right) , the interior of \mathcal{B} , then x^{\star} is a solution of ( VIP ) in \mathcal{B} (see ( 58 , Lemma 1) ). For ( DVIP ), we can also define similar dual gap functions and restricted dual gap functions [ 58 ] . Gap functions have widely been used in the literature to characterize approximate solutions generated by many numerical methods for solving monotone ( VIP ) or ( DVIP ), see, e.g., [ 22 , 24 , 28 , 38 , 54 , 58 ] as concrete examples. If J_{\eta T} is well-defined and single-valued for some \eta>0 , and F is single-valued, then we can use the following forward-backward splitting residual: \mathcal{G}_{\eta}(x):=\tfrac{1}{\eta}\left(x-J_{\eta T}(x-\eta Fx)\right),\vskip-2.15277pt (4) to characterize solutions of ( NI ), where F is single-valued and J_{\eta T} is the resolvent of \eta T for any \eta>0 . It is clear that \mathcal{G}_{\eta}(x^{\star})=0 iff x^{\star}\in\mathrm{zer}(\Phi) . In addition, if J_{\eta T} is firmly nonexpansive, then we also have \|\mathcal{G}_{\eta}(x)\|\leq\|Fx+\xi\|,\quad(x,\xi)\in\mathrm{gra}(T). (5) Hence, for a given tolerance \epsilon>0 , if \|\mathcal{G}_{\eta}(\tilde{x})\|\leq\epsilon , then we can say that \tilde{x} is an \epsilon -approximate solution of ( NI ). If T:=\mathcal{N}_{\mathcal{X}} , i.e. ( NI ) reduces to ( VIP ), then \mathcal{G}_{1}(x) reduces to the classical natural map \Pi_{F,\mathcal{X}}(x)=x-\mathrm{proj}_{\mathcal{X}}(x-Fx) of ( VIP ), and r_{n}(x):=\|\mathcal{G}_{1}(x)\|=\|\Pi_{F,\mathcal{X}}(x)\| is the corresponding natural residual at x . From ( 5 ), we have r_{n}(x)\leq\|Fx+\xi\| for any \xi\in\mathcal{N}_{\mathcal{X}}(x) . The natural residual is a fundamental metric to study generalized equations of the form ( NI ) and ( VIP ), see, e.g., [ 28 ] for more details. Remark 1 Note that characterizing an [approximate] solution via gap or restricted gap function for ( VIP ) requires the monotonicity of F to upper bound \mathbb{G}(x^{k}) by \mathbb{G}(x^{k})\leq\max_{y\in\mathcal{X}}\langle Fx^{k},x^{k}-y\rangle . For the nonmonotone case, including co-hypomonotonicity, these gap functions are not applicable in general. 3 A Class of Extra-Anchored Gradient Methods for Monotone ( NI ) In this section, we generalize the extra-anchored gradient method (EAG) in [ 87 ] for ( NE ) and in [ 18 ] for ( VIP ) to a more general class of algorithms and for monotone inclusions of the form ( NI ). We provide a unified convergence rate analysis for our scheme and derive special instances. 3.1 A Class of Extra-Anchored Gradient Methods for ( NI ) \mathrm{(a)} The proposed method. We propose the following scheme to solve ( NI ). Starting from x^{0}\in\mathrm{dom}(\Phi) , at each iteration k\geq 0 , given u^{k} , we update \left\{\begin{array}[]{lcl}y^{k}&:=&J_{\hat{\eta}_{k}T}\left(\tau_{k}x^{0}+(1-\tau_{k})x^{k}-\hat{\eta}_{k}u^{k}\right),\\ x^{k+1}&:=&J_{\eta T}\left(\tau_{k}x^{0}+(1-\tau_{k})x^{k}-\eta Fy^{k}\right),\end{array}\right. (GEAG) where \tau_{k}\in(0,1) , \hat{\eta}_{k}>0 , and \eta>0 are given and will be determined later, and u^{k}\in\mathbb{R}^{p} is a user-defined direction satisfying the following condition: \|u^{k}\!-\!Fx^{k}\|^{2}\leq\kappa\|Fx^{k}\!-\!Fy^{k-1}\|^{2}+\hat{\kappa}\|x^{k}\!-\!y^{k-1}\!+\!\hat{\eta}_{k-1}(Fx^{k-1}\!-\!u^{k-1})\|^{2}, (6) for given parameters \kappa\geq 0 and \hat{\kappa}\geq 0 , y^{-1}:=x^{0} , and u^{-1}=u^{0}:=Fx^{0} . This condition looks slightly technical, but we will explain later how it is constructed. Our generalization in ( GEAG ) consists of the following new points: \mathrm{(i)} We assume that T is maximally 3 -cyclically monotone. This choice covers T=\mathcal{N}_{\mathcal{X}} , the normal cone of \mathcal{X} , and T=\partial{g} , the subdifferential of a convex function g as special cases, but it is more general than the subdifferentials, and is not identical to the class of maximally monotone operators. \mathrm{(ii)} We use a user-defined direction u^{k} that satisfies ( 6 ), which covers u^{k}:=Fx^{k} and u^{k}:=Fy^{k-1} as special instances. \mathrm{(b)} Three instances. By different choices of u^{k} , ( GEAG ) covers both existing and new variants of the extra-anchored gradient (EAG) methods for solving monotone inclusions of the form ( NI ). Here, let us consider at least three instances of ( GEAG ) as follows. \mathrm{(i)} Variant 1 (EAG). If we choose u^{k}:=Fx^{k} , then ( 6 ) holds with \kappa=0 and \hat{\kappa}=0 . Clearly, this variant of ( GEAG ) covers the method in [ 18 ] as a special case when T=\mathcal{N}_{\mathcal{X}} . If, in addition, T=0 , then this variant reduces to the extra-anchored gradient (EAG) scheme from [ 87 ] . In fact, GEAG purely generalizes [ 87 ] from ( NE ) to ( NI ) with a general extra-gradient direction u^{k} , determined appropriately, and a 3 -cyclically monotone T . \mathrm{(ii)} Variant 2 (Popov’s past EAG). If u^{k}:=Fy^{k-1} , then ( 6 ) holds with \kappa=1 and \hat{\kappa}=0 . This variant of ( GEAG ) covers the past-extra-anchored gradient method in [ 79 ] as a special case with T:=0 . However, this variant has not yet been studied for ( VIP ) and a general monotone inclusion ( NI ). \mathrm{(iii)} Variant 3 (Generalized EAG). If we choose u^{k}:=(1-\alpha)Fx^{k}+\alpha Fy^{k-1}+\hat{\alpha}\big(x^{k}-y^{k-1}+\hat{\eta}_{k-1}(Fx^{k-1}-u^{k-1})\big), (7) for any \alpha,\hat{\alpha}\in\mathbb{R} , then u^{k} satisfies ( 6 ) with \kappa:=\alpha^{2}\geq 0 and \hat{\kappa}:=\hat{\alpha}^{2}\geq 0 . Clearly, this choice takes the advantages of both Variant 1 and Variant 2 , while still covers other possibilities generated by the term v^{k-1}:=x^{k}-y^{k-1}+\hat{\eta}_{k-1}(Fx^{k-1}-u^{k-1}) . In particular, if T=0 , then it is trivial to show that the term v^{k} reduces to v^{k-1}:=\hat{\eta}_{k-1}Fx^{k-1}-\eta Fy^{k-1} . In this case, we get u^{k}=(1-\alpha)Fx^{k}+(\alpha-\hat{\alpha}\eta)Fy^{k-1}+\hat{\alpha}\hat{\eta}_{k-1}Fx^{k-1} , which is a linear combination of Fx^{k} , Fy^{k-1} , and Fx^{k-1} . Note that the per-iteration cost of Variant 1 and Variant 3 is essentially the same with two evaluations Fx^{k} and Fy^{k} of F , while Variant 2 only requires one evaluation Fy^{k} of F . Note also that, due to the flexibility of choosing \alpha and also \hat{\alpha} , Variant 3 covers Variant 1 and Variant 2 as special cases. If T=0 , then our methods presented above can be applied to solve ( NE ). If T=\mathcal{N}_{\mathcal{X}} , the normal cone of a convex set \mathcal{X} , then they can be utilized to solve ( VIP ). If T:=\partial{g} , the subdifferential of a convex function g , then our methods reduce to the variants for solving ( MVIP ). Obviously, our methods can also be specified to solve ( FixP ) and ( 1 ). However, we omit the details of these applications and leave them open to the readers. 3.2 Key Estimates for Convergence Analysis To facilitate the form of condition ( 6 ) and to analyze the convergence of ( GEAG ), we define the following quantities: \begin{array}[]{llcl}&w^{k}:=Fx^{k}+\xi^{k},&&\quad\hat{w}^{k}:=Fy^{k-1}+\xi^{k},\\ &\tilde{w}^{k}:=Fx^{k}+\zeta^{k},&\quad\textrm{and}&\quad\tilde{z}^{k}:=u^{k}+\zeta^{k},\end{array} (8) for some \xi^{k}\in Tx^{k} and \zeta^{k}\in Ty^{k} . Then, we can easily see that w^{k}-\hat{w}^{k}=Fx^{k}-Fy^{k-1} and \tilde{w}^{k}-\tilde{z}^{k}=Fx^{k}-u^{k} . \mathrm{(a)} Equivalent form. Using these quantities and the fact that w=J_{\eta T}(v) iff v-w\in\eta Tw , we can equivalently rewrite ( GEAG ) as \left\{\begin{array}[]{lcl}y^{k}&:=&\tau_{k}x^{0}+(1-\tau_{k})x^{k}-\hat{\eta}_{k}\tilde{z}^{k},\\ x^{k+1}&:=&\tau_{k}x^{0}+(1-\tau_{k})x^{k}-\eta\hat{w}^{k+1}.\end{array}\right. (9) Moreover, by ( 8 ), ( 9 ), and \hat{\eta}_{k-1}:=\eta(1-\tau_{k-1}) from ( 15 ) below, we get \begin{array}[]{lcl}x^{k}-y^{k-1}+\hat{\eta}_{k-1}(Fx^{k-1}-u^{k-1})&\overset{\tiny\eqref{eq:EAG4NI_ex2},\eqref{eq:EAG4NI_reform}}{=}&-\eta\hat{w}^{k}+\hat{\eta}_{k-1}\tilde{z}^{k-1}+\hat{\eta}_{k-1}(\tilde{w}^{k-1}-\tilde{z}^{k-1})\\ &=&-\eta[\hat{w}^{k}-(1-\tau_{k-1})\tilde{w}^{k-1}].\end{array} Therefore, the condition ( 6 ) is equivalent to the following form: \begin{array}[]{ll}\|u^{k}-Fx^{k}\|^{2}\leq\kappa\|Fx^{k}-Fy^{k-1}\|^{2}+\hat{\kappa}\eta^{2}\|\hat{w}^{k}-(1-\tau_{k-1})\tilde{w}^{k-1}\|^{2}.\end{array} (10) This condition will be used in our analysis below, and it also explains how the condition ( 6 ) is constructed. \mathrm{(b)} Lyapunov function. To establish the convergence of ( GEAG ), we define the following functions: \begin{array}[]{lcl}\mathcal{V}_{k}&:=&\frac{a_{k}}{2}\|w^{k}\|^{2}+b_{k}\langle w^{k},x^{k}-x^{0}\rangle+\frac{b_{0}}{\eta}\|x^{0}-x^{\star}\|^{2},\\ \mathcal{L}_{k}&:=&\mathcal{V}_{k}+\frac{c_{k}}{2}\|u^{k}-Fx^{k}\|^{2},\end{array} (11) where x^{\star}\in\mathrm{zer}(\Phi) , a_{k} , b_{k} and c_{k} are nonnegative parameters, determined later. Note that \mathcal{V}_{k} has been widely used in the literature, including [ 18 , 40 , 77 , 87 ] as a Lyapunov function to analyze convergence of extra-anchored gradient-type methods. Since we study the generalized scheme ( GEAG ), we need to add one more term \frac{c_{k}}{2}\|u^{k}-Fx^{k}\|^{2} to \mathcal{V}_{k} to handle the difference between u^{k} and Fx^{k} . \mathrm{(c)} Descent property. Now, we state the following descent property of \mathcal{V}_{k} . Lemma 1 For ( NI ), assume that \mathrm{zer}(\Phi)\neq\emptyset , F is L -Lipschitz continuous and monotone, and T is maximally 3 -cyclically monotone. Let \{(x^{k},y^{k})\} be generated by ( GEAG ) satisfying ( 6 ) and \mathcal{V}_{k} be defined by ( 11 ). Suppose that \begin{array}[]{ll}\tau_{k}\in(0,1),\quad\hat{\eta}_{k}:=\eta(1-\tau_{k}),\quad a_{k}:=\frac{\eta b_{k}(1-\tau_{k})}{\tau_{k}},\quad\text{and}\quad b_{k+1}:=\frac{b_{k}}{1-\tau_{k}}.\end{array} (12) Then, for any \omega>0 , r>0 , and c>0 , let M_{c}:=(1+c)(1+\omega)L^{2} , we have \hskip-4.30554pt\begin{array}[]{lcl}\mathcal{V}_{k}-\mathcal{V}_{k+1}&\geq&\frac{\eta b_{k+1}[\tau_{k+1}-\tau_{k}(1-\tau_{k+1})]}{2\tau_{k}\tau_{k+1}}\|w^{k+1}\|^{2}+\frac{\omega\eta b_{k+1}}{2\tau_{k}}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\frac{(1-r)a_{k}}{2}\|w^{k}-\tilde{w}^{k}\|^{2}-\frac{(c+rM_{c}\eta^{2})a_{k}}{2rc}\|Fx^{k}-u^{k}\|^{2}\\ &&+{~}\frac{\eta b_{k+1}(1-M_{c}\eta^{2})}{2\tau_{k}}\|\hat{w}^{k+1}-(1-\tau_{k})\tilde{w}^{k}\|^{2}.\end{array}\hskip-12.91663pt (13) 3.3 Convergence Analysis of ( GEAG ) and Its Special Instances \mathrm{(a)} The convergence of ( GEAG ). For \kappa and \hat{\kappa} from ( 6 ), let r\in(0,1] and \begin{array}[]{lcl}\bar{\eta}:=\frac{\sqrt{r}}{\sqrt{(1+r)(r+2\kappa)L^{2}+2\kappa\hat{\kappa}}}.\end{array} (14) Now, we are ready to state the convergence of ( GEAG ). Theorem 3.1 For ( NI ), assume that \mathrm{zer}(\Phi)\neq\emptyset , F is L -Lipschitz continuous and monotone, and T is maximally 3 -cyclically monotone. Let \{(x^{k},y^{k})\} be generated by ( GEAG ) such that u^{k} satisfies ( 6 ) and \begin{array}[]{lcl}\tau_{k}:=\frac{1}{k+\nu},\quad\eta_{k}:=\eta\in\left(0,\bar{\eta}\right],\quad\text{and}\quad\hat{\eta}_{k}:=\eta(1-\tau_{k}),\end{array} (15) where \nu>1 is given and \bar{\eta} is given in ( 14 ). Then, for any x^{\star}\in\mathrm{zer}(\Phi) and \xi^{k}\in Tx^{k} , the following result holds: \|Fx^{k}+\xi^{k}\|^{2}\leq\frac{4\|x^{0}-x^{\star}\|^{2}+\eta^{2}\|Fx^{0}+\xi^{0}\|^{2}}{\eta^{2}(k+\nu-1)^{2}}. (16) \mathrm{(b)} Two special instances. Next, we derive the convergence of the two special instances of ( GEAG ): the extra-anchored gradient (EAG) method with u^{k}:=Fx^{k} ( Variant 1 ), and the past-extra-anchored gradient (PEAG) method with u^{k}:=Fy^{k-1} ( Variant 2 ). Corollary 1 For ( NI ), assume that \mathrm{zer}(\Phi)\neq\emptyset , F is L -Lipschitz continuous and monotone, and T is maximally 3 -cyclically monotone. Let \{(x^{k},y^{k})\} be generated by ( GEAG ) using either u^{k}:=Fx^{k} or u^{k}:=Fy^{k-1} . \mathrm{(i)} Variant 1. We choose u^{k}:=Fx^{k} and the parameters as follows: \begin{array}[]{lcl}\tau_{k}:=\frac{1}{k+\nu},\quad\eta_{k}:=\eta\in\big(0,\frac{1}{L}\big],\quad\text{and}\quad\hat{\eta}_{k}:=\eta(1-\tau_{k}),\quad(\nu>1).\end{array} (17) \mathrm{(ii)} Variant 2. We choose u^{k}:=Fy^{k-1} and the parameters as follows: \begin{array}[]{lcl}\tau_{k}:=\frac{1}{k+\nu},\quad\eta_{k}:=\eta\in\big(0,\frac{1}{L\sqrt{6}}\big],\quad\text{and}\quad\hat{\eta}_{k}:=\eta(1-\tau_{k}).\end{array} (18) Then, for \xi^{k}\in Tx^{k} and x^{\star}\in\mathrm{zer}(\Phi) , the following result holds: \|Fx^{k}+\xi^{k}\|^{2}\leq\frac{4\|x^{0}-x^{\star}\|^{2}+\eta^{2}\|Fx^{0}+\xi^{0}\|^{2}}{\eta^{2}(k+\nu-1)^{2}}. (19) Proof \mathrm{(i)} If u^{k}:=Fx^{k} , then ( 6 ) holds with \kappa=\hat{\kappa}=0 . From ( 14 ), we have \bar{\eta}:=\frac{1}{L\sqrt{1+r}} . Letting r\downarrow 0^{+} , the condition \eta\in(0,\bar{\eta}] in Theorem 3.1 becomes \eta\in\big(0,\frac{1}{L}\big] as in ( 17 ). \mathrm{(ii)} If u^{k}=Fy^{k-1} , then ( 6 ) holds with \kappa=1 and \hat{\kappa}=0 . In this case, from ( 14 ), we have \bar{\eta}:=\frac{\sqrt{r}}{L\sqrt{(1+r)(2+r)}} . If we choose r:=1 , then we get \bar{\eta}:=\frac{1}{L\sqrt{6}} , leading to the choice of \eta as in ( 18 ). In both cases, we can conclude that the bound ( 19 ) is a direct consequence of ( 16 ) from Theorem 3.1 . \square 4 A Class of Anchored FBFS Methods for Inclusion ( NI ) In this section, we develop another class of Halpern-type methods by generalizing the fast extragradient method in [ 40 ] to solve ( NI ). Note that this method is different from ( GEAG ) as it is rooted from Tseng’s forward-backward-forward splitting (FBFS) method [ 83 ] instead of the extragradient method from [ 39 ] . In addition, we relax our assumption from monotonicity to co-hypomonotonicity. 4.1 A Class of Anchored FBFS Methods First, for given x^{k}\in\mathrm{dom}(\Phi) and u^{k}\in\mathbb{R}^{p} , we define the following elements: w^{k}:=Fx^{k}+\xi^{k},\quad\hat{w}^{k}:=Fy^{k-1}+\xi^{k},\quad\text{and}\quad z^{k}:=u^{k}+\xi^{k}, (20) for some \xi^{k}\in Tx^{k} . \mathrm{(a)} The proposed method. Our algorithm is described as follows. Starting from x^{0}\in\mathrm{dom}(\Phi) , at each iteration k\geq 0 , for given (x^{k},u^{k}) , we update \hskip-12.91663pt\left\{\begin{array}[]{lcl}y^{k}&:=&x^{k}+\tau_{k}(x^{0}-x^{k})-(\hat{\eta}_{k}-\beta_{k})z^{k},\\ x^{k+1}&:=&x^{k}+\tau_{k}(x^{0}-x^{k})-\eta\hat{w}^{k+1}+\beta_{k}z^{k},\end{array}\right.\hskip-12.91663pt (GFEG) where \tau_{k}\in(0,1) , \eta>0 , \hat{\eta}_{k}>0 , and \beta_{k}\geq 0 are given, determined later. The term u^{k} in z^{k} is chosen such that it satisfies the following condition: \begin{array}[]{lcl}\|Fx^{k}-u^{k}\|^{2}&\leq&\kappa\|Fx^{k}-Fy^{k-1}\|^{2}+\hat{\kappa}\|w^{k}-w^{k-1}\|^{2},\end{array} (21) for given constants \kappa\geq 0 and \hat{\kappa}\geq 0 , y^{-1}=x^{-1}:=x^{0} , and u^{0}:=Fx^{0} . \mathrm{(b)} Three special instances. We consider three special instances of ( GFEG ). \mathrm{(i)} Variant 1: The anchored FBFS method. If we choose u^{k}:=Fx^{k} , then ( 21 ) holds with \kappa=0 and \hat{\kappa}=0 . Moreover, we can see that ( GFEG ) covers the following existing methods as special cases. • If T=0 , \beta_{k}:=0 , and \hat{\eta}_{k}=\eta=\eta_{k} (fixed or varying), then ( GFEG ) reduces to the extra-anchored gradient (EAG) scheme for solving ( NE ) in [ 87 ] under the monotonicity of F as \left\{\begin{array}[]{lcl}y^{k}&:=&\tau_{k}x^{0}+(1-\tau_{k})x^{k}-\eta_{k}Fx^{k},\\ x^{k+1}&=&\tau_{k}x^{0}+(1-\tau_{k})x^{k}-\eta_{k}Fy^{k}.\end{array}\right. • If T=0 , \beta_{k}:=2\rho(1-\tau_{k}) , and \hat{\eta}_{k}:=\eta(1-\tau_{k}) , then ( GFEG ) reduces to the fast extragradient (FEG) variant for solving ( NE ) in [ 40 ] , but under the co-hypomonotonicity of F . • If T is a maximally monotone operator (e.g., T:=\mathcal{N}_{\mathcal{X}} , the normal cone of a nonempty, closed, and convex set \mathcal{X} ), \beta_{k}:=2\rho(1-\tau_{k}) and \hat{\eta}_{k}:=\eta(1-\tau_{k}) , then ( GFEG ) is exactly the variant of EAG studied in [ 18 ] . \mathrm{(ii)} Variant 2: Anchored FRBS method. If we choose u^{k}:=Fy^{k-1} , then ( 21 ) holds with \kappa=1 and \hat{\kappa}=0 . In this case, ( GFEG ) reduces to \left\{\begin{array}[]{lcl}y^{k}&:=&\tau_{k}x^{0}+(1-\tau_{k})x^{k}-(\hat{\eta}_{k}-\beta_{k})\hat{w}^{k}\\ x^{k+1}&:=&\tau_{k}x^{0}+(1-\tau_{k})x^{k}-\eta\hat{w}^{k+1}+\beta_{k}\hat{w}^{k}.\end{array}\right. (22) This scheme is a variant of Popov’s past extra-gradient method in [ 62 ] . We call it the anchored FRBS scheme that covers variants studied in [ 19 , 77 ] . This scheme can also be viewed as a Halpern’s accelerated variant of the forward-reflected backward splitting (FRBS) or the optimistic gradient method in the literature, see also [ 25 , 51 , 53 ] . When T=0 , as discussed in Part 1 of our work [ 81 ] , we can view ( GFEG ) as a Halpern’s acceleration of the forward-reflected-backward splitting scheme in [ 50 ] , which is also equivalent to the reflected gradient method in [ 48 ] , the reflected forward-backward splitting scheme in [ 20 ] , and the golden ratio method in [ 49 ] for solving ( NE ). This scheme can also be viewed as a Halpern’s accelerated variant of the optimistic gradient method in the literature, see also [ 25 , 51 , 53 ] . Hence, ( GFEG ) is sufficiently general to cover several existing methods as its instances. \mathrm{(iii)} Variant 3: Generalization. We can construct a generalized direction: \begin{array}[]{lcl}u^{k}&:=&\alpha Fx^{k}+\hat{\alpha}Fy^{k-1}+(1-\alpha-\hat{\alpha})Fx^{k-1}+\hat{\alpha}(\xi^{k}-\xi^{k-1}),\end{array} (23) for any given constants \alpha,\hat{\alpha}\in\mathbb{R} . Clearly, u^{k} is an affine combination of Fx^{k} , Fy^{k-1} , Fx^{k-1} , and \xi^{k}-\xi^{k-1} . Moreover, we can verify that u^{k} satisfies ( 21 ) with \kappa=(1+m)\hat{\alpha}^{2} and \hat{\kappa}=(1+m^{-1})(1-\alpha-\hat{\alpha})^{2} by Young’s inequality for any m>0 . Generally, u^{k} requires Fx^{k} and/or Fx^{k-1} , and thus ( GFEG ) needs at most two evaluations of F at each iteration k , which is the same cost as Variant 1 . In fact, ( GFEG ) is rooted from Tseng’s forward-backward-forward splitting method [ 83 ] instead of the extragradient method [ 39 ] because it only requires one resolvent evaluation J_{\eta T} per iteration. Recently, [ 77 ] provides an elementary convergence analysis for ( GFEG ), which relies on the technique in [ 87 ] . In this paper, we generalize these variants to ( GFEG ), which covers a wide range of variants, including [ 40 , 77 , 87 ] as special instances. In particular, if T=0 , then ( GFEG ) solves ( NE ), and ( 21 ) reduces to \begin{array}[]{lcl}\|Fx^{k}-u^{k}\|^{2}&\leq&\kappa\|Fx^{k}-Fy^{k-1}\|^{2}+\hat{\kappa}\|Fx^{k}-Fx^{k-1}\|^{2}.\end{array} In addition, u^{k} in ( 23 ) reduces to u^{k}:=\alpha Fx^{k}+\hat{\alpha}Fy^{k-1}+(1-\alpha-\hat{\alpha})Fx^{k-1} . \mathrm{(c)} The implementation of ( GFEG ) . Since x^{k+1} are in both sides of line 2 of ( GFEG ), by using the resolvent J_{\eta T} of T , we can rewrite ( GFEG ) as \left\{\begin{array}[]{lcl}y^{k}&:=&x^{k}+\tau_{k}(x^{0}-x^{k})-(\hat{\eta}_{k}-\beta_{k})(u^{k}+\xi^{k}),\\ x^{k+1}&:=&J_{\eta T}\left(y^{k}-\eta Fy^{k}+\hat{\eta}_{k}(u^{k}+\xi^{k})\right),\\ \xi^{k+1}&:=&\frac{1}{\eta}\big[y^{k}+\hat{\eta}_{k}(u^{k}+\xi^{k})-x^{k+1}\big]-Fy^{k},\end{array}\right. (24) where w^{k}:=Fx^{k}+\xi^{k} , \xi^{0}\in Tx^{0} is arbitrary. Generally, we do not require J_{\eta T} to be single-valued, but only assume that \mathrm{ran}(J_{\eta T})\subseteq\mathrm{dom}(F)=\mathbb{R}^{p} , and \mathrm{dom}(J_{\eta T})=\mathbb{R}^{p} so that the iterates are well-defined. However, to simplify our analysis, we assume that J_{\eta T} is single-valued. In this case, we say that J_{\eta T} is well-defined . Since w^{k}:=Fx^{k}+\xi^{k} , the condition ( 21 ) can be rewritten as \|Fx^{k}-u^{k}\|^{2}\leq\kappa\|Fx^{k}-Fy^{k-1}\|^{2}+\hat{\kappa}\|Fx^{k}-Fx^{k-1}+\xi^{k}-\xi^{k-1}\|^{2}. We have provided three concrete choices of u^{k} as discussed above, but any direction u^{k} satisfying this condition still works. Remark 2 In fact, we can relax our condition ( 21 ) to the following one: \|Fx^{k}-u^{k}\|^{2}\leq\kappa\|Fx^{k}-Fy^{k-1}\|^{2}+\hat{\kappa}\|w^{k}-w^{k-1}\|^{2}+\bar{\kappa}\|\hat{w}^{k}-(1-\tau_{k-1})w^{k-1}\|^{2}, for given constants \kappa,\hat{\kappa},\bar{\kappa}\in\mathbb{R}_{+} . Then, our analysis bellow still goes through. Nevertheless, analyzing this case is relatively involved as we need to process an additional parameter \bar{\kappa} . Hence, we do not include it in this paper. 4.2 Key Estimates for Convergence Analysis \mathrm{(a)} Lyapunov function. Similar to Section 3 , we will use the following Lyapunov function to analyze the convergence of ( GFEG ): \begin{array}[]{lcl}\mathcal{L}_{k}&:=&\frac{a_{k}}{2}\|w^{k}\|^{2}+b_{k}\langle w^{k},x^{k}-x^{0}\rangle+\frac{c_{k}}{2}\|u^{k}-Fx^{k}\|^{2},\end{array} (25) where a_{k} , b_{k} , and c_{k} are given nonnegative parameters, determined later. Compared to \mathcal{L}_{k} in ( 11 ) of Section 3 , we drop the last term \frac{b_{0}}{\eta}\|x^{0}-x^{\star}\|^{2} here. \mathrm{(b)} Key lemma. The following lemma provides a key step to analyze the convergence of ( GFEG ). Lemma 2 For ( NI ), suppose that F is L -Lipschitz continuous, J_{\eta T} is well-defined, and \Phi is \rho -co-hypomonotone. Let \{(x^{k},y^{k})\} be generated by ( GFEG ) starting from x^{0}\in\mathrm{dom}(\Phi) such that u^{k} satisfies ( 21 ). For a fixed \eta>0 and a given b_{k}>0 , suppose that \begin{array}[]{lcl}\tau_{k}\in(0,1),\quad\eta_{k}:=\eta(1-\tau_{k}),\quad\text{and}\quad b_{k+1}:=\frac{b_{k}}{1-\tau_{k}}.\end{array} (26) For any \mu\in[0,1] , r>0 , and \gamma>0 , let us denote \hskip-12.91663pt\left\{\begin{array}[]{lclllcl}a_{k+1}&:=&\frac{b_{k+1}[\eta-\beta_{k}(1+\gamma\tau_{k})]}{\tau_{k}},&&\hat{a}_{k}&:=&\frac{b_{k}[\eta(1-\tau_{k})^{2}-\beta_{k}(1-2\tau_{k}+\gamma\tau_{k}(1-\tau_{k}))]}{\tau_{k}(1-\tau_{k})},\\ \alpha_{k+1}&:=&\frac{(1-\mu)\eta b_{k+1}}{\mu\tau_{k}},&&\hat{\alpha}_{k+1}&:=&\frac{b_{k+1}[\beta_{k}(1-\gamma+\gamma\tau_{k})-2\rho(1-\tau_{k})]}{\tau_{k}},\\ \delta_{k+1}&:=&\frac{\eta b_{k+1}[\mu-(1+r)L^{2}\eta^{2}]}{\mu\tau_{k}}&\quad\text{and}&c_{k}&:=&\frac{b_{k}[(1+r)L^{2}\eta^{3}\gamma(1-\tau_{k})^{2}+r\mu\beta_{k}]}{r\mu\gamma\tau_{k}(1-\tau_{k})}.\end{array}\right.\hskip-25.83325pt (27) Then the function \hskip-8.61108pt\begin{array}[]{lcl}\mathcal{P}_{k}&:=&\frac{\hat{a}_{k}}{2}\|w^{k}\|^{2}+b_{k}\langle w^{k},x^{k}-x^{0}\rangle+\frac{c_{k}}{2}\|Fx^{k}-u^{k}\|^{2}\end{array}\hskip-8.61108pt (28) satisfies the following property: \hskip-8.61108pt\begin{array}[]{lcl}\mathcal{P}_{k}&\geq&\frac{a_{k+1}}{2}\|w^{k+1}\|^{2}+b_{k+1}\langle w^{k+1},x^{k+1}-x^{0}\rangle+\frac{\alpha_{k+1}}{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\frac{\hat{\alpha}_{k+1}}{2}\|w^{k+1}-w^{k}\|^{2}+\frac{\delta_{k+1}}{2}\|\hat{w}^{k+1}-(1-\tau_{k})w^{k}\|^{2}.\end{array}\hskip-8.61108pt (29) \mathrm{(c)} Descent property. Next, we establish a descent property of \mathcal{L}_{k} in ( 25 ). Lemma 3 Under the same settings as in Lemma 2 , assume that a_{k} , \hat{a}_{k} , b_{k} , \alpha_{k} , \hat{\alpha}_{k} , c_{k} , and \delta_{k} given in ( 27 ) satisfy: \begin{array}[]{lcl}a_{k}\geq\hat{a}_{k}\geq 0,\quad\alpha_{k}\geq\kappa c_{k},\quad\text{and}\quad\hat{\alpha}_{k}\geq\hat{\kappa}c_{k}.\end{array} (30) Then, \mathcal{L}_{k} defined by ( 25 ) satisfies the following inequality: \begin{array}[]{lcl}\mathcal{L}_{k}\geq\mathcal{L}_{k+1}+\frac{\delta_{k+1}}{2}\|\hat{w}^{k+1}-(1-\tau_{k})w^{k}\|^{2}.\end{array} (31) Lemma 4 Under the same settings as in Lemma 3 , suppose that for a fixed \nu>2 , \tau_{k} and \beta_{k} are chosen as follows: \begin{array}[]{lcl}\tau_{k}:=\frac{1}{k+\nu}\quad\text{and}\quad{\beta_{k}:=\beta(1-\tau_{k})},\end{array} (32) where \beta satisfies three conditions for given r>0 , \gamma>0 , and \mu\in[0,1] : \left\{\begin{array}[]{lcl}\beta&\geq&\frac{(\nu-2)\gamma}{(\nu-2)\gamma(1-\gamma)-\nu\hat{\kappa}}\Big[\frac{(\nu-1)(1+r)\hat{\kappa}L^{2}\eta^{3}}{(\nu-2)r\mu}+2\rho\Big],\\ \beta&\leq&\frac{(\nu-1)\gamma\eta}{\nu\kappa\mu}\Big[1-\mu-\frac{(1+r)\kappa L^{2}\eta^{2}}{r}\Big],\\ \beta&\leq&\frac{\nu\eta}{\nu+\gamma}.\end{array}\right. (33) provided that the right-hand sides of these conditions are well-defined. Suppose further that ( 33 ) holds for a given choice of r , \mu , and \gamma . Then, the parameters \tau_{k} , \eta , \hat{\eta}_{k} , and \beta_{k} satisfy ( 30 ), and thus ( 31 ) still holds. 4.3 Convergence Guarantees of ( GFEG ) and Its Special Cases \mathrm{(a)} The convergence of ( GFEG ). Now, we state the main convergence result of ( GFEG ) for a general direction u^{k} satisfying the condition ( 21 ). Theorem 4.1 For ( NI ), suppose that \mathrm{zer}(\Phi)\neq\emptyset , F is L -Lipschitz continuous, \Phi is \rho -co-hypomonotone, and J_{\eta T} is well-defined and single-valued. Let \{(x^{k},y^{k})\} be generated by ( GFEG ) such that u^{k} satisfies ( 21 ). Suppose further that \tau_{k} , \hat{\eta}_{k} , and \beta_{k} are updated as follows: \begin{array}[]{ll}&\tau_{k}:=\frac{1}{k+\nu},\quad\hat{\eta}_{k}:=\eta(1-\tau_{k}),\quad\text{and}\quad\beta_{k}:=\beta(1-\tau_{k}),\end{array} (34) where \nu>2 is given, and \eta and \beta are chosen as follows: \mathrm{(i)} If \kappa=0 and \hat{\kappa}=0 in ( 21 ), and 2L\rho\leq 1 , then we choose \begin{array}[]{lcl}0<\eta\leq\frac{1}{L}\quad\text{and}\quad 2\rho\leq\beta<\eta.\end{array} \mathrm{(ii)} If \kappa=0 and 0<\hat{\kappa}<\frac{\nu-2}{4\nu} in ( 21 ), and L\rho<\frac{(\nu-2-4\nu\hat{\kappa})\sigma_{1}}{12(\nu-2)} , then we choose \begin{array}[]{lcl}0<\eta\leq\frac{\sigma_{1}}{L}\quad\text{and}\quad\frac{2(\nu-2)}{\nu-2-4\nu\hat{\kappa}}\big[\frac{2(\nu-1)\hat{\kappa}\eta\sigma_{1}^{2}}{\nu-2}+2\rho\big]\leq\beta<\frac{2\eta}{3},\end{array} where \sigma_{1}^{2}:=\min\left\{\frac{1}{2},\frac{\nu-2-4\nu\hat{\kappa}}{12(\nu-1)\hat{\kappa}}\right\} . \mathrm{(iii)} If \kappa>0 and \hat{\kappa}=0 in ( 21 ), and L\rho\leq\frac{\nu-1}{32\nu\sqrt{\kappa+1}} , then we choose \begin{array}[]{lcl}0<\eta\leq\frac{1}{2L\sqrt{\kappa+1}}\quad\text{and}\quad 4\rho\leq\beta\leq\frac{(\nu-1)\eta}{4\nu}.\end{array} \mathrm{(iv)} If \kappa>0 and 0<\hat{\kappa}<\frac{\nu-2}{4\nu} in ( 21 ), and L\rho\leq\frac{(\nu-1)(\nu-2-4\nu\hat{\kappa})\sigma_{2}}{32\nu(\nu-2)} , then we choose \begin{array}[]{lcl}0<\eta\leq\frac{1}{2L\sqrt{\kappa+1}}\quad\text{and}\quad\frac{2(\nu-2)}{\nu-2-4\nu\hat{\kappa}}\big[\frac{2(\nu-1)(\kappa+1)\hat{\kappa}\eta\sigma_{2}^{2}}{\nu-2}+2\rho\big]\leq\beta\leq\frac{(\nu-1)\eta}{4\nu},\end{array} where \sigma_{2}^{2}:=\frac{1}{\kappa+1}\min\left\{\frac{1}{2},\frac{\nu-2-4\nu\hat{\kappa}}{32\nu\hat{\kappa}}\right\} . Then, in all cases, we have the following bound: \|Fx^{k}+\xi^{k}\|^{2}\leq\|Fx^{k}+\xi^{k}\|^{2}+\psi\|Fx^{k}-u^{k}\|^{2}\leq\frac{\mathcal{R}_{0}^{2}}{(k+\nu-1)^{2}}, (35) where \psi:=0 if \kappa=\hat{\kappa}=0 , and \psi:=\frac{8(L^{2}\eta^{3}+\beta)}{2\eta-3\beta} , otherwise; x^{\star}\in\mathrm{zer}(\Phi) , and \mathcal{R}_{0}^{2}:=\left\{\begin{array}[]{lll}&\frac{4(\nu-1)}{(\eta-\beta)^{2}}\|x^{0}-x^{\star}\|^{2}+\frac{4(\nu-1)[\eta(\nu-1)-\beta(\nu-2)]}{\eta-\beta}\|Fx^{0}+\xi^{0}\|^{2},&{\!\!\!\!}\textrm{for Case $\mathrm{(i)}$},\\ &\frac{8(\nu-1)}{(2\eta-3\beta)^{2}}\|x^{0}-x^{\star}\|^{2}+\frac{4[2\eta(\nu-1)^{2}-\beta(\nu-2)(2\nu-1)]}{2\eta-3\beta}\|Fx^{0}+\xi^{0}\|^{2},&~\textrm{otherwise}.\end{array}\right. Remark 3 Theorem 4.1 proves the \mathcal{O}\left(1/k^{2}\right) -last-iterate convergence rate of ( GFEG ) under the condition ( 21 ). While the parameter \kappa>0 is arbitrary, \hat{\kappa} must be in a certain range 0\leq\hat{\kappa}<\frac{\nu-2}{4\nu} . Letting \nu\to+\infty , we get 0\leq\hat{\kappa}<\frac{1}{4} . Moreover, we also obtain L\rho<\frac{1}{32\sqrt{2(\kappa+1)}}\cdot\min\big\{1,\frac{1}{4\sqrt{\hat{\kappa}}}\big\} when \kappa>0 . \mathrm{(b)} Two special instances. Now, we consider two special cases of ( GFEG ) when u^{k}:=Fx^{k} , corresponding to the extra-anchored-gradient (also called fast extragradient) method in [ 40 , 87 ] and u^{k}:=Fy^{k-1} , corresponding to the past extra-anchored gradient method in [ 79 ] . Corollary 2 For ( NI ), suppose that \mathrm{zer}(\Phi)\neq\emptyset , F is L -Lipschitz continuous, J_{\eta T} is singled-valued and well-defined, and \Phi is \rho -co-hypomonotone. Let \{(x^{k},y^{k})\} be generated by ( GFEG ) starting from x^{0}\in\mathrm{dom}(\Phi) and using u^{k}:=Fx^{k} . Suppose further that 2L\rho\leq 1 , and for any fixed \nu>2 , \eta and \beta are respectively chosen such that \eta\in(0,\frac{1}{L}] and \beta\in[2\rho,\eta) , and \begin{array}[]{ll}&\tau_{k}:=\frac{1}{k+\nu},\quad\hat{\eta}_{k}:=\eta(1-\tau_{k}),\quad\text{and}\quad\beta_{k}:=\beta(1-\tau_{k}).\end{array} (36) Then, the following result holds: \|Fx^{k}+\xi^{k}\|^{2}\leq\frac{4\hat{\mathcal{R}}_{0}^{2}}{(\eta-\beta)(k+\nu-1)^{2}}, (37) where \hat{\mathcal{R}}_{0}^{2}:=\frac{\nu-1}{\eta-\beta}\|x^{0}-x^{\star}\|^{2}+\big[\eta(\nu-1)^{2}-\beta(\nu-2)(\nu-1)\big]\|Fx^{0}+\xi^{0}\|^{2} . Proof Since u^{k}:=Fx^{k} , ( 21 ) holds with \kappa=\hat{\kappa}=0 . Applying Theorem 4.1 with \kappa=\hat{\kappa}=0 , we obtain ( 36 ) from ( 34 ), and ( 35 ) reduces to ( 37 ). \square Corollary 3 For ( NI ), suppose that \mathrm{zer}(\Phi)\neq\emptyset , F is L -Lipschitz continuous, J_{\eta T} is well-defined, and \Phi is \rho -co-hypomonotone. Let \{(x^{k},y^{k})\} be generated by ( GFEG ) starting from x^{0}\in\mathrm{dom}(\Phi) and y^{-1}:=x^{0} using u^{k}:=Fy^{k-1} . Suppose further that for a given \nu>2 , we have L\rho\leq\frac{\nu-1}{32\sqrt{2}\nu} , and \eta and \beta are respectively chosen such that \eta\in\left(0,\frac{1}{2\sqrt{2}L}\right] and \beta\in\left[4\rho,\frac{(\nu-1)\eta}{4\nu}\right] , and \begin{array}[]{ll}&\tau_{k}:=\frac{1}{k+\nu},\quad\hat{\eta}_{k}:=\eta(1-\tau_{k}),\quad\text{and}\quad\beta_{k}:=\beta(1-\tau_{k}).\end{array} (38) Then, the following result holds: \|Fx^{k}+\xi^{k}\|^{2}\leq\|Fx^{k}+\xi^{k}\|^{2}+\psi\cdot\|Fx^{k}-Fy^{k-1}\|^{2}\leq\frac{\tilde{\mathcal{R}}_{0}^{2}}{(k+\nu-1)^{2}}, (39) where \tilde{\mathcal{R}}_{0}^{2}:=\frac{8(\nu-1)}{(2\eta-3\beta)^{2}}\|x^{0}-x^{\star}\|^{2}+\frac{4[2\eta(\nu-1)^{2}-\beta(\nu-2)(2\nu-1)]}{2\eta-3\beta}\|Fx^{0}+\xi^{0}\|^{2} for x^{\star}\in\mathrm{zer}(\Phi) and \psi:=\frac{8(L^{2}\eta^{3}+\beta)}{2\eta-3\beta} . Proof Since u^{k}:=Fy^{k-1} , ( 21 ) holds with \kappa=1 and \hat{\kappa}=0 . Applying Theorem 4.1 with \kappa=1 and \hat{\kappa}=0 , we obtain the update rule ( 38 ) from ( 34 ), and ( 35 ) reduces to ( 39 ). \square Remark 4 The analysis of Theorem 4.1 is much more complicated than that of Theorem 3.1 due to the co-hypomonotonicity of \Phi . In addition, we have not tried to optimally select the parameters in our analysis. The ranges of \hat{\kappa} , L\rho , and \eta can be improved by refining our analysis, but we omit it here. 5 A Class of Moving Anchor FBFS Methods Motivation. The anchored EG-type methods developed in Sections 3 and 4 have at least three aspects that can be improved. • First, the fixed anchor point x^{0} is propagated over all the iterations. Although the coefficient \tau_{k} of x^{0} is decreasing when k is growing, this term is still slowing down the actual performance of the algorithm. • Second, the update rule \tau_{k}:=\frac{1}{k+\nu} of \tau_{k} is tight, and has less freedom to improve the practical performance of algorithm. • Third, we are unable to prove faster convergence rates (i.e. o\left(1/k\right) rates) and the convergence of the iterate sequences (see [ 21 ] ). Here, we explore the ideas of restarting strategies in optimization [ 59 ] and recently moving anchor trick in [ 1 ] to propose a class of moving anchor FBFS methods for solving ( NI ). Our method can resolve the above three aspects. 5.1 A Class of Moving Anchor FBFS Methods First, for given x^{k}\in\mathrm{dom}(\Phi) and u^{k}\in\mathbb{R}^{p} , we recall the following elements: w^{k}:=Fx^{k}+\xi^{k},\quad\hat{w}^{k}:=Fy^{k-1}+\xi^{k},\quad\text{and}\quad z^{k}:=u^{k}+\xi^{k}, (40) for some \xi^{k}\in Tx^{k} . \mathrm{(a)} The proposed method. Instead of fixing x^{0} , we update x^{0} by a forward step similar to those in [ 1 , 88 ] , but using the direction z^{k} , not w^{k+1} . More specifically, our algorithm is described as follows. Starting from x^{0}\in\mathrm{dom}(\Phi) , we set \bar{x}^{0}:=x^{0} , and at each iteration k\geq 0 , for given x^{k} and u^{k} , we update \hskip-12.91663pt\left\{\begin{array}[]{lcl}y^{k}&:=&x^{k}+\tau_{k}(\bar{x}^{k}-x^{k})-(\hat{\eta}_{k}-\beta_{k})z^{k},\\ x^{k+1}&:=&x^{k}+\tau_{k}(\bar{x}^{k}-x^{k})-\eta\hat{w}^{k+1}+\beta_{k}z^{k},\\ \bar{x}^{k+1}&:=&\bar{x}^{k}-\gamma z^{k},\end{array}\right.\hskip-12.91663pt (GFEG + ) where \tau_{k}:=\frac{1}{t_{k}}\in(0,1] for t_{k}>1 , \eta>0 , \hat{\eta}_{k}>0 , \beta_{k}\geq 0 , and \gamma>0 are given parameters, determined later. The direction u^{k} in z^{k} from ( 40 ) is chosen such that it satisfies the following condition: \begin{array}[]{lcl}\|Fx^{k}-u^{k}\|^{2}&\leq&\kappa\|Fx^{k}-Fy^{k-1}\|^{2}+\hat{\kappa}\eta^{2}\|\hat{w}^{k}-(1-\tau_{k-1})w^{k-1}\|^{2},\end{array} (41) for given constants \kappa\geq 0 and \hat{\kappa}\geq 0 , y^{-1}=x^{-1}:=x^{0} , and u^{0}:=Fx^{0} . The scheme ( GFEG+ ) looks similar to ( GFEG ), but it has two main differences. First, \bar{x}^{k} is updated by \bar{x}^{k+1}:=\bar{x}^{k}-\gamma z^{k} instead of fixing at \bar{x}^{k}=x^{0} . Second, the parameters \tau_{k} and \beta_{k} are updated differently. \mathrm{(b)} Three special instances. Similar to ( GFEG ), our scheme ( GFEG+ ) covers a class of methods by choosing different direction u^{k} . Again, we consider at least three special instances of ( GFEG+ ). \mathrm{(i)} Variant 1: Moving anchor FBFS method. If we choose u^{k}:=Fx^{k} , then ( 41 ) holds with \kappa=0 and \hat{\kappa}=0 . This variant is similar to the moving anchor extragradient (EAG-V) scheme in [ 1 ] when T=0 as well as the symplectic extragradient method in [ 88 ] . However, our last line \bar{x}^{k+1}:=\bar{x}^{k}-\gamma z^{k} of ( GFEG+ ) is different from \bar{x}^{k+1}:=\bar{x}^{k}-\gamma w^{k+1} in [ 1 , 88 ] . Though this change is not essential in deterministic methods, it is very useful when developing randomized and stochastic variants of ( GFEG+ ). \mathrm{(ii)} Variant 2: Moving anchor past-FBFS method. If we choose u^{k}:=Fy^{k-1} , then ( 41 ) holds with \kappa=1 and \hat{\kappa}=0 , and ( GFEG+ ) reduces to \left\{\begin{array}[]{lcl}y^{k}&:=&\tau_{k}\bar{x}^{k}+(1-\tau_{k})x^{k}-(\hat{\eta}_{k}-\beta_{k})\hat{w}^{k}\\ x^{k+1}&:=&\tau_{k}\bar{x}^{k}+(1-\tau_{k})x^{k}-\eta\hat{w}^{k+1}+\beta_{k}\hat{w}^{k},\\ \bar{x}^{k+1}&:=&\bar{x}^{k}-\gamma\hat{w}^{k}.\end{array}\right.\vskip-2.15277pt (42) This scheme is new. It can be viewed as a variant of the forward-reflected-backward splitting, or an optimistic gradient method in the literature. One main advantage here is that it saves one evaluation of F per iteration. \mathrm{(iii)} Variant 3: Generalization. We can construct a generalized direction: \begin{array}[]{lcl}u^{k}&:=&\alpha Fx^{k}+(1-\alpha+{\color[rgb]{0,0,0}\eta}\hat{\alpha})Fy^{k-1}{\color[rgb]{0,0,0}-}{\color[rgb]{0,0,0}\eta}\hat{\alpha}(1-\tau_{k-1})Fx^{k-1}\\ &&+{~}{\color[rgb]{0,0,0}\eta}\hat{\alpha}[\xi^{k}-(1-\tau_{k-1})\xi^{k-1}],\end{array}\vskip-2.15277pt (43) for any given constants \alpha,\hat{\alpha}\in\mathbb{R} . Clearly, u^{k} is an affine combination of Fx^{k} , Fy^{k-1} , Fx^{k-1} , \xi^{k} , and \xi^{k-1} . Moreover, we can verify that u^{k} satisfies ( 41 ) with \kappa=(1+m)(1-\alpha)^{2} and \hat{\kappa}=(1+m^{-1})\hat{\alpha}^{2} by Young’s inequality for any m>0 . Generally, u^{k} requires Fx^{k} and/or Fy^{k-1} , and thus ( GFEG+ ) needs at most two evaluations of F at each iteration k . \mathrm{(c)} The implementation of ( GFEG+ ) . If the resolvent J_{\eta T} of \eta T is well-defined and single-valued, then we can rewrite ( GFEG+ ) equivalently to \left\{\begin{array}[]{lcl}y^{k}&:=&x^{k}+\tau_{k}(\bar{x}^{k}-x^{k})-(\hat{\eta}_{k}-\beta_{k})(u^{k}+\xi^{k}),\\ x^{k+1}&:=&J_{\eta T}\left(y^{k}-\eta Fy^{k}+\hat{\eta}_{k}(u^{k}+\xi^{k})\right),\\ \xi^{k+1}&:=&\frac{1}{\eta}\big[y^{k}+\hat{\eta}_{k}(u^{k}+\xi^{k})-x^{k+1}\big]-Fy^{k},\\ \bar{x}^{k+1}&:=&\bar{x}^{k}-\gamma(u^{k}+\xi^{k}),\end{array}\right. (44) where \xi^{0}\in Tx^{0} is arbitrary given, \bar{x}^{0}:=x^{0} , and u^{0}:=Fx^{0} . In fact, we only require J_{\eta T} to be well-defined and not necessarily single-valued. 5.2 Key Estimates for Convergence Analysis \mathrm{(a)} Lyapunov function. To establish the convergence of ( GFEG+ ), we define the following functions: \begin{array}[]{lcl}\mathcal{V}_{k}&:=&\frac{a_{k}}{2}\|w^{k}\|^{2}+t_{k-1}\langle w^{k},x^{k}-\bar{x}^{k}\rangle+\frac{(1-\mu)}{2\gamma}\|\bar{x}^{k}-x^{\star}\|^{2},\\ \mathcal{L}_{k}&:=&\mathcal{V}_{k}+\frac{\lambda\eta t_{k}^{2}}{2}\|u^{k}-Fx^{k}\|^{2},\end{array} (45) where x^{\star}\in\mathrm{zer}(\Phi) , a_{k} , t_{k} , and \lambda are nonnegative parameters, determined later. Note that \mathcal{V}_{k} is similar to the one in ( 45 ), but x^{0} is replaced by \bar{x}^{k} . Again, since we study the generalized scheme ( GFEG+ ), we need to add one more term \frac{\lambda\eta t_{k}^{2}}{2}\|u^{k}-Fx^{k}\|^{2} to \mathcal{V}_{k} to handle the mismatch between u^{k} and Fx^{k} . \mathrm{(b)} Descent property and boundedness. First, we state the following key lemma for our convergence analysis. Lemma 5 For ( NI ), suppose that F is L -Lipschitz continuous and \Phi is \rho -co-hypomonotone. Given \mu\in[0,1) and c_{1}\in\big[0,\frac{1}{2}\big) , let \{(x^{k},\bar{x}^{k},z^{k})\} be generated by ( GFEG+ ) starting from x^{0}\in\mathrm{dom}(\Phi) such that u^{k} satisfies ( 41 ). Suppose that the parameters are updated as \tau_{k}:=\frac{1}{t_{k}} and \begin{array}[]{lcl}t_{k}:=t_{k-1}+\mu,\quad\hat{\eta}_{k}:=\eta(1-\tau_{k}),\quad\textrm{and}\quad\beta_{k}:=-\gamma\tau_{k}+\frac{2\rho(1-\tau_{k})}{1-2c_{1}}.\end{array} (46) Given any \omega\geq 0 and c>0 , define M^{2}:=(1+\omega)(1+c)L^{2} and e^{k}:=u^{k}-Fx^{k} . Then \mathcal{V}_{k} defined by ( 45 ) satisfies \hskip-1.07639pt\begin{array}[]{lcl}\mathcal{V}_{k}&\geq&\mathcal{V}_{k+1}+\frac{(a_{k}-\hat{a}_{k})}{2}\|w^{k}\|^{2}+\frac{\omega\eta t_{k}^{2}}{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\frac{{\color[rgb]{0,0,0}\eta}(1-M^{2}\eta^{2})}{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}+(1-\mu)\langle w^{k},x^{k}-x^{\star}\rangle\\ &&+{~}(1-\mu)\langle e^{k},\bar{x}^{k}-x^{\star}\rangle-\frac{b_{k}}{2}\|e^{k}\|^{2},\end{array}\hskip-8.61108pt (47) where, for any c_{2}>0 and \hat{c}>0 , a_{k+1} , \hat{a}_{k} , and b_{k} are respectively given by \left\{\begin{array}[]{lcl}a_{k+1}&:=&t_{k}\big[\eta t_{k}-\frac{2\rho(t_{k}-1)}{1-2c_{1}}\big],\\ \hat{a}_{k}&:=&\eta(t_{k}-1)^{2}+2\gamma(1+c_{2})(t_{k}-1)-\frac{2\rho(t_{k}-1)[t_{k}-2(1+c_{2})]}{1-2c_{1}}\\ &&+{~}(1+\hat{c})(1-\mu)\gamma,\\ b_{k}&:=&\frac{M^{2}\eta^{3}(t_{k}-1)^{2}}{c}+\frac{(t_{k}-1)}{2(1-2c_{1})}\big[\frac{2\rho t_{k}}{c_{1}}+\frac{2\rho+\gamma(1-2c_{1})}{c_{2}}\big]+\frac{(1+\hat{c})(1-\mu)\gamma}{\hat{c}}.\end{array}\right. (48) We also need to show that \mathcal{V}_{k} in ( 45 ) is bounded from bellow. Lemma 6 Let \mathcal{V}_{k} be defined by ( 45 ) and \Phi be \rho -co-hypomonotone. Then, for any \tilde{c}>0 , we have \begin{array}[]{lcl}\mathcal{V}_{k}&\geq&\frac{[(1-\mu)\tilde{c}\eta-\gamma]}{2\tilde{c}\gamma\eta}\|\bar{x}^{k}-x^{\star}\|^{2}+\frac{(a_{k}-\tilde{c}\eta t_{k-1}^{2}-2\rho t_{k-1})}{2}\|w^{k}\|^{2}.\end{array} (49) 5.3 Convergence Analysis for The General Case \mathrm{(a)} The \mathcal{O}\big(1/k\big) convergence rates. First, we establish the convergence of ( GFEG+ ) for the general case when u^{k}\neq Fx^{k} . Then, we investigate a special case u^{k}:=Fx^{k} to expand the range of L\rho . Theorem 5.1 For ( NI ), suppose that \mathrm{zer}(\Phi)\neq\emptyset , F is L -Lipschitz continuous, and \Phi is \rho -co-hypomonotone. Let \{(x^{k},y^{k})\} be generated by ( GFEG+ ) such that u^{k} satisfies ( 41 ). Suppose further that \tau_{k} , \hat{\eta}_{k} , and \beta_{k} are updated as \begin{array}[]{ll}&\tau_{k}:=\frac{1}{t_{k}}=\frac{1}{\mu(k+r)},\quad\hat{\eta}_{k}:=\eta(1-\tau_{k}),\ \ \text{and}\ \ \beta_{k}:=-\gamma\tau_{k}+4\rho(1-\tau_{k}),\end{array} (50) where r\geq 1+\frac{2}{\mu} and \mu\in(0,\frac{1}{2}) are given, and \eta and \gamma are chosen such that \begin{array}[]{ll}8\rho<\eta\leq\bar{\eta}:=\frac{1}{\sqrt{2(1+4\kappa)L^{2}+4\hat{\kappa}}}\quad\textrm{and}\quad\gamma:=\frac{(1-2\mu)\eta}{6},\end{array} (51) provided that 128[(1+4\kappa)L^{2}+2\hat{\kappa}]\rho^{2}<1 . Then, we have \|Fx^{k}+\xi^{k}\|^{2}\leq\frac{6e^{\frac{\theta}{r}}\cdot\mathcal{R}_{0}^{2}}{\eta\mu^{2}(k+r-1)^{2}}, (52) where \theta:=\frac{(1-2\mu)(1-\mu)^{2}}{2{\color[rgb]{0,0,0}\nu}(2-\mu)\mu^{2}} and \mathcal{R}_{0}^{2}:=\frac{\eta\mu^{2}(r-1)^{2}}{2}\|Fx^{0}+\xi^{0}\|^{2}+\frac{3(1-\mu)}{(1-2\mu)\eta}\|x^{0}-x^{\star}\|^{2} . Moreover, we also have \begin{array}[]{lcl}\sum_{k=0}^{\infty}[\mu(k+r)-1]\|Fx^{k}+\xi^{k}\|^{2}&\leq&\frac{4\mathcal{R}_{0}^{2}}{(3-2\mu)(\eta-8\rho)}\big(1+\frac{\theta}{r}e^{\frac{\theta}{r}}\big).\end{array} (53) The choice of \eta and \gamma and the update rule of \beta_{k} can be improved to get a larger range by optimizing the constants in Lemma 5 . In Theorem 5.1 , we only provide one choice of these constants, which may not be the best one. For instance, if u^{k}:=Fx^{k} , then we have \kappa=\hat{\kappa}=0 , and the range of L\rho in Theorem 5.1 reduces to 8\sqrt{2}L\rho<1 , which is much worse than 2L\rho<1 stated in Theorem 5.3 below. Though Theorem 5.1 provides a unified result for all choices of u^{k} satisfying ( 41 ), for specific choice of u^{k} , it is still possible to improve the range of parameters. Let us consider the following examples. \mathrm{(i)} If u^{k}:=Fx^{k} , then we separate its analysis in Subsection 5.4 . \mathrm{(ii)} If u^{k}:=Fy^{k-1} , then \kappa=1 and \hat{\kappa}=0 , the range of L\rho reduces to 8\sqrt{10}L\rho<1 . Certainly, this range can be further improved by refining our analysis in Theorem 5.1 . \mathrm{(iii)} If \rho=0 , i.e. \Phi is monotone, then we can improve the range of \eta and \gamma respectively to, e.g., 0<\eta\leq\frac{1}{\sqrt{2(1+3\kappa)L^{2}+3\hat{\kappa}}} and \gamma:=\frac{(1-\mu)\eta}{3} . \mathrm{(b)} The o\big(1/k^{2}\big) Convergence Rates and Convergence of Iterates. Now, we prove o\big(1/k^{2}\big) convergence rates of ( GFEG+ ) and the convergence of iterate sequences to a solution of ( NI ). Theorem 5.2 Under the conditions and settings of Theorem 5.1 , we additionally assume that \eta<\bar{\eta} . Then \begin{array}[]{lcl}\sum_{k=0}^{\infty}(k+r)^{2}\|Fx^{k+1}-Fy^{k}\|^{2}&<&+\infty,\\ \sum_{k=0}^{\infty}(k+r)^{2}\|u^{k}-Fx^{k}\|^{2}&<&+\infty,\\ \sum_{k=0}^{\infty}(k+r)\|y^{k}-x^{k}\|^{2}&<&+\infty,\\ \sum_{k=0}^{\infty}(k+r)\|x^{k+1}-x^{k}\|^{2}&<&+\infty.\end{array} (54) We also have \begin{array}[]{lcl}\lim_{k\to\infty}k^{2}\|Fx^{k}+\xi^{k}\|^{2}&=&0,\\ {\color[rgb]{0,0,0}\lim_{k\to\infty}k^{2}\|y^{k}-x^{k}\|^{2}}&=&0,\\ {\color[rgb]{0,0,0}\lim_{k\to\infty}k^{2}\|x^{k+1}-x^{k}\|^{2}}&=&0.\end{array} (55) Moreover, if T is closed, then all three sequences \{x^{k}\} , \{y^{k}\} , and \{\bar{x}^{k}\} converge to a solution x^{\star}\in\mathrm{zer}(\Phi) of ( NI ). The limits in ( 55 ) show that \|Fx^{k}+\xi^{k}\|^{2}=o\big(1/k^{2}\big) , \|x^{k+1}-x^{k}\|^{2}=o\big(1/k^{2}\big) , and \|y^{k}-x^{k}\|^{2}=o\big(1/k^{2}\big) , stating a faster convergence rate than \mathcal{O}\big(1/k^{2}\big) in Theorem 5.1 . However, this faster rate is achieved when k is sufficiently large, while the \mathcal{O}\big(1/k^{2}\big) bound holds for all k\geq 0 . Since F is continuous, the closedness of T implies the closedness of \mathrm{gra}(\Phi) , which guarantees that if (x^{k},v^{k})\in\mathrm{gra}(\Phi) and (x^{k},v^{k}) converges to (x,v) , then (x,v)\in\mathrm{gra}(\Phi) . In particular, this condition holds if T is maximally monotone ( 9 , Proposition 20.38) . 5.4 Special Case – Moving Anchor Extragradient Method The choice u^{k}:=Fx^{k} leads to e^{k}:=u^{k}-Fx^{k}=0 , making our analysis simpler. Moreover, it allows us to expand the range of L\rho to 2L\rho<1 . Hence, we state it separately in this subsection. This variant is similar to the ones in [ 1 , 88 ] , but the update of \bar{x}^{k} uses w^{k} instead of w^{k+1} as in [ 1 , 88 ] . Theorem 5.3 For ( NI ), suppose that \mathrm{zer}(\Phi)\neq\emptyset , F is L -Lipschitz continuous, and \Phi is \rho -co-hypomonotone such that 2L\rho<1 . Let \{(x^{k},y^{k},\bar{x}^{k})\} be generated by ( GFEG+ ) using u^{k}:=Fx^{k} and \begin{array}[]{ll}&\tau_{k}:=\frac{1}{t_{k}}=\frac{1}{\mu(k+r)},\quad\hat{\eta}_{k}:=\eta(1-\tau_{k}),\ \ \text{and}\ \ \beta_{k}:=-\gamma\tau_{k}+2\rho(1-\tau_{k}),\end{array} (56) where r\geq\frac{1}{\mu} and \mu\in(0,1) are given, and \eta and \gamma are chosen such that \begin{array}[]{ll}2\rho<\eta\leq\frac{1}{L}\quad\textrm{and}\quad 0<\gamma<(1-\mu)(\eta-2\rho).\end{array} (57) Then, we have \begin{array}[]{ll}&\|Fx^{k}+\xi^{k}\|^{2}\leq\dfrac{{\color[rgb]{0,0,0}2}(1-\mu)\mathcal{R}_{0}^{2}}{[(1-\mu)(\eta-2\rho)-\gamma]\mu^{2}(k+r-1)^{2}},\\ &{\displaystyle\sum_{k=0}^{\infty}}{\color[rgb]{0,0,0}[2\mu(k+r)-1-\mu]}\|Fx^{k}+\xi^{k}\|^{2}\leq\dfrac{{\color[rgb]{0,0,0}2}\mathcal{R}_{0}^{2}}{{\color[rgb]{0,0,0}(1-\mu)(\eta-2\rho)-\gamma}},\end{array} (58) where \mathcal{R}_{0}^{2}:=\frac{\mu(r-1)[\mu(r-1)(\eta-2\rho)+2\rho]}{2}\|Fx^{0}+\xi^{0}\|^{2}+\frac{1-\mu}{2\gamma}\|x^{0}-x^{\star}\|^{2} . Similar to Theorem 5.2 , we can prove the summable results in ( 54 ), the o\left(1/k^{2}\right) convergence rates in ( 55 ), and the convergence of the iterate sequences \{x^{k}\} , \{y^{k}\} , and \{\bar{x}^{k}\} for this special case. However, we omit the details since it repeats the proof of Theorem 5.2 using Theorem 5.3 . 6 Nesterov’s Accelerated Extragradient Methods for Inclusions ( NI ) Sections 3 , 4 , and 5 have studied Halpern’s accelerated (or anchored) methods. Now, we will focus on Nesterov’s accelerated algorithms in the next two sections. In this section, we unify and generalize Nesterov’s accelerated extragradient method from [ 77 ] to a broader class of algorithms for solving ( NI ). 6.1 A Class of Nesterov’s Accelerated Extragradient Algorithms \mathrm{(a)} The proposed method. First, for given x^{k}\in\mathrm{dom}(\Phi) , u^{k}\in\mathbb{R}^{p} , and \xi^{k}\in Tx^{k} , we recall the following quantities from ( 20 ): w^{k}:=Fx^{k}+\xi^{k},\quad\hat{w}^{k}:=Fy^{k-1}+\xi^{k},\quad\text{and}\quad z^{k}:=u^{k}+\xi^{k}. (59) Next, starting from x^{0}\in\mathrm{dom}(\Phi) , let u^{0}:=Fx^{0} and y^{-1}=\hat{x}^{-1}=\hat{x}^{0}:=x^{0} . At each iteration k\geq 0 , we update the sequence \{(x^{k},\hat{x}^{k},y^{k})\} as \left\{\begin{array}[]{lcl}x^{k+1}&:=&y^{k}-\eta(\hat{w}^{k+1}-\gamma_{k}z^{k}),\\ \hat{x}^{k+1}&:=&x^{k+1}-\lambda z^{k+1},\\ y^{k+1}&:=&\hat{x}^{k+1}+\theta_{k}(\hat{x}^{k+1}-\hat{x}^{k})+\nu_{k}(y^{k}-\hat{x}^{k+1}),\end{array}\right. (GAEG) where \eta>0 , \gamma_{k}\in[0,1] , \lambda>0 , \theta_{k}\in(0,1) , and \nu_{k}\geq 0 are given parameters (determined later), and u^{k}\in\mathbb{R}^{p} is a search direction satisfying \|Fx^{k}-u^{k}\|^{2}\leq\kappa\|Fx^{k}-Fy^{k-1}\|^{2}+\hat{\kappa}\|w^{k}-w^{k-1}\|^{2}, (60) for given constants \kappa\geq 0 and \hat{\kappa}\geq 0 , and w^{-1}:=w^{0} . In particular, if T=0 , then w^{k}-w^{k-1}=Fx^{k}-Fx^{k-1} , and thus u^{k} depends on Fx^{k} , Fy^{k-1} , and Fx^{k-1} . Note that we allow \kappa\geq 0 to be arbitrary, but \hat{\kappa} will be chosen in a certain range ( cf. Theorem 6.1 ). \mathrm{(b)} Three special instances. As before, our method ( GAEG ) possibly covers a wide class of schemes by instantiating different directions u^{k} , which satisfy ( 60 ). We consider at least the following three special choices of u^{k} . \mathrm{(i)} Variant 1. If we choose u^{k}:=Fx^{k} , then z^{k}=w^{k} and ( 60 ) holds with \kappa=\hat{\kappa}=0 . In this case, ( GAEG ) reduces to \left\{\begin{array}[]{lcl}x^{k+1}&:=&y^{k}-\eta(\hat{w}^{k+1}-\gamma_{k}w^{k}),\\ \hat{x}^{k+1}&:=&x^{k+1}-\lambda w^{k+1},\\ y^{k+1}&:=&\hat{x}^{k+1}+\theta_{k}(\hat{x}^{k+1}-\hat{x}^{k})+\nu_{k}(y^{k}-\hat{x}^{k+1}).\end{array}\right. (61) Clearly, ( 61 ) is exactly the accelerated extragradient method in [ 77 ] . \mathrm{(ii)} Variant 2. If we choose u^{k}:=Fy^{k-1} , then z^{k}=\hat{w}^{k} and ( 60 ) holds with \kappa=1 and \hat{\kappa}=0 . Then, ( GAEG ) reduces to \left\{\begin{array}[]{lcl}x^{k+1}&:=&y^{k}-\eta(\hat{w}^{k+1}-\gamma_{k}\hat{w}^{k}),\\ \hat{x}^{k+1}&:=&x^{k+1}-\lambda\hat{w}^{k+1},\\ y^{k+1}&:=&\hat{x}^{k+1}+\theta_{k}(\hat{x}^{k+1}-\hat{x}^{k})+\nu_{k}(y^{k}-\hat{x}^{k+1}).\end{array}\right. (62) This variant seems to be new and it is different from the second algorithm in [ 77 ] . This method is also different from the one in [ 71 ] when specified to solve the monotone ( VIP ). However, we only obtain \mathcal{O}\left(1/k\right) convergence rate, while [ 71 ] can achieve both \mathcal{O}\left(1/k\right) and o\left(1/k\right) convergence rates. \mathrm{(iii)} Variant 3: Generalization. Let v^{k-1}:=w^{k-1}-\xi^{k} . We can construct \begin{array}[]{lcl}u^{k}:=\alpha Fx^{k}+\hat{\alpha}Fy^{k-1}+(1-\alpha-\hat{\alpha})v^{k-1}\end{array} as an affine combination of Fx^{k} , Fy^{k-1} and v^{k-1} for given constants \alpha,\hat{\alpha}\in\mathbb{R} . Then, by Young’s inequality, for any m>0 , one can show that \begin{array}[]{lcl}\|Fx^{k}-u^{k}\|^{2}&=&\|\hat{\alpha}(Fx^{k}-Fy^{k-1})+(1-\alpha-\hat{\alpha})(Fx^{k}-v^{k-1})\|^{2}\\ &\leq&\kappa\|Fx^{k}-Fy^{k-1}\|^{2}+\hat{\kappa}\|w^{k}-w^{k-1}\|^{2},\end{array} where \kappa:=(1+m)\hat{\alpha}^{2} and \hat{\kappa}:=(1\!+\!m^{-1})(1\!-\!\alpha\!-\!\hat{\alpha})^{2} . Thus u^{k} satisfies ( 60 ). \mathrm{(c)} The implementation of ( GAEG ). To implement ( GAEG ), we assume that the resolvent J_{\eta T} of \eta T is well-defined (see Section 3 ) for any \eta>0 . Generally, we do not require J_{\eta T} to be single-valued, but to avoid complicating our presentation, we again assume that J_{\eta T} is single-valued. Using the resolvent J_{\eta T} , we can rewrite ( GAEG ) equivalently to \left\{\begin{array}[]{lcl}x^{k+1}&:=&J_{\eta T}\big(y^{k}-\eta(Fy^{k}-\gamma_{k}(u^{k}+\xi^{k}))\big),\\ \xi^{k+1}&:=&\frac{1}{\eta}(y^{k}-x^{k+1})-Fy^{k}+\gamma_{k}(u^{k}+\xi^{k}),\\ \hat{x}^{k+1}&:=&x^{k+1}-\lambda(u^{k+1}+\xi^{k+1}),\\ y^{k+1}&:=&\hat{x}^{k+1}+\theta_{k}(\hat{x}^{k+1}-\hat{x}^{k})+\nu_{k}(y^{k}-\hat{x}^{k+1}).\end{array}\right. (63) This scheme requires one evaluation Fy^{k} of F , one evaluation J_{\eta T} of \eta T , and one construction of u^{k} . Under the condition ( 60 ), constructing u^{k} may require at most one more evaluation of F . In particular, if u^{k}:=Fx^{k} , then we need one more evaluation Fx^{k} of F . If u^{k}:=Fy^{k-1} , then we can reuse the value Fy^{k-1} evaluated from the previous iteration k-1 . 6.2 Key Estimates for Convergence Analysis \mathrm{(a)} Key lemma. To analyze ( GAEG ), we first state the following lemma. Lemma 7 For ( NI ), suppose that \Phi is \rho -co-hypomonotone, F is L -Lipschitz continuous, and \mathrm{zer}(\Phi)\neq\emptyset . Given \lambda>0 , \hat{c}>0 , \beta>0 , \mu\geq 0 , and t_{0}>0 , let \{(x^{k},\hat{x}^{k},y^{k})\} be generated by ( GAEG ) using the following parameters: \begin{array}[]{ll}&\eta:=(1+2\hat{c})\lambda+2\rho+2\beta+2\mu,\quad\quad t_{k+1}:=t_{k}+1,\\ &\gamma_{k}:=\frac{t_{k}-1}{t_{k}},\quad\theta_{k}:=\frac{t_{k}-1}{t_{k+1}},\quad\text{ and }\quad\nu_{k}:=\frac{t_{k}}{t_{k+1}}.\end{array} (64) For any x^{\star}\in\mathrm{zer}(\Phi) , we consider the following function: \begin{array}[]{lcl}\mathcal{P}_{k}&:=&\frac{a_{k}}{2}\|w^{k}\|^{2}+b_{k}\langle w^{k},\hat{x}^{k}-y^{k}\rangle+\|\hat{x}^{k}-x^{\star}+t_{k}(y^{k}-\hat{x}^{k})\|^{2},\end{array} (65) where a_{k}:=\frac{b_{k}}{t_{k}}\big[\lambda t_{k}+(1+2\hat{c})\lambda+2\rho+2\mu\big] and b_{k+1}:=\frac{b_{k}}{\theta_{k}} for b_{k}>0 . Then, for any c>0 and M_{c}:=(1+c)L^{2} , we have \begin{array}[]{lcl}\mathcal{P}_{k}-\mathcal{P}_{k+1}&\geq&\mu b_{k}\|w^{k+1}-w^{k}\|^{2}+\frac{b_{k}(1-M_{c}\eta^{2})}{2M_{c}\eta\gamma_{k}}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&-{~}\frac{b_{k}}{2}\big(\frac{\lambda}{\hat{c}}+\frac{\eta^{2}\gamma_{k}}{\beta}+\frac{\eta\gamma_{k}}{c}\big)\|u^{k}-Fx^{k}\|^{2}\\ &&-{~}\frac{b_{k}}{2}\big(\frac{\lambda}{\hat{c}}+\frac{\lambda^{2}}{\beta\gamma_{k}}\big)\|u^{k+1}-Fx^{k+1}\|^{2}.\end{array} (66) \mathrm{(b)} Lyapunov function and descent property. Next, for given \mathcal{P}_{k} defined in ( GAEG ) we consider the following Lyapunov function: \begin{array}[]{lcl}\mathcal{L}_{k}&:=&\mathcal{P}_{k}+\frac{c_{k}}{2}\|u^{k}-Fx^{k}\|^{2}\\ &=&\frac{a_{k}}{2}\|w^{k}\|^{2}+b_{k}\langle w^{k},\hat{x}^{k}-y^{k}\rangle+\|\hat{x}^{k}-x^{\star}+t_{k}(y^{k}-\hat{x}^{k})\|^{2}\\ &&+{~}\frac{c_{k}}{2}\|u^{k}-Fx^{k}\|^{2},\end{array} (67) where a_{k}:=\frac{b_{k}}{t_{k}}\left[\lambda t_{k}+(1+2\hat{c})\lambda+2\rho+2\mu\right] given in Lemma 7 , b_{k+1}:=\frac{b_{k}}{\theta_{k}} for b_{k}>0 , and c_{k}:=b_{k}\big(\frac{\lambda}{\hat{c}}+\frac{\eta^{2}\gamma_{k}}{\beta}+\frac{\eta\gamma_{k}}{c}\big) . Then, we can prove the following descent property of \mathcal{L}_{k} . Lemma 8 Under the same settings as in Lemma 7 , if u^{k} satisfies ( 60 ) and t_{0}>1 , then \mathcal{L}_{k} defined by ( 67 ) satisfies \hskip-4.30554pt\begin{array}[]{lcl}\mathcal{L}_{k}-\mathcal{L}_{k+1}&\geq&\frac{b_{k}t_{k}}{2t_{k-1}}\big[\frac{1}{M_{c}\eta}-\eta-\kappa\big(\frac{2\lambda}{\hat{c}}+\frac{\lambda^{2}}{\beta}+\frac{\eta^{2}}{\beta}+\frac{\eta}{c}\big)\big]\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\frac{b_{k}t_{k}}{2t_{k-1}}\big[\frac{2\mu(t_{0}-1)}{t_{0}}-\hat{\kappa}\big(\frac{2\lambda}{\hat{c}}+\frac{\lambda^{2}}{\beta}+\frac{\eta^{2}}{\beta}+\frac{\eta}{c}\big)\big]\|w^{k+1}-w^{k}\|^{2}.\end{array} (68) \mathrm{(c)} Lower bound of \mathcal{L}_{k} . Finally, we can lower bound \mathcal{L}_{k} as follows. Lemma 9 Under the same settings as in Lemma 8 and b_{0}:=\frac{3\lambda t_{0}(t_{0}-1)}{2} , \mathcal{L}_{k} defined by ( 67 ) is lower bounded by \begin{array}[]{lcl}\mathcal{L}_{k}&\geq&\frac{b_{k}}{2t_{k}}\big[\frac{\lambda t_{k}}{4}+(2\hat{c}-\frac{1}{2})\lambda+2\mu\big]\|w^{k}\|^{2}\\ &&+{~}\frac{b_{k}}{2t_{k}}\big[\frac{\lambda}{\hat{c}}(t_{k}-4\hat{c})+(\frac{\eta}{\beta}+\frac{1}{c})\eta t_{k-1}\big]\|u^{k}-Fx^{k}\|^{2}.\end{array} (69) 6.3 Convergence Guarantees of ( GAEG ) and Its Special Cases \mathrm{(a)} The convergence of ( GAEG ). Given \kappa\geq 0 , \sigma:=\frac{5}{24} , and L and \rho such that 4(1+\sigma)\sqrt{2(5\kappa+1)}L\rho<1 , we define \hskip-12.91663pt\begin{array}[]{ll}\bar{\lambda}:=\frac{2\bar{c}}{\bar{b}+\sqrt{\bar{b}^{2}+4\bar{c}}},\ \ \text{where}\ \ \bar{b}:=\frac{(1+\sigma)(49\kappa+8)\rho}{29\kappa+4}\ \textrm{and}\ \bar{c}:=\frac{1-32(1+\sigma)^{2}(5\kappa+1)L^{2}\rho^{2}}{8L^{2}(29\kappa+4)}.\end{array}\hskip-12.91663pt (70) Now, we are ready to prove the convergence of ( GAEG ) as follows. Theorem 6.1 For ( NI ), suppose that \mathrm{zer}(\Phi)\neq\emptyset , F is L -Lipschitz continuous, \Phi is \rho -co-hypomonotone, and J_{\eta T} is well-defined. Let \{(x^{k},\hat{x}^{k},y^{k})\} be generated by ( GAEG ) using u^{k} satisfying ( 60 ) with \kappa\geq 0 and \hat{\kappa}\geq 0 . Given r>1 and \bar{\lambda} defined in ( 70 ), suppose that \hskip-8.61108pt\left\{\begin{array}[]{ll}&0\leq\hat{\kappa}\leq\frac{r-1}{58r},\ \quad L\rho<\frac{6}{29\sqrt{2(5\kappa+1)}},\ \quad 0<\lambda\leq\bar{\lambda},\ \quad\eta:=4\lambda+\frac{29}{6}\rho,\\ &t_{k}:=k+r,\qquad\gamma_{k}:=\frac{t_{k}-1}{t_{k}},\qquad\theta_{k}:=\frac{t_{k}-1}{t_{k+1}},\quad\textrm{ and }\quad\nu_{k}:=\frac{t_{k}}{t_{k+1}}.\end{array}\right.\hskip-8.61108pt (71) Then, for any x^{\star}\in\mathrm{zer}(\Phi) , the following bound holds: \|Fx^{k}+\xi^{k}\|^{2}\leq\frac{\mathcal{R}_{0}^{2}}{3\lambda^{2}(k+r-1)(k+r+2)}, (72) where \mathcal{R}_{0}^{2}:=\lambda(r-1)\big[12(r+2)\lambda+29\rho\big]\|Fx^{0}+\xi^{0}\|^{2}+16\|x^{0}-x^{\star}\|^{2} . Remark 5 The momentum parameter \theta_{k} in ( 71 ) is exactly \theta_{k}=\frac{k+r-1}{k+r+1} , aligned with the one in standard Nesterov’s accelerated methods. However, this choice does not allow us to prove faster convergence rates (i.e. o\left(1/k\right) rates) as well as the convergence of iterate sequences as recognized in [ 5 , 21 ] . \mathrm{(b)} Special instances. Finally, we consider two special cases of ( GAEG ) corresponding to u^{k}=Fx^{k} and u^{k}=Fy^{k-1} . The proof of this result is similar to Theorem 6.1 , and we provide its details in the appendix. Corollary 4 For ( NI ), suppose that \mathrm{zer}(\Phi)\neq\emptyset , F is L -Lipschitz continuous, \Phi is \rho -co-hypomonotone, and J_{\eta T} is well-defined. Let \{(x^{k},\hat{x}^{k},y^{k})\} be generated by ( GAEG ). \mathrm{(i)} If we choose u^{k}:=Fx^{k} , then we require 2L\rho<1 , and choose 0<\lambda\leq\frac{1}{L}-2\rho , \eta:=\lambda+2\rho , and other parameters as in ( 71 ) with r>2 . Then \|Fx^{k}+\xi^{k}\|^{2}\leq\frac{\mathcal{R}_{0}^{2}}{\lambda^{2}(k+r-2)(k+r-1)}, (73) where \mathcal{R}_{0}^{2}:=4\lambda(r-1)[(r+1)\lambda+2\rho]\|Fx^{0}+\xi^{0}\|^{2}+\frac{16}{3}\|x^{0}-x^{\star}\|^{2} . \mathrm{(ii)} If we choose u^{k}:=Fy^{k-1} , then we require 8\sqrt{3}L\rho<1 , and choose 0<\lambda\leq\bar{\lambda} , \eta:=3\lambda+4\rho , and other parameters as in ( 71 ), where \hskip-4.30554pt\begin{array}[]{lcl}\bar{\lambda}:=\frac{2\bar{c}}{\bar{b}+\sqrt{\bar{b}^{2}+4\bar{c}}}\quad\text{with}\quad\bar{b}:=\frac{272\rho}{123}\geq 0\quad\textrm{and}\quad\bar{c}:=\frac{1-129L^{2}\rho^{2}}{164L^{2}}>0.\end{array} In particular, if \rho=0 ( i.e. \Phi is monotone ) , then we choose 0<\lambda\leq\bar{\lambda}:=\frac{1}{2\sqrt{41}L} and \eta:=3\lambda . Then, the following bound holds: \|Fx^{k}+\xi^{k}\|^{2}\leq\frac{\mathcal{R}_{0}^{2}}{\lambda^{2}(k+r-1)(k+r)}, (74) where \mathcal{R}_{0}^{2}:=2\lambda(r-1)[(2r+3)\lambda+4\rho]\|Fx^{0}+\xi^{0}\|^{2}+\frac{16}{3}\|x^{0}-x^{\star}\|^{2} . Remark 6 The range of L\rho in Theorem 6.1 and in Part (b) of Corollary 4 can potentially be improved by properly choosing parameters c , \hat{c} , \beta , and \mu in our proof. Here, we have not tried to optimize these ranges of L\rho , \lambda , and \eta . 7 New Accelerated EG-Type Methods for ( NI ): Better Guarantees In this section, we develop a novel class of generalized Nesterov’s accelerated extragradient methods to solve ( NI ), which can achieve faster convergence rates than the one in Section 6 . In addition, we can prove the convergence of the iterate sequences, which is still open in Section 6 . Note that this framework is different from the moving anchor FBFS method in Section 5 as it relies on Nesterov’s acceleration techniques. A detailed comparison between two types of methods can be found in [ 82 ] . 7.1 A Class of Nesterov’s Accelerated Extragradient Algorithms \mathrm{(a)} The proposed method. First, for given x^{k}\in\mathrm{dom}(\Phi) , u^{k}\in\mathbb{R}^{p} , and \xi^{k}\in Tx^{k} , we recall the following quantities from the previous sections: w^{k}:=Fx^{k}+\xi^{k},\quad\hat{w}^{k}:=Fy^{k-1}+\xi^{k},\quad\text{and}\quad z^{k}:=u^{k}+\xi^{k}. (75) Next, starting from x^{0}\in\mathrm{dom}(\Phi) , we set u^{0}:=Fx^{0} and y^{-1}=y^{0}:=x^{0} , and at each iteration k\geq 0 , we update the iterate sequence \{(x^{k},y^{k})\} as \left\{\begin{array}[]{lcl}x^{k+1}&:=&y^{k}-\eta d^{k},\\ y^{k+1}&:=&x^{k+1}+\theta_{k}(x^{k+1}-x^{k})-p^{k},\end{array}\right. (GAEG + ) where d^{k} and p^{k} are respectively defined by \left\{\begin{array}[]{lcl}d^{k}&:=&\hat{w}^{k+1}-\gamma_{k}z^{k},\\ p^{k}&:=&\eta_{k}z^{k+1}-\lambda_{k}\hat{w}^{k+1}+\nu_{k}z^{k}.\end{array}\right. (76) The involved parameters in ( GAEG+ ) and ( 76 ) are updated by \begin{array}[]{ll}&t_{k+1}:=t_{k}+1,\qquad\theta_{k}:=\frac{t_{k}-r-\mu}{t_{k+1}},\qquad\gamma_{k}:=\frac{t_{k}-r+1}{t_{k}},\\ &\eta_{k}:=\frac{(\eta-\beta)t_{k}-\delta}{t_{k+1}},\qquad\lambda_{k}:=\frac{\eta t_{k}}{t_{k+1}},\quad\textrm{and}\quad\nu_{k}:=\frac{\beta t_{k}}{t_{k+1}},\end{array} (77) where \eta>\beta\geq 0 , r>0 , and \mu>0 are determined later and \delta:=(r-1)\beta+\frac{(r-2)(\eta-\beta)}{\mu+1} . Moreover, the direction u^{k} in ( 75 ) satisfies the following condition: \|Fx^{k}-u^{k}\|^{2}\leq\kappa\|Fx^{k}-Fy^{k-1}\|^{2}+\hat{\kappa}\|d^{k-1}\|^{2}, (78) for given constants \kappa\geq 0 and \hat{\kappa}\geq 0 , and d^{-1}:=0 . Here, we allow \kappa to be arbitrary, but we require \hat{\kappa} to be sufficiently small, determined later. Remark 7 At first glance, one may question: why are the parameters updated as in ( 77 )? Let us explain. The choice of \theta_{k} is suggested by Nesterov’s accelerated methods in convex optimization and monotone inclusions, which is often of the form \theta_{k}=\frac{k-s}{k+2} for s\geq 0 . The direction d^{k} can be written as d^{k}=\big(1-\frac{1}{\gamma_{k}}\big)\hat{w}^{k+1}+\gamma_{k}(\hat{w}^{k+1}-z^{k}) , which can be viewed as a generalization of the forward-reflected-backward splitting step. Here, \gamma_{k}(\hat{w}^{k+1}-z^{k}) can be seen as an extrapolation or a correction term. The parameters \eta_{k} , \lambda_{k} , and \nu_{k} in p^{k} satisfy \eta_{k}-\lambda_{k}+\nu_{k}=\frac{\delta}{t_{k+1}} as often seen in Nesterov’s accelerated methods for monotone inclusions, see, e.g., [ 3 , 6 , 15 , 16 , 37 , 46 , 47 ] . However, the specific form of parameters in ( 77 ) comes from the derivations of our convergence analysis, when we enforce a descent direction of an appropriate Lyapunov function (defined later). This step is rather technical, and we only provide the update rule ( 77 ) instead of presenting the detailed derivations in this paper. Remark 8 Note that ( GAEG ) studied in Section 6 can be written into the form ( GAEG+ ). However, the choice of parameters is significantly different. \mathrm{(b)} Three special instances. The generalized method ( GAEG+ ) possibly covers a wide class of schemes by instantiating different directions u^{k} , which satisfy ( 78 ). As before, we consider the following three special choices of u^{k} . \mathrm{(i)} Variant 1: EG-Type Variant. If we choose u^{k}:=Fx^{k} , then ( 78 ) automatically holds with \kappa=\hat{\kappa}=0 . Thus ( GAEG+ ) reduces to \left\{\begin{array}[]{lcl}x^{k+1}&:=&y^{k}-\eta(\hat{w}^{k+1}-\gamma_{k}w^{k}),\\ y^{k}&:=&x^{k+1}+\theta_{k}(x^{k+1}-x^{k})-\eta_{k}w^{k+1}+\lambda_{k}\hat{w}^{k+1}-\nu_{k}w^{k},\end{array}\right. (AEG) where \gamma_{k} , \theta_{k} , \eta_{k} , \lambda_{k} , and \nu_{k} are updated as in ( 77 ). Clearly, ( AEG ) can be viewed as an accelerated extragradient (AEG) method for solving ( NI ). Note that our method ( AEG ) is different from existing works such as [ 88 ] due to both the update rule and the choice of parameters. As in any EG method, ( AEG ) requires two evaluations Fx^{k} and Fy^{k} of F , but we only need one resolvent J_{\eta T} of T at each iteration. \mathrm{(ii)} Variant 2: Past-EG-Type Variant. If we choose u^{k}:=Fy^{k-1} , then ( 78 ) holds with \kappa=1 and \hat{\kappa}=0 . Clearly, ( GAEG+ ) reduces to the following variant: \left\{\begin{array}[]{lcl}x^{k+1}&:=&y^{k}-\eta(\hat{w}^{k+1}-\gamma_{k}\hat{w}^{k}),\\ y^{k}&:=&x^{k+1}+\theta_{k}(x^{k+1}-x^{k})-(\eta_{k}-\lambda_{k})\hat{w}^{k+1}-\nu_{k}\hat{w}^{k}.\end{array}\right. (OG) This scheme can be viewed as an accelerated [Popov’s] past-extragradient (or equivalently, optimistic gradient) method for solving ( NI ). Unlike ( AEG ), ( OG ) only requires one evaluation Fy^{k} and one resolvent J_{\eta T} of T at each iteration, saving one evaluation of F . \mathrm{(iii)} Variant 3: Generalization. We can construct u^{k} as follows: u^{k}:=\hat{\alpha}d^{k-1}+(1-\alpha)Fx^{k}+\alpha Fy^{k-1}. (79) Clearly, u^{k} is a linear combination of Fx^{k} , Fy^{k-1} and d^{k-1} for \alpha,\hat{\alpha}\in\mathbb{R} . Then, u^{k} satisfies ( 78 ) with \kappa=(1+m)\alpha^{2} and \hat{\kappa}=(1+m^{-1})\hat{\alpha}^{2} by Young’s inequality for any m>0 . However, to guarantee small \hat{\kappa} as in ( 78 ), we choose \hat{\alpha} sufficiently small. When \alpha\in(0,1] , ( GAEG+ ) requires two evaluations Fx^{k} and Fy^{k-1} of F and one resolvent J_{\eta T} as in ( AEG ). \mathrm{(c)} Comparison. Let us compare our variant ( OG ) with some related works [ 15 , 71 , 88 ] . If T=0 , then our scheme ( OG ) reduces to \left\{\begin{array}[]{lcl}x^{k+1}&:=&y^{k}-\eta(Fy^{k}-\gamma_{k}Fy^{k-1}),\\ y^{k}&:=&x^{k+1}+\theta_{k}(x^{k+1}-x^{k})-(\eta_{k}-\lambda_{k})Fy^{k}-\nu_{k}Fy^{k-1}.\end{array}\right. (80) This scheme now solves the equation ( NE ) as [ 15 ] . However, the \gamma_{k} parameter and last term \nu_{k}Fy^{k-1} in ( 80 ) make our variant here different from ( 15 , Algorithm 2) and allow us to handle the co-hypomonotonicity of F instead of the monotonicity of F as in [ 15 ] . In particular, when \beta=0 in ( 77 ), ( 80 ) does not reduce to ( 15 , Algorithm 2) . If T=\mathcal{N}_{\mathcal{X}} , the normal cone of a convex set \mathcal{X} , then our variant ( OG ) is also different from [ 71 ] even when we set \beta=0 . \mathrm{(d)} The implementation of ( GAEG+ ). To use the resolvent J_{\eta T} of T , we can rewrite ( GAEG+ ) equivalently to the following scheme: \left\{\begin{array}[]{lcl}x^{k+1}&:=&J_{\eta T}\big(y^{k}-\eta Fy^{k}+\eta\gamma_{k}z^{k}\big),\\ \xi^{k+1}&:=&\frac{1}{\eta}\big(y^{k}-\eta Fy^{k}+\eta\gamma_{k}z^{k}-x^{k+1}\big),\\ z^{k+1}&:=&u^{k+1}+\xi^{k+1},\\ y^{k+1}&:=&x^{k+1}+\theta_{k}(x^{k+1}-x^{k})-\eta_{k}z^{k+1}+\lambda_{k}(Fy^{k}+\xi^{k+1})-\nu_{k}z^{k}.\end{array}\right. (81) As before, we assume that J_{\eta T} is well-defined and single-valued. 7.2 Key Estimates for Convergence Analysis of ( GAEG+ ) In this subsection, we first establish some key bounds for our convergence analysis of ( GAEG+ ) and its instances. These bounds are independent of the choice of u^{k} , and cover all three instances discussed in (i), (ii), and (iii). \mathrm{(a)} Auxiliary functions. First, given r , \mu , \eta , \beta , and t_{k} in ( 77 ), let us define the following parameters: \begin{array}[]{lcl}\psi:=\frac{\eta-\beta}{\mu+1},\quad\omega:=\frac{\mu(\eta-\beta)}{\mu+1},\quad\textrm{and}\quad c_{k}:=\big[\psi(t_{k}-r+1)-(r-1)\beta\big]t_{k}.\end{array} (82) Next, given c_{k} in ( 82 ), we define the following two functions: \hskip-8.61108pt\begin{array}[]{lcl}\mathcal{P}_{k}&:=&\|r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})\|^{2}+r\mu\|x^{k}-x^{\star}\|^{2}+2c_{k}\langle z^{k},y^{k}-x^{k}\rangle,\\ {\color[rgb]{0,0,0}\mathcal{E}_{k}}&:=&\eta^{2}t_{k}^{2}\|d^{k}\|^{2}-t_{k+1}^{2}\|p^{k}\|^{2}+2\eta c_{k}\langle z^{k},d^{k}\rangle+2c_{k+1}\langle z^{k+1},p^{k}\rangle.\end{array}\hskip-8.61108pt (83) \mathrm{(b)} Expanding \mathcal{P}_{k}-\mathcal{P}_{k+1} . We expand \mathcal{P}_{k}-\mathcal{P}_{k+1} as in the following lemma. Lemma 10 Let \{(x^{k},y^{k})\} be computed by ( GAEG+ ), and \mathcal{P}_{k} and {\color[rgb]{0,0,0}\mathcal{E}_{k}} be defined by ( 76 ). Suppose that the parameters are updated by ( 77 ) and ( 82 ). Then \begin{array}[]{lcl}\mathcal{P}_{k}-\mathcal{P}_{k+1}&=&{\color[rgb]{0,0,0}\mathcal{E}_{k}}+\mu(2t_{k}-r-\mu)\|x^{k+1}-x^{k}\|^{2}\\ &&+{~}2r[(\eta-\beta)t_{k}-\delta]\langle z^{k+1},x^{k+1}-x^{\star}\rangle\\ &&-{~}2r[(\eta-\beta)t_{k}-\eta(r-1)]\langle z^{k},x^{k}-x^{\star}\rangle\\ &&+{~}2t_{k}\big[\omega(t_{k}-\mu-r)+\mu\eta\big]\langle z^{k+1}-z^{k},x^{k+1}-x^{k}\rangle\\ &&+{~}2\mu\eta t_{k}\langle\hat{w}^{k+1}-z^{k+1},x^{k+1}-x^{k}\rangle.\end{array} (84) \mathrm{(c)} Lower bounding {\color[rgb]{0,0,0}\mathcal{E}_{k}} . First, let us expand {\color[rgb]{0,0,0}\mathcal{E}_{k}} defined by ( 83 ) as follows. Lemma 11 Under the same setting as in Lemma 10 , {\color[rgb]{0,0,0}\mathcal{E}_{k}} defined by ( 83 ) can be expressed as \hskip-4.30554pt\begin{array}[]{lcl}{\color[rgb]{0,0,0}\mathcal{E}_{k}}&=&\big[(1-\mu)\psi t_{k}-\delta\big][(\eta-\beta)t_{k}-\delta]\|z^{k+1}\|^{2}\\ &&+{~}\big\{\eta(t_{k}-r+1)\big[(\eta-2\psi)(t_{k}-r+1)+2(r-1)\beta\big]-\beta^{2}t_{k}^{2}\big\}\|z^{k}\|^{2}\\ &&+{~}2\eta\omega t_{k}^{2}\langle z^{k+1},\hat{w}^{k+1}\rangle-2\beta\omega t_{k}^{2}\langle z^{k+1},z^{k}\rangle\\ &&-{~}2\eta\omega t_{k}(t_{k}-r+1)\langle\hat{w}^{k+1},z^{k}\rangle.\end{array}\hskip-4.30554pt (85) Now, we can lower bound \mathcal{E}_{k} as in the following lemma. Lemma 12 Under the same setting as in Lemma 10 and the L -Lipschitz continuity of F , {\color[rgb]{0,0,0}\mathcal{E}_{k}} defined by ( 83 ) satisfies \begin{array}[]{lcl}{\color[rgb]{0,0,0}\mathcal{E}_{k}}&\geq&\Lambda_{k+1}\|z^{k+1}\|^{2}-\Lambda_{k}\|z^{k}\|^{2}+S_{k}\|z^{k+1}\|^{2}+\beta\omega t_{k}^{2}\|z^{k+1}-z^{k}\|^{2}\\ &&+{~}\eta\omega(1-M^{2}\eta^{2})t_{k}^{2}\|d^{k}\|^{2}+\eta\omega\hat{\phi}t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\eta\omega\phi t_{k}^{2}\|z^{k+1}-\hat{w}^{k+1}\|^{2}-\frac{\eta\omega(1+\phi)(1+c_{1})t_{k}^{2}}{c_{1}}\|z^{k+1}-w^{k+1}\|^{2},\end{array} (86) where M^{2}:=\big[(1+\phi)(1+c_{1})+\hat{\phi}\big]L^{2} for any \phi\geq 0 , \hat{\phi}\geq 0 , and c_{1}>0 , and \left\{\begin{array}[]{lcl}\Lambda_{k}&:=&\frac{[(\eta-\beta)(t_{k}-r+1)-(r-1)\beta]^{2}+\mu(r-1)^{2}\eta\beta}{\mu+1},\\ \Gamma&:=&\frac{\mu(r-2)^{2}\eta^{2}+\mu[(\mu-1)r^{2}-2(\mu-3)r+\mu-7]\eta\beta-\mu[\mu r^{2}-2(\mu-1)r+\mu-3]\beta^{2}}{(\mu+1)^{2}},\\ S_{k}&:=&\frac{2\mu(r-2)(\eta-\beta)^{2}}{(\mu+1)^{2}}t_{k}-\Gamma.\end{array}\right. (87) 7.3 Convergence Analysis of The AEG Method Let us first establish the convergence of the instance ( AEG ) (the accelerated extragradient scheme) of ( GAEG+ ) with u^{k}:=Fx^{k} , which is less complicated than the generalization case when u^{k}\neq Fx^{k} in ( 78 ). We separate this case since we can leverage larger stepsizes and a wider range of L\rho (i.e. L\rho<\frac{1}{2} ). \mathrm{(a)} The Lyapunov function and technical lemmas. Given c_{k} in ( 82 ) and \Lambda_{k} in ( 87 ), consider the following Lyapunov function: \hskip-8.61108pt\begin{array}[]{lcl}\mathcal{L}_{k}&:=&\|r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})\|^{2}+r\mu\|x^{k}-x^{\star}\|^{2}+2c_{k}\langle w^{k},y^{k}-x^{k}\rangle\\ &&+{~}\Lambda_{k}\|w^{k}\|^{2}+2r[(\eta-\beta)t_{k}-\eta(r-1)]\langle w^{k},x^{k}-x^{\star}\rangle.\end{array}\hskip-8.61108pt (88) First, we prove the following key bound to establish convergence of ( AEG ). Lemma 13 For ( NI ), suppose that \mathrm{zer}(\Phi)\neq\emptyset , F is L -Lipschitz continuous, and \Phi is \rho -co-hypomonotone. Let \{(x^{k},y^{k})\} be generated by ( AEG ) ( i.e. u^{k}:=Fx^{k} ) using the parameters in ( 77 ) and ( 82 ). Let \mathcal{L}_{k} be defined by ( 88 ). Then \hskip-8.61108pt\begin{array}[]{lcl}\mathcal{L}_{k}-\mathcal{L}_{k+1}&\geq&\mu\big(2t_{k}-r-\mu-\frac{\mu\eta}{\hat{\phi}\omega}\big)\|x^{k+1}-x^{k}\|^{2}+\eta\omega(1-M^{2}\eta^{2})t_{k}^{2}\|d^{k}\|^{2}\\ &&+{~}t_{k}\big[\omega(\beta-2\rho)t_{k}+2\rho(\mu\omega+r\omega-\mu\eta)\big]\|w^{k+1}-w^{k}\|^{2}\\ &&+{~}\big[S_{k}-2\rho\omega r(r-2)\big]\|w^{k+1}\|^{2}.\end{array}\hskip-8.61108pt (89) Next, we lower bound the Lyapunov function \mathcal{L}_{k} in ( 88 ) as follows. Lemma 14 Under the same setting as in Lemma 13 , we have \hskip-8.61108pt\begin{array}[]{lcl}\mathcal{L}_{k}&\geq&\|r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})+[\psi(t_{k}-r+1)-(r-1)\beta]w^{k}\|^{2}\\ &&+{~}\sigma_{k}\|w^{k}\|^{2}+r\mu\|x^{k}-x^{\star}\|^{2},\end{array}\hskip-8.61108pt (90) where \sigma_{k}=\mu\big[\psi(t_{k}-r+1)-r\rho\big]^{2}-\mu\big[r^{2}\rho^{2}-\frac{(r-1)^{2}\beta(\eta-\beta)}{\mu+1}\big] . Moreover, we also have \hskip-8.61108pt\begin{array}[]{lcl}\mathcal{L}_{0}&\leq&(C_{0}+r^{2}+r\mu)\|x^{0}-x^{\star}\|^{2}+(C_{0}+\Lambda_{0})\|w^{0}\|^{2},\end{array}\hskip-8.61108pt (91) where C_{0}:=r[(\eta-\beta)t_{0}-\eta(r-1)] provided that t_{0}\geq\frac{\eta(r-1)}{\eta-\beta} . \mathrm{(b)} The \mathcal{O}\left(1/k\right) -last iterate convergence rates and summable bounds. For given \hat{\Gamma}:=\frac{2\Gamma}{r-2}+4\rho\omega r and \hat{\phi}:=\frac{1-L^{2}\eta^{2}}{2L^{2}\eta^{2}}>0 , we define \begin{array}[]{lcl}\mathcal{R}_{0}^{2}&:=&(C_{0}+r^{2}+r\mu)\|x^{0}-x^{\star}\|^{2}+(C_{0}+\Lambda_{0})\|w^{0}\|^{2},\\ t_{0}&:=&\max\Big\{\frac{r+1}{2}+\frac{\eta}{2\hat{\phi}\omega},\ \frac{\hat{\Gamma}}{(\eta-\beta)^{2}},\ \frac{4\rho[\eta-\omega(r+1)]}{\omega(\beta-2\rho)},\ \frac{\eta(r-1)}{\eta-\beta}\Big\},\end{array} (92) where C_{0}:=r[(\eta-\beta)t_{0}-\eta(r-1)]\geq 0 . Now, we are ready to prove the convergence of ( AEG ) as follows. Theorem 7.1 For ( NI ), suppose that \mathrm{zer}(\Phi)\neq\emptyset , F is L -Lipschitz continuous, and \Phi is \rho -co-hypomonotone such that 2L\rho<1 . Let \{(x^{k},y^{k})\} be generated by ( AEG ) ( i.e. u^{k}:=Fx^{k} ) using the parameters in ( 77 ) and ( 82 ) with \mu:=1 . Moreover, we choose r>2 , and t_{k} , \beta , \eta , and \hat{\phi} such that \begin{array}[]{ll}t_{k}:=k+t_{0},\quad 2\rho<\beta<\eta<\frac{1}{L},\quad\text{and}\quad\hat{\phi}:=\frac{1-L^{2}\eta^{2}}{2L^{2}\eta^{2}}>0.\end{array} (93) Then, the following summable bounds hold: \begin{array}[]{lcl}\sum_{k=0}^{\infty}\big(2k+2t_{0}-r-1-\frac{\eta}{\hat{\phi}\omega}\big)\|x^{k+1}-x^{k}\|^{2}&\leq&\mathcal{R}_{0}^{2},\\ \sum_{k=0}^{\infty}\eta\omega(1-L^{2}\eta^{2})(k+t_{0})^{2}\|d^{k}\|^{2}&\leq&2\mathcal{R}_{0}^{2},\\ \sum_{k=0}^{\infty}\big[(\eta-\beta)^{2}(k+t_{0})-\hat{\Gamma}\big]\|w^{k+1}\|^{2}&\leq&\frac{2}{r-2}\mathcal{R}_{0}^{2},\\ \sum_{k=0}^{\infty}(k+t_{0})^{2}\|w^{k+1}-w^{k}\|^{2}&\leq&\frac{2}{\omega(\beta-2\rho)}\mathcal{R}_{0}^{2},\end{array} (94) where \omega:=\frac{\eta-\beta}{2}>0 , \hat{\Gamma}:=\frac{2\Gamma}{r-2}+4\rho\omega r , and \mathcal{R}_{0}^{2} and t_{0} are given in ( 92 ). Furthermore, we also have \|Fx^{k}+\xi^{k}\|^{2}\leq\frac{4\mathcal{R}_{0}^{2}}{(\eta-\beta)^{2}(k+t_{0}-r+1)^{2}},\quad\text{where}\quad\xi^{k}\in Tx^{k}. (95) Remark 9 We highlight that we can extend our results in Theorem 7.1 to cover the extreme cases \eta:=\frac{1}{L} and \beta:=2\rho . However, the analysis is different and thus we omit it here to avoid overloading the paper. \mathrm{(c)} The o\left(1/k\right) -convergence rates. Next, we establish o\left(1/k\right) rates of ( AEG ). Theorem 7.2 Under the same conditions and settings as in Theorem 7.1 , we have the following results: \begin{array}[]{lcl}\lim_{k\to\infty}k^{2}\|Fx^{k}+\xi^{k}\|^{2}=0,\\ \lim_{k\to\infty}k^{2}\|x^{k+1}-x^{k}\|^{2}=0,\\ \lim_{k\to\infty}k^{2}\|y^{k}-x^{k}\|^{2}=0.\end{array} (96) These expressions show that \|Fx^{k}+\xi^{k}\|=o\left(1/k\right) , \|x^{k+1}-x^{k}\|=o\left(1/k\right) , and \|y^{k}-x^{k}\|=o\left(1/k\right) , respectively for ( AEG ), where \xi^{k}\in Tx^{k} . \mathrm{(d)} The convergence of iterate sequences. Finally, we prove the convergence of \{x^{k}\} and also \{y^{k}\} to a solution x^{\star}\in\mathrm{zer}(\Phi) . Theorem 7.3 Under the same conditions and settings as in Theorem 7.1 , if, additionally, T is closed ( in particular, T is maximally monotone ) , then both \{x^{k}\} and \{y^{k}\} generated by ( AEG ) converge to x^{\star}\in\mathrm{zer}(\Phi) . Remark 10 Theorem 7.3 requires T to be closed. Clearly, if T is maximally monotone, then by ( 9 , Proposition 20.37) , T is closed. If T is upper semi-continuous on \mathrm{dom}(T) and has closed values, then T is also closed [ 38 ] . 7.4 Convergence Analysis of The Generalized AEG Method In this subsection, we will investigate the convergence of the generalized scheme ( GAEG+ ) under the condition ( 78 ). Our analysis is new and requires several additional technical steps compared to Subsection 7.3 . \mathrm{(a)} Technical lemmas. Since z^{k} can be different from w^{k} , our first step is to process the product term \langle z^{k+1}-z^{k},x^{k+1}-x^{k}\rangle from ( 84 ). Lemma 15 Denote a_{k}:=t_{k}\big[\omega(t_{k}-r-\mu)+\mu\eta\big] . We consider the quantity: \begin{array}[]{lcl}\mathcal{F}_{k}&:=&2a_{k}\langle z^{k+1}-z^{k},x^{k+1}-x^{k}\rangle.\end{array} (97) Then, for any \Delta>0 , r>2 , \mu\geq 1 , and \eta\geq\frac{(r+\mu-1)\beta}{r-2} , we have \begin{array}[]{lcl}\mathcal{F}_{k}&\geq&2a_{k+1}\langle z^{k+1}-w^{k+1},y^{k+1}-x^{k+1}\rangle-2a_{k}\langle z^{k}-w^{k},y^{k}-x^{k}\rangle\\ &&-{~}2\omega t_{k}[(\eta+\beta)t_{k}+\delta]\|z^{k+1}-w^{k+1}\|^{2}-\frac{\eta\omega t_{k}^{2}}{2}\|z^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&-{~}\frac{\beta\omega t_{k}^{2}}{2}\|z^{k+1}-z^{k}\|^{2}-\frac{\delta\omega t_{k}}{2}\|z^{k+1}\|^{2}-\frac{\eta\omega t_{k}^{2}}{\Delta r}\|d^{k}\|^{2}\\ &&-{~}\Delta\eta r\omega t_{k}^{2}\|z^{k}-w^{k}\|^{2}+2a_{k}\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle\\ &&+{~}2b_{k}\langle z^{k+1}-w^{k+1},x^{k+1}-x^{k}\rangle,\end{array} (98) where b_{k}:=\omega(r+\mu-1)(t_{k}-r-\mu)+\mu\eta(r+\mu) . Now, we combine Lemmas 10 , 12 , and 15 to prove the following result. Lemma 16 Suppose that \Phi is \rho -co-hypomonotone and F is L -Lipschitz continuous. Let \{(x^{k},y^{k})\} be generated by ( GAEG+ ) using the update rules ( 77 ) and ( 82 ). Let us consider the following function: \begin{array}[]{lcl}\mathcal{G}_{k}&:=&\Lambda_{k}\|z^{k}\|^{2}+\|r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})\|^{2}+r\mu\|x^{k}-x^{\star}\|^{2}\\ &&+{~}2r\big[(\eta-\beta)t_{k}-\eta(r-1)\big]\langle z^{k},x^{k}-x^{\star}\rangle\\ &&+{~}2c_{k}\langle z^{k},y^{k}-x^{k}\rangle+2a_{k}\langle z^{k}-w^{k},y^{k}-x^{k}\rangle.\end{array} (99) Then, for any \Delta>0 , r>2 , \mu\geq 1 , and \eta\geq\frac{(r+\mu-1)\beta}{r-2} , we have \hskip-8.61108pt\begin{array}[]{lcl}\mathcal{G}_{k}-\mathcal{G}_{k+1}&\geq&\big[S_{k}-\frac{\delta\omega t_{k}}{2}-4\rho\omega r(r-2)\big]\|z^{k+1}\|^{2}-\frac{\omega r^{2}(r-2)^{2}}{\eta t_{k}^{2}}\|x^{k+1}-x^{\star}\|^{2}\\ &&+{~}\mu\big(t_{k}-r-\mu-\frac{2\mu\eta}{\omega}\big)\|x^{k+1}-x^{k}\|^{2}\\ &&+{~}\eta\omega\big(1-\frac{1}{\Delta r}-M^{2}\eta^{2}\big)t_{k}^{2}\|d^{k}\|^{2}+\eta\omega\hat{\phi}t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\eta\omega(\phi-1)t_{k}^{2}\|z^{k+1}-\hat{w}^{k+1}\|^{2}-\big(\Delta r\eta+8\rho\big)\omega t_{k}^{2}\|z^{k}-w^{k}\|^{2}\\ &&-{~}\hat{\Theta}_{k}\|z^{k+1}-w^{k+1}\|^{2}+\frac{(\beta-8\rho)\omega t_{k}^{2}}{2}\|z^{k+1}-z^{k}\|^{2},\end{array}\hskip-12.91663pt (100) where \hat{\Theta}_{k}:=\omega t_{k}\big[(\eta(5+2\phi)+2\beta+8\rho)t_{k}+2\delta+\omega(r+\mu-1)^{2}\big]+4\rho\omega r(r-2) . \mathrm{(b)} The Lyapunov function and its descent property. Now, we define the following Lyapunov function to analyze our generalized scheme ( GAEG+ ): \begin{array}[]{lcl}\hat{\mathcal{L}}_{k}&:=&\|r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})\|^{2}+r\mu\|x^{k}-x^{\star}\|^{2}+\Lambda_{k}\|z^{k}\|^{2}\\ &&+{~}2r\big[(\eta-\beta)t_{k}-\eta(r-1)\big]\langle z^{k},x^{k}-x^{\star}\rangle\\ &&+{~}2c_{k}\langle z^{k},y^{k}-x^{k}\rangle+2a_{k}\langle z^{k}-w^{k},y^{k}-x^{k}\rangle+\alpha_{k}\|z^{k}-w^{k}\|^{2},\end{array} (101) where \alpha_{k}:=(\Delta r\eta+8\rho)\omega t_{k}^{2} for some \Delta>0 determined later, c_{k} is in ( 82 ), and \Lambda_{k} is in ( 87 ). We prove the following result. Lemma 17 For ( NI ), suppose that \Phi is \rho -co-hypomonotone and F is L -Lipschitz continuous. Let \{(x^{k},y^{k})\} be generated by ( GAEG+ ) using the update rules ( 77 ) and ( 82 ) and u^{k} satisfying ( 78 ). Let \hat{\mathcal{L}}_{k} be defined by ( 101 ). Given r>2 , \mu\geq 1 , \epsilon\geq 0 , and \hat{\epsilon}\geq 0 , we choose \beta , \eta , and t_{k} such that \begin{array}[]{ll}\beta:=8\rho+2\epsilon,\quad\frac{(r+\mu-1)\beta}{r-2}\leq\eta,\quad\text{and}\quad t_{k}\geq\bar{t}_{0}.\end{array} (102) where \bar{t}_{0}:=\frac{D+\sqrt{D^{2}+4\eta E}}{2\eta} with D:=2\delta+\omega(r+\mu-1)^{2}+2\Delta r\eta+2\beta+2{\color[rgb]{0,0,0}\hat{\epsilon}\eta} and E:=4\rho r(r-2)+\Delta r\eta+8\rho+\hat{\epsilon}\eta . Then, we have \hskip-8.61108pt\begin{array}[]{lcl}\hat{\mathcal{L}}_{k}-\hat{\mathcal{L}}_{k+1}&\geq&\frac{\omega[3(r-2)\psi-(r-1)\beta]t_{k}-\hat{\Gamma}}{2}\|z^{k+1}\|^{2}-\frac{\omega r^{2}(r-2)^{2}}{\eta t_{k}^{2}}\|x^{k+1}-x^{\star}\|^{2}\\ &&+{~}\eta\omega(\hat{\phi}-\kappa\Theta)t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\mu\big(t_{k}-r-\mu-\frac{2\mu\eta}{\omega}\big)\|x^{k+1}-x^{k}\|^{2}\\ &&+{~}\eta\omega\big(1-\frac{1}{\Delta r}-\hat{\kappa}\Theta-M^{2}\eta^{2}\big)t_{k}^{2}\|d^{k}\|^{2}\\ &&+{~}\hat{\epsilon}\omega\eta t_{k+1}^{2}\|z^{k+1}-w^{k+1}\|^{2}+\epsilon\omega t_{k}^{2}\|z^{k+1}-z^{k}\|^{2},\end{array}\hskip-8.61108pt (103) where \hat{\Gamma}:=2\Gamma+8\rho\omega r(r-2) and \Theta:=10+2\phi+\Delta r+\hat{\epsilon} . \mathrm{(c)} The lower bound of \hat{\mathcal{L}}_{k} . We also need to lower bound \hat{\mathcal{L}}_{k} as follows. Lemma 18 Under the same setting as in Lemma 17 , we have \begin{array}[]{lcl}\hat{\mathcal{L}}_{k}&\geq&r(\mu-1)\|x^{k}-x^{\star}\|^{2}+\frac{(\eta-\beta)^{2}t_{k}^{2}}{2(\mu+1)^{2}}\|z^{k}\|^{2}\\ &&+{~}\big(\frac{\Delta r\eta}{\omega}-4r-1\big)\omega^{2}t_{k}^{2}\|z^{k}-w^{k}\|^{2},\end{array} (104) provided that \mu\geq 1 , r>2 , t_{k}\geq r , and t_{k}\geq\hat{t}_{0} , in which \hat{t}_{0} is defined by \begin{array}[]{lcl}\hat{t}_{0}&:=&\begin{cases}\frac{B+\sqrt{B^{2}-AC}}{A},&\text{if }B^{2}\geq AC,\\ 0,&\text{otherwise,}\end{cases}\end{array} (105) where \left\{\begin{array}[]{lcl}A&:=&\frac{(\eta-\beta)^{2}}{(\mu+1)^{2}}\left(\mu-\frac{1}{2r}-\frac{1}{2}\right)>0,\\ B&:=&\frac{(\eta-\beta)}{(\mu+1)^{2}}\big[\frac{(r-1)[2r\mu(\eta-\beta)-(\eta+\mu\beta)]}{2r}+\rho(\mu+1)(2r\mu-1)\big],\\ C&:=&(r-1)\big\{\frac{(r-1)(\eta+\mu\beta)}{(\mu+1)^{2}}\big[\left(\mu-\frac{1}{2r}\right)\eta-\frac{(2r+1)\mu\beta}{2r}\big]-2\rho[\eta-(2r+1)\omega]\big\}.\end{array}\right. In particular, if we choose \Delta:=3 , then we have \begin{array}[]{lcl}\hat{\mathcal{L}}_{k}\geq r(\mu-1)\|x^{k}-x^{\star}\|^{2}+\frac{(\eta-\beta)^{2}t_{k}^{2}}{4(\mu+1)^{2}}\|w^{k}\|^{2}.\end{array} (106) \mathrm{(d)} The \mathcal{O}\left(1/k\right) convergence rates and summable bounds. Fix r>2 and \hat{\epsilon}\geq 0 , suppose that the constant \hat{\kappa} in ( 78 ) satisfies \begin{array}[]{lcl}0\leq\hat{\kappa}<\frac{3r-1}{3r(12+3r+\hat{\epsilon})}.\end{array} (107) Given \hat{\kappa} as in ( 107 ), \bar{t}_{0} defined in Lemma 17 , and \hat{t}_{0} defined by ( 105 ), we denote the following two constants: \hskip-8.61108pt\left\{\begin{array}[]{lcl}\Psi&:=&\frac{1}{\sqrt{4+12\kappa+3\kappa r+\kappa\hat{\epsilon}}}\big[1-\frac{1}{3r}-\hat{\kappa}(12+3r+\hat{\epsilon})\big]\in(0,1],\\ t_{0}&:=&\max\Big\{\bar{t}_{0},\ \hat{t}_{0},\ r,\ \frac{\eta(r-1)}{\eta-\beta},\ \frac{\mu}{\omega}(r-1)[\eta-(2r+1)\omega],\ \frac{(r-2)\sqrt{\omega r}}{\sqrt{\eta(\mu-1)}}\Big\}.\end{array}\right.\hskip-8.61108pt (108) Note that if \hat{\kappa}=0 , (e.g., when u^{k}:=Fy^{k-1} as in ( OG ), then ( 107 ) automatically holds, and \Psi in ( 108 ) reduces to \Psi:=\frac{3r-1}{3r\sqrt{4+\kappa(12+3r+\hat{\epsilon})}} . Now, we are ready to prove the convergence of ( GAEG+ ) as follows. Theorem 7.4 For ( NI ), suppose that \mathrm{zer}(\Phi)\neq\emptyset , \Phi is \rho -co-hypomonotone, and F is L -Lipschitz continuous. Fix r>2 and \mu>1 , let \{(x^{k},y^{k})\} be generated by ( GAEG+ ) using the update rules ( 77 ) and ( 82 ) and u^{k} such that ( 78 ) holds with \hat{\kappa} as in ( 107 ). Suppose that L\rho<\frac{(r-2)\Psi}{8(r+\mu-1)} for \Psi given by ( 108 ). Let us choose t_{k} , \beta , and \eta such that \begin{array}[]{ll}&t_{k}=k+t_{0},\quad\beta:=8\rho+2\epsilon,\ \ \text{and}\ \ \frac{(r+\mu-1)\beta}{r-2}\leq\eta\leq\bar{\eta}:=\frac{\Psi}{L},\end{array} (109) where t_{0} is given in ( 108 ) and \epsilon\geq 0 is given. Then, we have \begin{array}[]{lcl}\sum_{k=0}^{\infty}(k+t_{0})\|x^{k+1}-x^{k}\|^{2}&<&+\infty,\\ \sum_{k=0}^{\infty}(k+t_{0})^{2}\|z^{k}-w^{k}\|^{2}&<&+\infty,\\ \sum_{k=0}^{\infty}(k+t_{0})^{2}\|z^{k+1}-z^{k}\|^{2}&<&+\infty,\\ \sum_{k=0}^{\infty}(\bar{\eta}^{2}-\eta^{2})(k+t_{0})^{2}\|d^{k}\|^{2}&<&+\infty,\\ \sum_{k=0}^{\infty}(k+t_{0})\|z^{k}\|^{2}&<&+\infty,\\ \sum_{k=0}^{\infty}(k+t_{0})\|w^{k}\|^{2}&<&+\infty.\end{array} (110) Moreover, for \mathcal{R}_{0}^{2} given in ( 92 ), we also have \|Fx^{k}+\xi^{k}\|^{2}\leq\frac{4\Omega(\mu+1)^{2}\mathcal{R}_{0}^{2}}{(\eta-8\rho)^{2}(k+t_{0})^{2}},\quad\textrm{where}\quad\xi^{k}\in Tx^{k}, (111) where \Omega:=\left(\frac{1+t_{0}+c}{1+t_{0}-c}\right)^{\frac{c}{2}}\exp\left(\frac{c^{2}}{(1+t_{0})^{2}-c^{2}}\right)<\infty with c^{2}:=\frac{\omega r(r-2)^{2}}{\eta(\mu-1)} . \mathrm{(e)} The o\left(1/k\right) last-iterate convergence rates. Our next step is to establish o\left(1/k\right) convergence rates of ( GAEG+ ). Theorem 7.5 Under the same conditions and settings as in Theorem 7.4 and we additionally choose \eta<\bar{\eta} , then we have the following limits: \begin{array}[]{lcl}\lim_{k\to\infty}k^{2}\|Fx^{k}+\xi^{k}\|^{2}=0,\\ \lim_{k\to\infty}k^{2}\|x^{k+1}-x^{k}\|^{2}=0,\\ \lim_{k\to\infty}k^{2}\|y^{k}-x^{k}\|^{2}=0.\end{array} (112) These expressions show that \|Fx^{k}+\xi^{k}\|=o\left(1/k\right) , \|x^{k+1}-x^{k}\|=o\left(1/k\right) , and \|y^{k}-x^{k}\|=o\left(1/k\right) , respectively for ( GAEG+ ), where \xi^{k}\in Tx^{k} . \mathrm{(f)} The convergence of iterate sequences. Finally, we prove the convergence of the iterate sequences \{x^{k}\} and \{y^{k}\} generated by ( GAEG+ ) to a solution x^{\star}\in\mathrm{zer}(\Phi) . The proof of this result is rather involved and hence, we divide it into several technical lemmas (see Appendix F ). Theorem 7.6 Under the same conditions and settings as in Theorem 7.4 , if we additionally choose \eta<\bar{\eta} and assume that T is closed, then both \{x^{k}\} and \{y^{k}\} generated by ( GAEG+ ) converge to x^{\star}\in\mathrm{zer}(\Phi) . Remark 11 We have used several times of Young’s inequality in our analysis in Subsection 7.4 . Our analysis as well as the ranges of L\rho and other parameters can be improved by appropriately choosing the constant c whenever we applying Young’s inequality of the form \|u-v\|^{2}\leq(1+c)\|u\|^{2}+\frac{1+c}{c}\|v\|^{2} . 8 Numerical Experiments In this section, we provide a number of experiments to validate our algorithms and their theoretical results. All the algorithms are implemented in Python running on a single node of a Linux server (called Longleaf ) with the configuration: AMD EPYC 7713 64-Core Processor, 512KB cache, and 64GB RAM. 8.1 Mathematical model: Quadratic minimax optimization The mathematical model we use for our experiments is the following simplex constrained quadratic minimax problem: \displaystyle\min_{u\in\mathbb{R}^{p_{1}}}\max_{v\in\mathbb{R}^{p_{2}}}\Big\{\mathcal{L}(u,v)=f(u)+\mathcal{H}(x,y)-g(v)\Big\}, (113) where \mathcal{H}(x,y):=\frac{1}{2}u^{\top}Au+b^{\top}u+u^{\top}Lv-\frac{1}{2}v^{\top}Bv-c^{\top}v such that A\in\mathbb{R}^{p_{1}\times p_{1}} and B\in\mathbb{R}^{p_{2}\times p_{2}} are symmetric matrices, b\in\mathbb{R}^{p_{1}} , c\in\mathbb{R}^{p_{2}} are given vectors, and L\in\mathbb{R}^{p_{1}\times p_{2}} is a given matrix. The functions f(u)=\delta_{\Delta_{p_{1}}}(u) and g(v)=\delta_{\Delta_{p_{2}}}(v) are added to handle simplex constraints u\in\Delta_{p_{1}} and v\in\Delta_{{\color[rgb]{0,0,0}p_{2}}} , where \Delta_{p_{1}} and \Delta_{p_{2}} are the standard simplexes in \mathbb{R}^{p_{1}} and \mathbb{R}^{p_{2}} , respectively, and \delta_{\mathcal{X}} is the indicator function of a closed convex set \mathcal{X} . First, we denote x\coloneqq[u,v]\in\mathbb{R}^{p} for p:=p_{1}+p_{2} , which is the concatenation of the primal variable u\in\mathbb{R}^{p_{1}} and its dual variable v\in\mathbb{R}^{p_{2}} . Next, we define \mathbf{F}\coloneqq\left[[A,L],[-L^{\top},B]\right] as the KKT (Karush-Kuhn-Tucker) matrix in \mathbb{R}^{p\times p} constructed from the four blocks A,L,-L^{\top} , and B , and \mathbf{f}\coloneqq[b,c]\in\mathbb{R}^{p} . The operator F:\mathbb{R}^{p}\to\mathbb{R}^{p} is then defined as Fx\coloneqq\mathbf{F}x+\mathbf{f} . Then, we denote T\coloneqq[\partial\delta_{\Delta_{p_{1}}},\partial\delta_{\Delta_{p_{2}}}] the maximally monotone mapping constructed from the subdifferentials of \delta_{\Delta_{p_{1}}} and \delta_{\Delta_{p_{2}}} . Finally, the optimality condition of ( 113 ) becomes 0\in Fx+Tx covered by ( NI ). 8.2 Numerical experiments \mathrm{(a)} Data generation. Our aim is to verify the proposed algorithms and their corresponding theoretical results. In what follows, all matrices and vectors in F are generated randomly from the standard normal distribution. We generate the matrix A=QDQ^{\top} , where Q is an orthonormal matrix obtained from the QR factorization of a random matrix, and D=\mathrm{diag}(d_{1},\dots,d_{p_{1}}) is the diagonal matrix formed from d_{1},\dots,d_{p_{1}} randomly generated and then clipped by a lower bound \underline{d} , i.e. d_{j}\coloneqq\max\{d_{j},\underline{d}\} . The matrix B is also generated by the same way. The matrix L and vectors b and c are also randomly generated. \mathrm{(b)} Experiment setup. We perform four different experiments. In Experiment 1 and Experiment 2 , we choose \underline{d}=0.1 (monotone), and run the four sets of algorithms on 10 problem instances for each case: p=1000 and p=2000 , respectively. In Experiment 3 and Experiment 4 , we choose \underline{d}=-10^{-3} (possibly nonmonotone) and run the same four sets of algorithms on 10 problem instances for each case: p=1000 and p=2000 , respectively. Then, we report the mean of the relative operator norm \frac{\|\mathcal{G}_{\eta}x^{k}\|}{\|\mathcal{G}_{\eta}x^{0}\|} over 10 problem instances, where \mathcal{G}_{\eta}\coloneqq\eta^{-1}\left(x-J_{\eta T}(x-\eta Fx)\right) is defined in ( 4 ). \mathrm{(c)} Algorithms and parameters. In each Experiment , we examine the following four sets of algorithms. In the first set, we consider five variants using the direction u^{k}:=Fx^{k} of ( GEAG ), ( GFEG ), ( GFEG+ ), ( GAEG ), and ( GAEG+ ). Next, in the second one, we consider five variants of ( GEAG ), ( GFEG ), ( GFEG+ ), ( GAEG ), and ( GAEG+ ) using u^{k}:=Fy^{k-1} (in fact, ( GFEG ) uses a slightly different direction u^{k}=Fy^{k-1}+\xi^{k}-\xi^{k-1} due to the choice \alpha=0 , \hat{\alpha}=1 ). Then, in the third set, we investigate different variants of ( GFEG+ ) by fixing r=3 and choosing different values of \mu . Finally, in the fourth set, we study different variants of ( GAEG+ ) by modifying the choice of its corresponding parameters. The stepsize of each algorithm is tuned manually to obtain the best possible performance for all algorithms, but not necessarily optimal. The starting points are always chosen as x^{0}\coloneqq 0.01\cdot\texttt{ones}(p) . \mathrm{(d)} Results. The numerical results of Experiment 1 and Experiment 2 are reported in Figures 2 , 3 , 4 , and 5 . Figure 2: The behaviors of the first set of algorithms for solving ( NI ) in Experiments 1 and 2 when we choose u^{k}:=Fx^{k} . The plot reveals the mean of 10 problem instances. Figure 3: The behaviors of the second set of algorithms for solving ( NI ) in Experiments 1 and 2 when we choose u^{k}:=Fy^{k-1} . The plot reveals the mean of 10 problem instances. Figure 4: The behaviors of ( GFEG+ ) for solving ( NI ) in Experiments 1 and 2 . The plot reveals the mean of 10 problem instances. The legend presents the values of \mu used in the corresponding instant of ( GFEG+ ). Figure 5: The behaviors of ( GAEG+ ) for solving ( NI ) in Experiments 1 and 2 . The plot reveals the mean of 10 problem instances. The legend presents the values of r and \mu used in the corresponding instant of ( GAEG+ ). For example, (3,1) means the solid orange line with filled circle marker corresponds to the instance of ( GAEG+ ) using r=3 and \mu=1 . From Figure 2 , we can see that in both experiments, ( GEAG ), ( GFEG ), and ( GAEG ) have a similar performance within the accuracy of 10^{-4} after 5000 iterations. Our ( GFEG+ ) gives a better performance than these three schemes, but ( GAEG+ ) strongly outperforms its competitors when providing the accuracy of 10^{-7} after the same number of iterations. In fact, other experiments reveal that ( GAEG+ ) can reach the accuracy of 10^{-14} after 5000 iterations by appropriately tuning the parameters (see Figure 5 ). A similar observation can also be seen from Figure 3 , which presents the behaviors of the second set of algorithms using u^{k}:=Fy^{k-1} . However, ( GFEG+ ) now performs better and ( GAEG+ ) is slightly slowed down. Next, we test ( GFEG+ ) with u^{k}:=Fx^{k} by fixing r=3 and choosing 6 different values of \mu ranging from 0.05 to 0.05 . Figure 4 shows the results of the two experiments, where we observe that when \mu is decreasing, this method converges faster and achieves better accuracy. Finally, in Figure 5 , we fixed the values of \eta and \beta and examine the effect of r and \mu to the performance of ( GAEG+ ). The numerical results show that larger values of r and \mu can significantly improve the performance of ( GAEG+ ), which helps this algorithm become more effective and outperform the remaining competitors. \mathrm{(e)} Experiments with nonmonotone problems. We also perform a similar test with four sets of experiments as in Experiment 1 and Experiment 2 , but on a class of possibly nonmonotone problem instances of ( 113 ). We denote these experiments by Experiment 3 and Experiment 4 . Figure 6: The behaviors of the first set of algorithms for solving ( NI ) in Experiment 3 and Experiment 4 . The plot reveals the mean of 10 problem instances. Figure 7: The behaviors of the second set of algorithms for solving ( NI ) in Experiment 3 and Experiment 4 . The plot reveals the mean of 10 problem instances. Figure 8: The behaviors of ( GFEG+ ) for solving ( NI ) in Experiments 3 and 4 . The plot reveals the mean of 10 problem instances. The legend presents the values of \mu used in the corresponding instant of ( GFEG+ ). Figure 9: The behaviors of ( GAEG+ ) for solving ( NI ) in Experiments 3 and 4 . The plot reveals the mean of 10 problem instances. The legend presents the values of r and \mu used in the corresponding instant of ( GAEG+ ). For example, (3,1) means the solid orange line with filled circle marker corresponds to the instance of ( GAEG+ ) using r=3 and \mu=1 . Despite the relaxation of the monotonicity of the involved operator, the results presented in Figures 6 , 7 , 8 , and 9 are still consistent with what were observed in the monotone scenarios in Experiment 1 and Experiment 2 . These findings reinforce the reliability of the accelerated extragradient-type methods studied in this paper across diverse circumstances and futher establish their potential for applications where monotonicity is not guaranteed. Acknowledgements. This work is partially supported by the National Science Foundation (NSF), grant no. NSF-RTG DMS-2134107 and the Office of Naval Research (ONR), grant No. N00014-23-1-2588 (2023-2026). Appendix A Appendix 1: Useful Technical Lemmas We need the following lemmas for our convergence analysis in the paper. Lemma 19 (see Lemma 5.31, [ 9 ] ) Let \{u_{k}\} , \{v_{k}\} , and \{\gamma_{k}\} be nonnegative sequences in \mathbb{R} such that u_{k+1}\leq(1+\gamma_{k})u_{k}-v_{k},\quad\forall k\geq 0. (114) If A:=\sum_{k=0}^{\infty}\gamma_{k}<+\infty and \omega:=\prod_{k=0}^{\infty}(1+\gamma_{k})\in[1,+\infty) , then \begin{array}[]{lcl}u_{k+1}\leq u_{0}\prod_{l=0}^{k}(1+\gamma_{l})+\sum_{l=0}^{k}\varepsilon_{l}\prod_{j=l+1}^{k}(1+\gamma_{j})\leq\omega(u_{0}+B_{k}).\end{array}\vskip-2.15277pt (115) Moreover, \{u_{k}\} converges ( i.e. \lim_{k\to\infty}u_{k} exists ) and \begin{array}[]{lcl}\sum_{k=0}^{\infty}v_{k}&\leq&(1+\omega A)u_{0}.\end{array} (116) Lemma 20 (see [ 82 ] ) For any r>0 and c>0 , let A_{k}:=\sum_{l=0}^{{\color[rgb]{0,0,0}k}}\frac{1}{(l+r)^{2}} and \omega_{k}:=\prod_{l=0}^{k}\big(1+\frac{c}{(l+r)^{2}}\big) . Then, we have A_{k}\leq\frac{1}{r} and \omega_{k}\leq e^{c/r} for k\geq 0 . Appendix B The Proof of Technical Results in Section 3 This section provides the full proof of Lemmas and Theorems in Section 3 . B.1 The Proof of Lemma 1 Since T is maximally 3 -cyclically monotone, for any \xi^{k+1}\in Tx^{k+1} , \xi^{k}\in Tx^{k} , and \zeta^{k}\in Ty^{k} , we have \begin{array}[]{lcl}\langle\xi^{k+1},x^{k+1}-x^{k}\rangle+\langle\xi^{k},x^{k}-y^{k}\rangle+\langle\zeta^{k},y^{k}-x^{k+1}\rangle\geq 0.\end{array} By the monotonicity of F , we also have \langle Fx^{k+1}-Fx^{k},x^{k+1}-x^{k}\rangle\geq 0 . Summing up this inequality and the last one, and then using w^{k+1}=Fx^{k+1}+\xi^{k+1} , w^{k}:=Fx^{k}+\xi^{k} , and \tilde{w}^{k}:=Fx^{k}+\zeta^{k} from ( 8 ), we obtain \begin{array}[]{lcl}\langle w^{k+1},x^{k+1}-x^{k}\rangle-\langle\tilde{w}^{k},x^{k+1}-x^{k}\rangle+\langle w^{k}-\tilde{w}^{k},x^{k}-y^{k}\rangle\geq 0.\end{array} (117) Now, let us denote e^{k}:=u^{k}-Fx^{k}=\tilde{z}^{k}-\tilde{w}^{k} . Then, we get \tilde{z}^{k}=\tilde{w}^{k}+e^{k} . By this representation, we can derive from ( 9 ) that \left\{\begin{array}[]{lcl}x^{k+1}-x^{k}&=&-\frac{\tau_{k}}{1-\tau_{k}}(x^{k+1}-x^{0})-\frac{\eta}{1-\tau_{k}}\hat{w}^{k+1},\\ x^{k+1}-x^{k}&=&-\tau_{k}(x^{k}-x^{0})-\eta\hat{w}^{k+1},\\ x^{k}-y^{k}&=&\tau_{k}(x^{k}-x^{0})+\hat{\eta}_{k}\tilde{w}^{k}+\hat{\eta}_{k}e^{k}.\end{array}\right. Substituting these relations into ( 117 ), then using Young’s inequality in {\tiny\textcircled{1}} for any r>0 , and rearranging terms, we arrive at \begin{array}[]{lcl}\mathcal{T}_{[1]}&:=&\tau_{k}\langle w^{k},x^{k}-x^{0}\rangle-\frac{\tau_{k}}{1-\tau_{k}}\langle w^{k+1},x^{k+1}-x^{0}\rangle\\ &\geq&\frac{\eta}{1-\tau_{k}}\langle w^{k+1},\hat{w}^{k+1}\rangle-\eta\langle\tilde{w}^{k},\hat{w}^{k+1}\rangle-\hat{\eta}_{k}\langle w^{k},\tilde{w}^{k}\rangle+\hat{\eta}_{k}\|\tilde{w}^{k}\|^{2}\\ &&-{~}\hat{\eta}_{k}\langle e^{k},w^{k}-\tilde{w}^{k}\rangle\\ &\overset{\tiny{\tiny\textcircled{1}}}{\geq}&\frac{\eta}{1-\tau_{k}}\langle w^{k+1},\hat{w}^{k+1}\rangle-\eta\langle\tilde{w}^{k},\hat{w}^{k+1}\rangle-\hat{\eta}_{k}\langle w^{k},\tilde{w}^{k}\rangle+\hat{\eta}_{k}\|\tilde{w}^{k}\|^{2}\\ &&-{~}\frac{r\hat{\eta}_{k}}{2}\|w^{k}-\tilde{w}^{k}\|^{2}-\frac{\hat{\eta}_{k}}{2r}\|e^{k}\|^{2}.\end{array} Multiplying this inequality by \frac{b_{k}}{\tau_{k}} and using b_{k+1}=\frac{b_{k}}{1-\tau_{k}} from ( 12 ), we can show that \begin{array}[]{lcl}\mathcal{T}_{[2]}&:=&b_{k}\langle w^{k},x^{k}-x^{0}\rangle-b_{k+1}\langle w^{k+1},x^{k+1}-x^{0}\rangle\\ &\geq&\frac{\eta b_{k+1}}{\tau_{k}}\langle w^{k+1}-\tilde{w}^{k},\hat{w}^{k+1}\rangle+\eta b_{k+1}\langle\tilde{w}^{k},\hat{w}^{k+1}\rangle-\frac{b_{k}\hat{\eta}_{k}}{\tau_{k}}\langle w^{k},\tilde{w}^{k}\rangle\\ &&+{~}\frac{b_{k}\hat{\eta}_{k}}{\tau_{k}}\|\tilde{w}^{k}\|^{2}-\frac{rb_{k}\hat{\eta}_{k}}{2\tau_{k}}\|w^{k}-\tilde{w}^{k}\|^{2}-\frac{b_{k}\hat{\eta}_{k}}{2r\tau_{k}}\|e^{k}\|^{2}.\end{array} (118) Next, from ( 9 ), we have x^{k+1}-y^{k}=-\eta\hat{w}^{k+1}+\hat{\eta}_{k}\tilde{z}^{k}=-\eta\hat{w}^{k+1}+\hat{\eta}_{k}\tilde{w}^{k}+\hat{\eta}_{k}e^{k} . Using this expression, the L -Lipschitz continuity of F , and Young’s inequality in {\tiny\textcircled{2}} for any c>0 , we can derive that \begin{array}[]{lcl}\|w^{k+1}-\hat{w}^{k+1}\|^{2}&=&\|Fx^{k+1}-Fy^{k}\|^{2}\leq L^{2}\|x^{k+1}-y^{k}\|^{2}\\ &=&L^{2}\|\eta\hat{w}^{k+1}-\hat{\eta}_{k}\tilde{w}^{k}-\hat{\eta}_{k}e^{k}\|^{2}\\ &\overset{\tiny{\tiny\textcircled{2}}}{\leq}&(1+c)L^{2}\|\eta\hat{w}^{k+1}-\hat{\eta}_{k}\tilde{w}^{k}\|^{2}+\frac{(1+c)L^{2}\hat{\eta}_{k}^{2}}{c}\|e^{k}\|^{2}.\end{array} Multiplying both sides of this inequality by 1+\omega for some \omega>0 and rearranging the result, we get \begin{array}[]{lcl}0&\geq&\|w^{k+1}-\hat{w}^{k+1}\|^{2}+\omega\|w^{k+1}-\hat{w}^{k+1}\|^{2}-(1+c)(1+\omega)L^{2}\|\eta\hat{w}^{k+1}-\hat{\eta}_{k}\tilde{w}^{k}\|^{2}\\ &&-{~}\frac{(1+c)}{c}(1+\omega)L^{2}\hat{\eta}_{k}^{2}\|e^{k}\|^{2}.\end{array} For simplicity of notations, we denote M_{c}:=(1+c)(1+\omega)L^{2} . Using this M_{c} and expanding the last expression, we obtain \begin{array}[]{lcl}0&\geq&\omega\|w^{k+1}-\hat{w}^{k+1}\|^{2}+\|w^{k+1}\|^{2}-2\langle w^{k+1}-\tilde{w}^{k},\hat{w}^{k+1}\rangle+(1-M_{c}\eta^{2})\|\hat{w}^{k+1}\|^{2}\\ &&-{~}2(1-M_{c}\eta\hat{\eta}_{k})\langle\tilde{w}^{k},\hat{w}^{k+1}\rangle-M_{c}\hat{\eta}_{k}^{2}\|\tilde{w}^{k}\|^{2}-\frac{M_{c}\hat{\eta}_{k}^{2}}{c}\|e^{k}\|^{2}.\end{array} Multiplying this inequality by \frac{\eta b_{k+1}}{2\tau_{k}} , adding the result to ( 118 ), and using \hat{\eta}_{k}=\eta(1-\tau_{k}) and b_{k}=b_{k+1}(1-\tau_{k}) from ( 12 ), we can prove that \begin{array}[]{lcl}\mathcal{T}_{[2]}&:=&b_{k}\langle w^{k},x^{k}-x^{0}\rangle-b_{k+1}\langle w^{k+1},x^{k+1}-x^{0}\rangle\\ &\geq&\frac{\eta b_{k+1}}{2\tau_{k}}\|w^{k+1}\|^{2}+\frac{\eta b_{k+1}(1-M_{c}\eta^{2})}{2\tau_{k}}\|\hat{w}^{k+1}\|^{2}+\frac{\omega\eta b_{k+1}}{2\tau_{k}}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&-{~}\frac{r\eta b_{k+1}(1-\tau_{k})^{2}}{2\tau_{k}}\|w^{k}-\tilde{w}^{k}\|^{2}+\frac{\eta b_{k+1}(1-\tau_{k})^{2}(2-M_{c}\eta^{2})}{2\tau_{k}}\|\tilde{w}^{k}\|^{2}\\ &&-{~}\frac{\eta b_{k+1}(1-M_{c}\eta^{2})(1-\tau_{k})}{\tau_{k}}\langle\tilde{w}^{k},\hat{w}^{k+1}\rangle-\frac{\eta b_{k+1}(1-\tau_{k})^{2}}{\tau_{k}}\langle w^{k},\tilde{w}^{k}\rangle\\ &&-{~}\frac{\eta b_{k+1}(1-\tau_{k})^{2}(c+rM_{c}\eta^{2})}{2rc\tau_{k}}\|e^{k}\|^{2}.\end{array} Utilizing the following two identities \begin{array}[]{lcl}2(1-\tau_{k})\langle\tilde{w}^{k},\hat{w}^{k+1}\rangle&=&(1-\tau_{k})^{2}\|\tilde{w}^{k}\|^{2}+\|\hat{w}^{k+1}\|^{2}-\|\hat{w}^{k+1}-(1-\tau_{k})\tilde{w}^{k}\|^{2},\\ 2\langle w^{k},\tilde{w}^{k}\rangle&=&\|w^{k}\|^{2}+\|\tilde{w}^{k}\|^{2}-\|w^{k}-\tilde{w}^{k}\|^{2},\end{array} we can further lower bound \mathcal{T}_{[2]} above as \begin{array}[]{lcl}\mathcal{T}_{[2]}&\geq&\frac{\eta b_{k+1}}{2\tau_{k}}\|w^{k+1}\|^{2}-\frac{\eta b_{k+1}(1-\tau_{k})^{2}}{2\tau_{k}}\|w^{k}\|^{2}+\frac{\omega\eta b_{k+1}}{2\tau_{k}}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\frac{\eta b_{k+1}(1-M_{c}\eta^{2})}{2\tau_{k}}\|\hat{w}^{k+1}-(1-\tau_{k})\tilde{w}^{k}\|^{2}+\frac{(1-r)\eta b_{k+1}(1-\tau_{k})^{2}}{2\tau_{k}}\|w^{k}-\tilde{w}^{k}\|^{2}\\ &&-{~}\frac{\eta b_{k+1}(1-\tau_{k})^{2}(c+rM_{c}\eta^{2})}{2rc\tau_{k}}\|e^{k}\|^{2}.\end{array} Substituting a_{k}:=\frac{\eta b_{k}(1-\tau_{k})}{\tau_{k}}=\frac{\eta b_{k+1}(1-\tau_{k})^{2}}{\tau_{k}} and \mathcal{V}_{k} from ( 11 ) into the last estimate, we can show that \begin{array}[]{lcl}\mathcal{V}_{k}-\mathcal{V}_{k+1}&\geq&\frac{\eta b_{k+1}}{2}\big(\frac{1}{\tau_{k}}-\frac{1-\tau_{k+1}}{\tau_{k+1}}\big)\|w^{k+1}\|^{2}+\frac{\omega\eta b_{k+1}}{2\tau_{k}}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\frac{\eta b_{k+1}(1-M_{c}\eta^{2})}{2\tau_{k}}\|\hat{w}^{k+1}-(1-\tau_{k})\tilde{w}^{k}\|^{2}\\ &&+{~}\frac{(1-r)a_{k}}{2}\|w^{k}-\tilde{w}^{k}\|^{2}-\frac{(c+rM_{c}\eta^{2})a_{k}}{2rc}\|e^{k}\|^{2},\end{array} which proves ( 13 ) after replacing e^{k}:=u^{k}-Fx^{k} . \square B.2 The Proof of Theorem 3.1 First, employing \mathcal{L}_{k} from ( 11 ), and substituting \tau_{k}:=\frac{1}{k+\nu} and c_{k}:=\frac{(c+rM_{c}\eta^{2})a_{k}}{rc}=\frac{(c+rM_{c}\eta^{2})\eta b_{k}(k+\nu-1)}{rc} into ( 13 ), we can prove that \hskip-4.30554pt\begin{array}[]{lcl}\mathcal{L}_{k}-\mathcal{L}_{k+1}&\geq&\frac{\omega\eta b_{k+1}(k+\nu)}{2}\|w^{k+1}\!-\!\hat{w}^{k+1}\|^{2}+\frac{(1-M_{c}\eta^{2})\eta b_{k+1}(k+\nu)}{2}\|\hat{w}^{k+1}\!-\!(1-\tau_{k})\tilde{w}^{k}\|^{2}\\ &&+{~}\frac{(1-r)a_{k}}{2}\|w^{k}-\tilde{w}^{k}\|^{2}-\frac{(c+rM_{c}\eta^{2})\eta b_{k+1}(k+\nu)}{2rc}\|u^{k+1}-Fx^{k+1}\|^{2}.\end{array}\hskip-12.91663pt Next, combining the last estimate and ( 10 ), and using w^{k+1}-\hat{w}^{k+1}=Fx^{k+1}-Fy^{k} , we can show that \hskip-4.30554pt\begin{array}[]{lcl}\mathcal{L}_{k}-\mathcal{L}_{k+1}&\geq&\frac{\eta b_{k+1}(k+\nu)}{2}\big[\omega-\frac{\kappa(c+rM_{c}\eta^{2})}{rc}\big]\|w^{k+1}-\hat{w}^{k+1}\|^{2}+\frac{(1-r)a_{k}}{2}\|w^{k}-\tilde{w}^{k}\|^{2}\\ &&+{~}\frac{\eta b_{k+1}(k+\nu)}{2}\big[1-M_{c}\eta^{2}-\frac{\hat{\kappa}\eta^{2}(c+rM_{c}\eta^{2})}{rc}\big]\|\hat{w}^{k+1}-(1-\tau_{k})\tilde{w}^{k}\|^{2}.\end{array}\hskip-12.91663pt If we choose \eta and r such that r\in(0,1] , M_{c}\eta^{2}\leq 1 , and \begin{array}[]{lcl}\omega\geq\frac{\kappa(c+rM_{c}\eta^{2})}{rc}\quad\text{and}\quad M_{c}\eta^{2}+\frac{\hat{\kappa}\eta^{2}(c+rM_{c}\eta^{2})}{rc}\leq 1,\end{array} (119) then the last estimate leads to \mathcal{L}_{k+1}\leq\mathcal{L}_{k} for all k\geq 0 . The first condition of ( 119 ) holds if 0<\eta\leq\frac{\sqrt{c(r\omega-\kappa)}}{\sqrt{r\kappa M_{c}}} , where \kappa\leq r\omega due to the choice of \omega . The second condition of ( 119 ) holds if (M_{c}+\hat{\kappa}\omega)\eta^{2}\leq 1 , which is equivalent to \eta\leq\frac{1}{\sqrt{M_{c}+\hat{\kappa}\omega}} . Now, for any r\in(0,1] , let us choose \omega:=\frac{2\kappa}{r} and c:=r . Then, we have M_{c}+\hat{\kappa}\omega\geq\frac{r\kappa M_{c}}{c(r\omega-\kappa)}=M_{c} . Therefore, if we choose 0<\eta\leq\bar{\eta}:=\frac{\sqrt{r}}{\sqrt{(1+r)(r+2\kappa)L^{2}+2\kappa\hat{\kappa}}} , then \eta satisfies both conditions above. Clearly, this \bar{\eta} is given by ( 15 ). Finally, since b_{k+1}=\frac{b_{k}}{1-\tau_{k}}=\frac{b_{k}(k+\nu)}{k+\nu-1} , by induction, we obtain b_{k}=\frac{b_{0}(k+\nu-1)}{\nu-1} for some b_{0}>0 . Moreover, we also have \begin{array}[]{lcl}a_{k}=\frac{\eta b_{0}(k+\nu-1)^{2}}{\nu-1}\quad\text{and}\quad c_{k}=\frac{\eta b_{0}(c+rM_{c}\eta^{2})(k+\nu-1)^{2}}{rc(\nu-1)}.\end{array} Using these expressions, \langle w^{k},x^{k}-x^{\star}\rangle\geq 0 for any x^{\star}\in\mathrm{zer}(\Phi) , and Young’s inequality in {\tiny\textcircled{1}} , we can lower bound \mathcal{V}_{k} from ( 11 ) as \begin{array}[]{lcl}\mathcal{V}_{k}&=&\frac{b_{0}\eta(k+\nu-1)^{2}}{2(\nu-1)}\|w^{k}\|^{2}+\frac{b_{0}(k+\nu-1)}{\nu-1}\big[\langle w^{k},x^{\star}-x^{0}\rangle+\langle w^{k},x^{k}-x^{\star}\rangle\big]\\ &&+{~}\frac{b_{0}}{\eta(\nu-1)}\|x^{0}-x^{\star}\|^{2}\\ &\overset{\tiny{\tiny\textcircled{1}}}{\geq}&\frac{b_{0}\eta(k+\nu-1)^{2}}{4(\nu-1)}\|w^{k}\|^{2}.\end{array} Since we have chosen u^{0}:=Fx^{0} , from ( 11 ), we have \mathcal{L}_{0}:=\mathcal{V}_{0}+\frac{c_{0}}{2}\|Fx^{0}-u^{0}\|^{2}=\frac{\eta b_{0}}{4(\nu-1)}\|w^{0}\|^{2}+\frac{b_{0}}{\eta(\nu-1)}\|x^{0}-x^{\star}\|^{2}=\frac{b_{0}}{4\eta(\nu-1)}\big[\eta^{2}\|w^{0}\|^{2}+4\|x^{0}-x^{\star}\|^{2}\big] . Putting these bounds together, we can derive that \begin{array}[]{lcl}\frac{b_{0}\eta(k+\nu-1)^{2}}{4(\nu-1)}\|w^{k}\|^{2}&\leq&\mathcal{V}_{k}\leq\mathcal{L}_{k}\leq\mathcal{L}_{0}=\frac{b_{0}}{4\eta(\nu-1)}\big[\eta^{2}\|w^{0}\|^{2}+4\|x^{0}-x^{\star}\|^{2}\big],\end{array} which is exactly ( 16 ) after substituting w^{0}:=Fx^{0}+\xi^{0} and w^{k}:=Fx^{k}+\xi^{k} for \xi^{0}\in Tx^{0} and \xi^{k}\in Tx^{k} , respectively. \square Appendix C The Proof of Technical Results in Section 4 This appendix provides the full proof of Lemmas and Theorems in Section 4 . C.1 The Proof of Lemma 2 First, let us denote e^{k}:=u^{k}-Fx^{k}=z^{k}-w^{k} . Then, from the second line of ( GFEG ), we can easily show that \left\{\begin{array}[]{lcl}x^{k+1}-x^{k}&=&-\frac{\tau_{k}}{1-\tau_{k}}(x^{k+1}-x^{0})-\frac{\eta}{1-\tau_{k}}\hat{w}^{k+1}+\frac{\beta_{k}}{1-\tau_{k}}w^{k}+\frac{\beta_{k}}{1-\tau_{k}}e^{k},\\ x^{k+1}-x^{k}&=&-\tau_{k}(x^{k}-x^{0})-\eta\hat{w}^{k+1}+\beta_{k}w^{k}+\beta_{k}e^{k}.\end{array}\right. (120) Next, since \Phi is \rho -co-hypomonotone, we have \langle w^{k+1},x^{k+1}-x^{k}\rangle-\langle w^{k},x^{k+1}-x^{k}\rangle\geq-\rho\|w^{k+1}-w^{k}\|^{2} . Substituting ( 120 ) into this inequality, we can derive \begin{array}[]{lcl}\tau_{k}\langle w^{k},x^{k}-x^{0}\rangle&-&\frac{\tau_{k}}{1-\tau_{k}}\langle w^{k+1},x^{k+1}-x^{0}\rangle\geq-\rho\|w^{k+1}-w^{k}\|^{2}\\ &&+{~}\frac{\eta}{1-\tau_{k}}\langle w^{k+1},\hat{w}^{k+1}\rangle-\frac{\beta_{k}}{1-\tau_{k}}\langle w^{k+1},w^{k}\rangle-\eta\langle w^{k},\hat{w}^{k+1}\rangle\\ &&+{~}\beta_{k}\|w^{k}\|^{2}-\frac{\beta_{k}}{1-\tau_{k}}\langle w^{k+1}-(1-\tau_{k})w^{k},e^{k}\rangle.\end{array} Now, multiplying this inequality by \frac{b_{k}}{\tau_{k}} and using b_{k+1}=\frac{b_{k}}{1-\tau_{k}} , and then applying Young’s inequality \langle u,v\rangle\leq\frac{\gamma}{2}\|u\|^{2}+\frac{1}{2\gamma}\|v\|^{2} for \gamma>0 to {\tiny\textcircled{1}} , we get \begin{array}[]{lcl}\hat{\mathcal{T}}_{[1]}&:=&b_{k}\langle w^{k},x^{k}-x^{0}\rangle-b_{k+1}\langle w^{k+1},x^{k+1}-x^{0}\rangle\\ &\overset{\tiny{\tiny\textcircled{1}}}{\geq}&\frac{\eta b_{k}}{\tau_{k}(1-\tau_{k})}\langle w^{k+1},\hat{w}^{k+1}\rangle-\frac{b_{k}\beta_{k}}{\tau_{k}(1-\tau_{k})}\langle w^{k+1},w^{k}\rangle-\frac{\eta b_{k}}{\tau_{k}}\langle w^{k},\hat{w}^{k+1}\rangle\\ &&+{~}\frac{b_{k}\beta_{k}}{\tau_{k}}\|w^{k}\|^{2}-\frac{b_{k}\rho}{\tau_{k}}\|w^{k+1}-w^{k}\|^{2}-\frac{b_{k}\beta_{k}}{2\gamma\tau_{k}(1-\tau_{k})}\|e^{k}\|^{2}\\ &&-{~}\frac{\gamma b_{k}\beta_{k}}{2\tau_{k}(1-\tau_{k})}\|w^{k+1}-(1-\tau_{k})w^{k}\|^{2}.\end{array} Then, utilizing the following two elementary identities: \begin{array}[]{lcl}2\langle w^{k+1},w^{k}\rangle&=&\|w^{k+1}\|^{2}+\|w^{k}\|^{2}-\|w^{k+1}-w^{k}\|^{2},\\ \|w^{k+1}\!-\!(1-\tau_{k})w^{k}\|^{2}&=&\tau_{k}\|w^{k+1}\|^{2}-\tau_{k}(1-\tau_{k})\|w^{k}\|^{2}+(1\!-\!\tau_{k})\|w^{k+1}\!-\!w^{k}\|^{2},\end{array} the last inequality leads to \hskip-8.61108pt\begin{array}[]{lcl}\hat{\mathcal{T}}_{[1]}&:=&b_{k}\langle w^{k},x^{k}-x^{0}\rangle-b_{k+1}\langle w^{k+1},x^{k+1}-x^{0}\rangle\\ &\geq&\frac{\eta b_{k}}{\tau_{k}(1-\tau_{k})}\langle w^{k+1}-w^{k},\hat{w}^{k+1}\rangle+\frac{\eta b_{k}}{1-\tau_{k}}\langle w^{k},\hat{w}^{k+1}\rangle\\ &&-{~}\frac{b_{k}\beta_{k}(1+\gamma\tau_{k})}{2\tau_{k}(1-\tau_{k})}\|w^{k+1}\|^{2}+\frac{b_{k}\beta_{k}[1-2\tau_{k}+\gamma\tau_{k}(1-\tau_{k})]}{2\tau_{k}(1-\tau_{k})}\|w^{k}\|^{2}\\ &&+{~}\frac{b_{k}[\beta_{k}(1-\gamma+\gamma\tau_{k})-2\rho(1-\tau_{k})]}{2\tau_{k}(1-\tau_{k})}\|w^{k+1}-w^{k}\|^{2}-\frac{b_{k}\beta_{k}}{2\gamma\tau_{k}(1-\tau_{k})}\|e^{k}\|^{2}.\end{array}\hskip-8.61108pt (121) Since x^{k+1}-y^{k}=\hat{\eta}_{k}w^{k}-\eta\hat{w}^{k+1}+\hat{\eta}_{k}e^{k} from ( GFEG ), by the Lipschitz continuity of F and Young’s inequality in {\tiny\textcircled{2}} , for any r>0 , we have \begin{array}[]{lcl}\|w^{k+1}-\hat{w}^{k+1}\|^{2}&=&\|Fx^{k+1}-Fy^{k}\|^{2}\leq L^{2}\|x^{k+1}-y^{k}\|^{2}\\ &=&L^{2}\|\hat{\eta}_{k}w^{k}-\eta\hat{w}^{k+1}+\hat{\eta}_{k}e^{k}\|^{2}\\ &\overset{\tiny{\tiny\textcircled{2}}}{\leq}&(1+r)L^{2}\|\eta\hat{w}^{k+1}-\hat{\eta}_{k}w^{k}\|^{2}+\frac{(1+r)L^{2}\hat{\eta}_{k}^{2}}{r}\|e^{k}\|^{2}.\end{array} Since \|w^{k+1}-\hat{w}^{k+1}\|^{2}=\mu\|w^{k+1}-\hat{w}^{k+1}\|^{2}+(1-\mu)\|w^{k+1}-\hat{w}^{k+1}\|^{2} for any \mu\in[0,1] , partly expanding the last estimate yields \begin{array}[]{lcl}\mu\|w^{k+1}\|^{2}&+&\mu\|\hat{w}^{k+1}\|^{2}-2\mu\langle w^{k+1},\hat{w}^{k+1}\rangle+(1-\mu)\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &\leq&(1+r)L^{2}\eta^{2}\|\hat{w}^{k+1}\|^{2}+(1+r)L^{2}\hat{\eta}_{k}^{2}\|w^{k}\|^{2}\\ &&-{~}2(1+r)L^{2}\eta\hat{\eta}_{k}\langle w^{k},\hat{w}^{k+1}\rangle+\frac{(1+r)L^{2}\hat{\eta}_{k}^{2}}{r}\|e^{k}\|^{2}.\end{array} Rearranging this inequality, and multiplying the result by \frac{\eta b_{k}}{2\mu\tau_{k}(1-\tau_{k})} , we get \begin{array}[]{lcl}\frac{(1+r)L^{2}\eta b_{k}\hat{\eta}_{k}^{2}}{2r\mu\tau_{k}(1-\tau_{k})}\|e^{k}\|^{2}&\geq&\frac{(1-\mu)\eta b_{k}}{2\mu\tau_{k}(1-\tau_{k})}\|w^{k+1}-\hat{w}^{k+1}\|^{2}-\frac{(1+r)L^{2}\eta b_{k}\hat{\eta}_{k}^{2}}{2\mu\tau_{k}(1-\tau_{k})}\|w^{k}\|^{2}\\ &&+{~}\frac{\eta b_{k}}{2\tau_{k}(1-\tau_{k})}\|w^{k+1}\|^{2}+\frac{\eta b_{k}[\mu-(1+r)L^{2}\eta^{2}]}{2\mu\tau_{k}(1-\tau_{k})}\|\hat{w}^{k+1}\|^{2}\\ &&-{~}\frac{\eta b_{k}[\mu-(1+r)L^{2}\eta\hat{\eta}_{k}]}{\mu\tau_{k}(1-\tau_{k})}\langle w^{k},\hat{w}^{k+1}\rangle\\ &&-{~}\frac{\eta b_{k}}{\tau_{k}(1-\tau_{k})}\langle w^{k+1}-w^{k},\hat{w}^{k+1}\rangle.\end{array} Adding the last inequality to \hat{\mathcal{T}}_{[1]} in ( 121 ), we can derive that \begin{array}[]{lcl}\hat{\mathcal{T}}_{[2]}&:=&b_{k}\langle w^{k},x^{k}-x^{0}\rangle-b_{k+1}\langle w^{k+1},x^{k+1}-x^{0}\rangle\\ &&+{~}\frac{b_{k}}{2\tau_{k}(1-\tau_{k})}\big[\frac{(1+r)L^{2}\eta\hat{\eta}_{k}^{2}}{r\mu}+\frac{\beta_{k}}{\gamma}\big]\|e^{k}\|^{2}\\ &\geq&\frac{b_{k}[\eta-\beta_{k}(1+\gamma\tau_{k})]}{2\tau_{k}(1-\tau_{k})}\|w^{k+1}\|^{2}+\frac{b_{k}[\mu\beta_{k}(1-2\tau_{k}+\gamma\tau_{k}(1-\tau_{k}))-(1+r)L^{2}\eta\hat{\eta}_{k}^{2}]}{2\mu\tau_{k}(1-\tau_{k})}\|w^{k}\|^{2}\\ &&+{~}\frac{\eta b_{k}[\mu-(1+r)L^{2}\eta^{2}]}{2\mu\tau_{k}(1-\tau_{k})}\|\hat{w}^{k+1}\|^{2}-\frac{\eta b_{k}[\mu(1-\tau_{k})-(1+r)L^{2}\eta\hat{\eta}_{k}]}{\mu\tau_{k}(1-\tau_{k})}\langle w^{k},\hat{w}^{k+1}\rangle\\ &&+{~}\frac{(1-\mu)\eta b_{k}}{2\mu\tau_{k}(1-\tau_{k})}\|w^{k+1}-\hat{w}^{k+1}\|^{2}+\frac{b_{k}[\beta_{k}(1-\gamma+\gamma\tau_{k})-2\rho(1-\tau_{k})]}{2\tau_{k}(1-\tau_{k})}\|w^{k+1}-w^{k}\|^{2}.\end{array} Finally, substituting \hat{\eta}_{k}=\eta(1-\tau_{k}) and b_{k+1}=\frac{b_{k}}{1-\tau_{k}} into \hat{\mathcal{T}}_{[2]} , then using \begin{array}[]{lcl}2(1-\tau_{k})\langle w^{k},\hat{w}^{k+1}\rangle=(1-\tau_{k})^{2}\|w^{k}\|^{2}+\|\hat{w}^{k+1}\|^{2}-\|\hat{w}^{k+1}-(1-\tau_{k})w^{k}\|^{2}\end{array} and rearranging the result, we eventually arrive at \begin{array}[]{lcl}\hat{\mathcal{T}}_{[3]}&:=&\frac{b_{k}[\eta(1-\tau_{k})^{2}-\beta_{k}(1-2\tau_{k}+\gamma\tau_{k}(1-\tau_{k}))]}{2\tau_{k}(1-\tau_{k})}\|w^{k}\|^{2}+b_{k}\langle w^{k},x^{k}-x^{0}\rangle\\ &&+{~}\frac{b_{k}[(1+r)L^{2}\eta^{3}\gamma(1-\tau_{k})^{2}+r\mu\beta_{k}]}{2r\mu\gamma\tau_{k}(1-\tau_{k})}\|e^{k}\|^{2}\\ &\geq&\frac{b_{k+1}[\eta-\beta_{k}(1+\gamma\tau_{k})]}{2\tau_{k}}\|w^{k+1}\|^{2}+b_{k+1}\langle w^{k+1},x^{k+1}-x^{0}\rangle\\ &&+{~}\frac{(1-\mu)\eta b_{k+1}}{2\mu\tau_{k}}\|w^{k+1}-\hat{w}^{k+1}\|^{2}+\frac{\eta b_{k+1}[\mu-(1+r)L^{2}\eta^{2}]}{2\mu\tau_{k}}\|\hat{w}^{k+1}-(1-\tau_{k})w^{k}\|^{2}\\ &&+{~}\frac{b_{k+1}[\beta_{k}(1-\gamma+\gamma\tau_{k})-2\rho(1-\tau_{k})]}{2\tau_{k}}\|w^{k+1}-w^{k}\|^{2}.\end{array} Using the notations from ( 27 ) in \hat{\mathcal{T}}_{[3]} and rearranging the result yields ( 29 ). \square C.2 The Proof of Lemma 3 Substituting ( 21 ) into ( 29 ) with a notice that w^{k}-\hat{w}^{k}=Fx^{k}-Fy^{k-1} , and using ( 30 ), we can derive \begin{array}[]{lcl}\mathcal{L}_{k}&\overset{\tiny\eqref{eq:FEG4NI_Lyapunov_func}}{=}&\frac{a_{k}}{2}\|w^{k}\|^{2}+b_{k}\langle w^{k},x^{k}-x^{0}\rangle+\frac{c_{k}}{2}\|Fx^{k}-u^{k}\|^{2}\\ &\overset{\tiny\eqref{eq:FEG4NI_key_est2_cond}}{\geq}&\frac{\hat{a}_{k}}{2}\|w^{k}\|^{2}+b_{k}\langle w^{k},x^{k}-x^{0}\rangle+\frac{c_{k}}{2}\|Fx^{k}-u^{k}\|^{2}\\ &\overset{\tiny\eqref{eq:FEG4NI_key_estimate1}}{\geq}&\frac{a_{k+1}}{2}\|w^{k+1}\|^{2}+b_{k+1}\langle w^{k+1},x^{k+1}-x^{0}\rangle+\frac{\alpha_{k+1}}{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\frac{\hat{\alpha}_{k+1}}{2}\|w^{k+1}-w^{k}\|^{2}+\frac{\delta_{k+1}}{2}\|\hat{w}^{k+1}-(1-\tau_{k})w^{k}\|^{2}\\ &\overset{\tiny\eqref{eq:FEG4NI_key_est2_cond}}{\geq}&\frac{a_{k+1}}{2}\|w^{k+1}\|^{2}+b_{k+1}\langle w^{k+1},x^{k+1}-x^{0}\rangle+\frac{\kappa c_{k+1}}{2}\|Fx^{k+1}-Fy^{k}\|^{2}\\ &&+{~}\frac{\hat{\kappa}c_{k+1}}{2}\|w^{k+1}-w^{k}\|^{2}+\frac{\delta_{k+1}}{2}\|\hat{w}^{k+1}-(1-\tau_{k})w^{k}\|^{2}\\ &\overset{\tiny\eqref{eq:FEG4NI_u_cond}}{\geq}&\frac{a_{k+1}}{2}\|w^{k+1}\|^{2}+b_{k+1}\langle w^{k+1},x^{k+1}-x^{0}\rangle+\frac{c_{k+1}}{2}\|Fx^{k+1}-u^{k+1}\|^{2}\\ &&+{~}\frac{\delta_{k+1}}{2}\|\hat{w}^{k+1}-(1-\tau_{k})w^{k}\|^{2}\\ &\overset{\tiny\eqref{eq:FEG4NI_Lyapunov_func}}{=}&\mathcal{L}_{k+1}+\frac{\delta_{k+1}}{2}\|\hat{w}^{k+1}-(1-\tau_{k})w^{k}\|^{2},\end{array} which proves ( 31 ). \square C.3 The Proof of Lemma 4 Let us choose \tau_{k}:=\frac{1}{k+\nu} for some \nu>1 and \beta_{k}:=\beta(1-\tau_{k}) as in ( 32 ). We verify the first condition a_{k}\geq\hat{a}_{k} of ( 30 ), which is equivalent to \begin{array}[]{lcl}\eta\big(\frac{1}{\tau_{k-1}}-\frac{1}{\tau_{k}}+1\big)\geq\beta_{k-1}\big(\frac{1}{\tau_{k-1}}+\gamma\big)-\beta_{k}\big[\frac{1-2\tau_{k}}{\tau_{k}(1-\tau_{k})}+\gamma\big].\end{array} Since \tau_{k}:=\frac{1}{k+\nu} and \beta_{k}:=\beta(1-\tau_{k}) , this condition automatically holds. We still need \hat{a}_{k}\geq 0 , which is guaranteed if \eta\geq\beta\big(1-\frac{\tau_{k}}{1-\tau_{k}}+\gamma\tau_{k}\big) . Since this holds for all k\geq 0 , we impose a stricter condition \beta\leq\frac{\nu\eta}{\nu+\gamma} as in ( 33 ). Next, the second condition \alpha_{k}\geq\kappa c_{k} of ( 30 ) is equivalent to \begin{array}[]{lcl}\beta\leq\frac{\gamma\eta(k+\nu-1)}{\kappa\mu(k+\nu)}\Big[1-\mu-\frac{\kappa(1+r)L^{2}\eta^{2}}{r}\Big],\end{array} provided that \kappa(1+r)L^{2}\eta^{2}<(1-\mu)r . Since this condition must hold for all k\geq 0 , we impose \beta\leq\frac{\gamma\eta(\nu-1)}{\kappa\mu\nu}\left[1-\mu-\frac{\kappa(1+r)L^{2}\eta^{2}}{r}\right] as in ( 33 ). Finally, the third condition \hat{\alpha}_{k}\geq\kappa c_{k} of ( 30 ) becomes \begin{array}[]{lcl}\beta\Big[1-\gamma+\gamma\tau_{k-1}-\frac{\hat{\kappa}\tau_{k-1}}{\gamma\tau_{k}(1-\tau_{k-1})}\Big]\geq\frac{\hat{\kappa}(1+r)L^{2}\eta^{3}}{r\mu}\cdot\frac{(1-\tau_{k})\tau_{k-1}}{(1-\tau_{k-1})\tau_{k}}+2\rho.\end{array} Since it must hold for all k\geq 0 , we impose \begin{array}[]{lcl}\beta&\geq&\frac{(\nu-2)\gamma}{(\nu-2)\gamma(1-\gamma)-\nu\hat{\kappa}}\Big[\frac{(\nu-1)\hat{\kappa}(1+r)L^{2}\eta^{3}}{(\nu-2)r\mu}+2\rho\Big],\end{array} provided that (\nu-2)\gamma(1-\gamma)>\nu\hat{\kappa} . This is the first condition in ( 33 ), which guarantees the third condition of ( 30 ) to hold. Overall, we have shown that \tau_{k} and \beta_{k} chosen by ( 32 ) satisfy ( 30 ), where \beta satisfies ( 33 ). Therefore, we still obtain ( 31 ). \square C.4 The Proof of Theorem 4.1 First, for the right-hand side of ( 33 ) to be well-defined and for \delta_{k+1}\geq 0 in ( 31 ), we need to impose the following conditions: \begin{array}[]{ll}L^{2}\eta^{2}\leq\frac{(1-\mu)r}{(1+r)\kappa},\quad L^{2}\eta^{2}\leq\frac{\mu}{1+r},\ \ \text{and}\ \ \nu\hat{\kappa}<(\nu-2)(1-\gamma)\gamma.\end{array} (122) Furthermore, for \nu>2 , we can impose a stricter condition \beta<\frac{\eta}{1+\gamma}<\frac{\nu\eta}{\nu+\gamma} to guarantee the third line of ( 33 ). Now, we consider the following two cases. Case 1. If \kappa=0 , then the first condition of ( 122 ) automatically holds. The second condition of ( 122 ) becomes \eta\leq\frac{\sqrt{\mu}}{L\sqrt{1+r}} . Therefore, we consider the following two sub-cases: (i) If \hat{\kappa}=0 , then the last condition of ( 122 ) holds automatically. Moreover, we can choose \begin{array}[]{ll}\frac{2\rho}{1-\gamma}\leq\beta<\frac{\eta}{1+\gamma}\end{array} to guarantee ( 33 ) in Lemma 4 . Let \gamma\to 0^{+} , r\to 0^{+} and \mu=1 , the last two conditions reduce to 2\rho\leq\beta<\eta . However, since \eta\leq\frac{\sqrt{\mu}}{L\sqrt{1+r}}=\frac{1}{L} , we finally get 2\rho\leq\beta<\eta\leq\frac{1}{L} , provided that 2L\rho\leq 1 . (ii) If \hat{\kappa}>0 , then the last condition of ( 122 ) becomes 0<\hat{\kappa}<\frac{(\nu-2)(1-\gamma)\gamma}{\nu} . We can choose \gamma:=\frac{1}{2} to obtain 0<\hat{\kappa}<\frac{\nu-2}{4\nu} , provided that \nu>2 . In this case, if we choose \mu=1 and r=1 , then ( 33 ) in Lemma 4 holds if \begin{array}[]{ll}\frac{2(\nu-2)}{\nu-2-4\nu\hat{\kappa}}\big[\frac{2(\nu-1)\hat{\kappa}L^{2}\eta^{3}}{\nu-2}+2\rho\big]\leq\beta<\frac{2\eta}{3}<\frac{2\nu\eta}{2\nu+1},\end{array} leading to \begin{array}[]{lcl}2\rho<\frac{(\nu-2-4\nu\hat{\kappa})\eta}{3(\nu-2)}-\frac{2(\nu-1)\hat{\kappa}L^{2}\eta^{3}}{\nu-2}=\frac{[(\nu-2-4\nu\hat{\kappa})-6(\nu-1)\hat{\kappa}L^{2}\eta^{2}]\eta}{3(\nu-2)}.\end{array} If we impose L^{2}\eta^{2}\leq\frac{\nu-2-4\nu\hat{\kappa}}{12(\nu-1)\hat{\kappa}} , then to guarantee that \eta\leq\frac{\sqrt{\mu}}{L\sqrt{1+r}}=\frac{1}{L\sqrt{2}} , we require \eta\leq\frac{\sigma_{1}}{L} , where \sigma_{1}^{2}:=\min\left\{\frac{1}{2},\frac{\nu-2-4\nu\hat{\kappa}}{12(\nu-1)\hat{\kappa}}\right\} . Using this bound, the last inequality holds if 2\rho<\frac{(\nu-2-4\nu\hat{\kappa})\sigma_{1}}{6(\nu-2)L} , or equivalently, L\rho<\frac{(\nu-2-4\nu\hat{\kappa})\sigma_{1}}{12(\nu-2)} . Moreover, we can choose \begin{array}[]{ll}\frac{2(\nu-2)}{\nu-2-4\nu\hat{\kappa}}\big[\frac{2(\nu-1)\hat{\kappa}\eta\sigma_{1}^{2}}{\nu-2}+2\rho\big]\leq\beta<\frac{2\eta}{3}.\end{array} Case 2. If \kappa>0 , then we can choose r=1 and \mu:=\frac{r}{\kappa+r}=\frac{1}{\kappa+1} such that L^{2}\eta^{2}\leq\frac{(1-\mu)r}{(1+r)\kappa}=\frac{\mu}{1+r}=\frac{r}{(1+r)(\kappa+r)} . Hence, we get \eta\leq\frac{\sqrt{r}}{L\sqrt{(1+r)(\kappa+r)}}=\frac{1}{L\sqrt{2(\kappa+1)}} . We instead impose the following condition \eta\leq\frac{1}{2L\sqrt{\kappa+1}} , which is stricter than the last one. Then, we have 1-\mu-\frac{(1+r)\kappa L^{2}\eta^{2}}{r}\geq\frac{\kappa}{2(\kappa+1)} . Now, we consider the following two sub-cases. (i) If \hat{\kappa}=0 , then the last condition of ( 122 ) automatically holds. In addition, we can choose \gamma:=\frac{1}{2} , and thus ( 33 ) of Lemma 4 holds if \begin{array}[]{ll}4\rho\leq\beta\leq\frac{(\nu-1)\eta}{4\nu}<\frac{2\eta}{3}<\frac{2\nu\eta}{2\nu+1}.\end{array} Moreover, since \eta\leq\frac{1}{2L\sqrt{\kappa+1}} , we eventually get 4\rho\leq\beta\leq\frac{(\nu-1)\eta}{4\nu}\leq\frac{1}{8L\nu\sqrt{\kappa+1}} , provided that L\rho\leq\frac{\nu-1}{32\nu\sqrt{\kappa+1}} . (ii) If \hat{\kappa}>0 , then the last condition ( 122 ) imposes that 0<\hat{\kappa}<\frac{(\nu-2)(1-\gamma)\gamma}{\nu} . Again, we can choose \gamma:=\frac{1}{2} to obtain 0<\hat{\kappa}<\frac{\nu-2}{4\nu} , provided that \nu>2 . In this case, ( 33 ) of Lemma 4 holds if \begin{array}[]{ll}\frac{2(\nu-2)}{\nu-2-4\nu\hat{\kappa}}\big[\frac{2(\nu-1)(\kappa+1)\hat{\kappa}L^{2}\eta^{3}}{\nu-2}+2\rho\big]\leq\beta\leq\frac{(\nu-1)\eta}{4\nu}<\frac{2\eta}{3}<\frac{2\nu\eta}{2\nu+1},\end{array} leading to \begin{array}[]{ll}2\rho\leq\frac{(\nu-1)(\nu-2-4\nu\hat{\kappa})\eta}{8\nu(\nu-2)}-\frac{2(\nu-1)(\kappa+1)\hat{\kappa}L^{2}\eta^{3}}{\nu-2}=\frac{(\nu-1)\left[\nu-2-4\nu\hat{\kappa}-16\nu(\kappa+1)\hat{\kappa}L^{2}\eta^{2}\right]\eta}{8\nu(\nu-2)}.\end{array} If we enforce L^{2}\eta^{2}\leq\frac{\nu-2-4\nu\hat{\kappa}}{32\nu(\kappa+1)\hat{\kappa}} , then to guarantee \eta\leq\frac{1}{2L\sqrt{\kappa+1}} , we require \eta\leq\frac{\sigma_{2}}{L} , where \sigma_{2}^{2}:=\frac{1}{\kappa+1}\min\left\{\frac{1}{2},\frac{\nu-2-4\nu\hat{\kappa}}{32\nu\hat{\kappa}}\right\} . Using this bound, the last inequality holds if 2\rho\leq\frac{(\nu-1)(\nu-2-4\nu\hat{\kappa})\sigma_{2}}{16\nu(\nu-2)L} , or equivalently, L\rho\leq\frac{(\nu-1)(\nu-2-4\nu\hat{\kappa})\sigma_{2}}{32\nu(\nu-2)} . Moreover, we can choose \begin{array}[]{ll}\frac{2(\nu-2)}{\nu-2-4\nu\hat{\kappa}}\big[\frac{2(\nu-1)(\kappa+1)\hat{\kappa}\eta\sigma_{2}^{2}}{\nu-2}+2\rho\big]\leq\beta\leq\frac{(\nu-1)\eta}{4\nu}.\end{array} Overall, under the choice of parameters as discussed in the two cases above, we can guarantee the conditions in ( 31 ) of Lemma 4 . Moreover, in all cases, we also have \beta<\frac{\eta}{1+\gamma} , or equivalently, \eta-(1+\gamma)\beta>0 . Next, using the update rules ( 34 ), we have \beta_{k}=\frac{\beta(k+\nu-1)}{k+\nu} and b_{k}=\frac{b_{0}(k+\nu-1)}{\nu-1} for all k\geq 0 . Furthermore, we can easily show that \begin{array}[]{lcl}a_{k}&:=&\frac{b_{k}[\eta-\beta_{k-1}(1+\gamma\tau_{k-1})]}{\tau_{k-1}}\geq\frac{b_{0}[\eta-(1+\gamma)\beta](k+\nu-1)^{2}}{(s-1)}+2\rho b_{k}.\end{array} Since x^{\star}\in\mathrm{zer}(\Phi) , and w^{k}\in\Phi x^{k}=Fx^{k}+Tx^{k} , we have \langle w^{k},x^{k}-x^{\star}\rangle\geq-\rho\|w^{k}\|^{2} due to the \rho -co-hypomonotonicity of \Phi . Using this bound and the elementary inequality uv\leq su^{2}+\frac{1}{4s}v^{2} for some s>0 , we can show that \begin{array}[]{lcllcl}\hat{\mathcal{P}}_{k}&:=&\frac{a_{k}}{2}\|w^{k}\|^{2}+b_{k}\langle w^{k},x^{k}-x^{0}\rangle\\ &=&\frac{a_{k}}{2}\|w^{k}\|^{2}+b_{k}\langle w^{k},x^{\star}-x^{0}\rangle+b_{k}\langle w^{k},x^{k}-x^{\star}\rangle\\ &\geq&(\frac{a_{k}}{2}-\rho b_{k})\|w^{k}\|^{2}-b_{k}\|w^{k}\|\|x^{0}-x^{\star}\|\\ &\geq&\left(\frac{a_{k}}{2}-\rho b_{k}-\frac{[\eta-(1+\gamma)\beta]b_{k}^{2}}{4b_{0}}\right)\|w^{k}\|^{2}-\frac{b_{0}}{\eta-(1+\gamma)\beta}\|x^{0}-x^{\star}\|^{2}\\ &\geq&\frac{b_{0}[\eta-(1+\gamma)\beta](k+\nu-1)^{2}}{4(\nu-1)}\|w^{k}\|^{2}-\frac{b_{0}}{\eta-(1+\gamma)\beta}\|x^{0}-x^{\star}\|^{2}.\end{array} (123) Here, we have used a_{k}\geq\frac{b_{0}[\eta-(1+\gamma)\beta](k+\nu-1)^{2}}{(\nu-1)}+2\rho b_{k} and b_{k}:=\frac{b_{0}(k+\nu-1)}{\nu-1} in the last equality. Finally, from ( 31 ) of Lemma 3 , by induction, we have \hat{\mathcal{P}}_{k}+\tfrac{c_{k}}{2}\|Fx^{k}-u^{k}\|^{2}=\mathcal{L}_{k}\leq\mathcal{L}_{0}. Due to the choice of y^{-1}=x^{-1}=x^{0} and u^{0}:=Fx^{0} , we get \begin{array}[]{lcl}\mathcal{L}_{0}=\frac{a_{0}}{2}\|Fx^{0}+\xi^{0}\|^{2}=\frac{b_{0}\left[\eta(\nu-1)^{2}-\beta(\nu-2)(\nu+\gamma-1)\right]}{\nu-1}\|Fx^{0}+\xi^{0}\|^{2}.\end{array} Therefore, we eventually obtain \begin{array}[]{lcl}\mathcal{L}_{k}\leq\frac{b_{0}\left[\eta(\nu-1)^{2}-\beta(\nu-2)(\nu+\gamma-1)\right]}{\nu-1}\|Fx^{0}+\xi^{0}\|^{2}.\end{array} We also note that \begin{array}[]{lcl}c_{k}&=&\frac{b_{k}[(1+r)L^{2}\eta^{3}\gamma(1-\tau_{k})^{2}+r\mu\beta_{k}]}{r\mu\gamma\tau_{k}(1-\tau_{k})}=\frac{b_{0}(k+\nu-1)[(1+r)L^{2}\eta^{3}\gamma(k+\nu-1)+r\mu\beta(k+\nu)]}{r\mu\gamma(\nu-1)}\\ &\geq&\frac{b_{0}[(1+r)L^{2}\eta^{3}\gamma+r\mu\beta](k+\nu-1)^{2}}{r\mu\gamma(\nu-1)}=b_{0}C_{0}(k+\nu-1)^{2},\end{array} where C_{0}:=\frac{[(1+r)L^{2}\eta^{3}\gamma+r\mu\beta]}{r\mu\gamma(\nu-1)} . We further note that since either r=0 or r=1 , \gamma=0 or \gamma=\frac{1}{2} , and \mu=1 or \mu=\frac{1}{1+\kappa} , we can set C_{0}:=0 if r=0 (corresponding to Case (i)), and evaluate C_{0}\geq\frac{2(L^{2}\eta^{3}+\beta)}{s-1} if r=1 (corresponding to the other cases). Combining \mathcal{L}_{k}\leq\frac{b_{0}\left[\eta(\nu-1)^{2}-\beta(\nu-2)(\nu+\gamma-1)\right]}{\nu-1}\|Fx^{0}+\xi^{0}\|^{2} and ( 123 ), and using the above coefficients a_{k} , \psi , and b_{0}>0 , we can show that \begin{array}[]{lcl}\|Fx^{k}+\xi^{k}\|^{2}+\psi\|Fx^{k}-u^{k}\|^{2}&\leq&\frac{1}{(k+\nu-1)^{2}}\cdot\Big[\frac{4(\nu-1)\|x^{0}-x^{\star}\|^{2}}{[\eta-(1+\gamma)\beta]^{2}}\\ &&+{~}\frac{4[\eta(\nu-1)^{2}-\beta(\nu-2)(\nu+\gamma-1)]}{[\eta-(1+\gamma)\beta]}\|Fx^{0}+\xi^{0}\|^{2}\Big].\end{array} Here, \psi:=0 if \kappa=\hat{\kappa}=0 and \psi:=\frac{4(L^{2}\eta^{3}+\beta)}{\eta-(1+\gamma)\beta} , otherwise. Since \eta-(1+\gamma)\beta>0 , this inequality implies ( 35 ). We note that, for Case (i), we set \gamma=0 , and for other cases, we have used \gamma=\frac{1}{2} . Hence, the final term on the right-hand side of the last inequality reduces to \mathcal{R}_{0}^{2} defined in ( 35 ) of the theorem. \square Appendix D The Proof of Technical Results in Section 5 This appendix presents the full proof of the results in Section 5 . D.1 The Proof of Lemma 5 First, since \tau_{k}:=\frac{1}{t_{k}} and e^{k}:=u^{k}-Fx^{k}=z^{k}-w^{k} , we can rewrite the second line of ( GFEG+ ) as \begin{array}[]{lcl}t_{k}x^{k+1}=\bar{x}^{k}+(t_{k}-1)x^{k}-\eta t_{k}\hat{w}^{k+1}+\beta_{k}t_{k}w^{k}+\beta_{k}t_{k}e^{k}.\end{array} Rearranging this expression in two different ways, and using \bar{x}^{k}=\bar{x}^{k+1}+\gamma z^{k}=\bar{x}^{k+1}+\gamma w^{k}+\gamma e^{k} from the third line of ( GFEG+ ), we get \left\{\begin{array}[]{lcl}t_{k}(t_{k}-1)(x^{k}-x^{k+1})&=&(t_{k}-1)(x^{k}-\bar{x}^{k})+\eta t_{k}(t_{k}-1)\hat{w}^{k+1}\\ &&-{~}\beta_{k}t_{k}(t_{k}-1)w^{k}-\beta_{k}t_{k}(t_{k}-1)e^{k},\\ t_{k}(t_{k}-1)(x^{k}-x^{k+1})&=&t_{k}(x^{k+1}-\bar{x}^{k})+\eta t_{k}^{2}\hat{w}^{k+1}-\beta_{k}t_{k}^{2}w^{k}-\beta_{k}t_{k}^{2}e^{k}\\ &=&t_{k}(x^{k+1}-\bar{x}^{k+1})+\eta t_{k}^{2}\hat{w}^{k+1}-t_{k}(\beta_{k}t_{k}+\gamma)w^{k}\\ &&-{~}t_{k}(\beta_{k}t_{k}+\gamma)e^{k}.\end{array}\right. Next, by the \rho -co-hypomonotonicity of \Phi , we have \begin{array}[]{lcl}\mathcal{T}_{[1]}&:=&t_{k}(t_{k}-1)\langle w^{k},x^{k}-x^{k+1}\rangle-t_{k}(t_{k}-1)\langle w^{k+1},x^{k}-x^{k+1}\rangle\\ &\geq&-\rho t_{k}(t_{k}-1)\|w^{k+1}-w^{k}\|^{2}.\end{array} Substituting the above two expressions into \mathcal{T}_{[1]} , we can show that \begin{array}[]{lcl}\mathcal{T}_{[2]}&:=&(t_{k}-1)\langle w^{k},x^{k}-\bar{x}^{k}\rangle-t_{k}\langle w^{k+1},x^{k+1}-\bar{x}^{k+1}\rangle\\ &\geq&\eta t_{k}^{2}\langle\hat{w}^{k+1},w^{k+1}\rangle-\eta t_{k}(t_{k}-1)\langle\hat{w}^{k+1},w^{k}\rangle-\rho t_{k}(t_{k}-1)\|w^{k+1}-w^{k}\|^{2}\\ &&-{~}t_{k}(\beta_{k}t_{k}+\gamma)\langle w^{k+1},w^{k}\rangle+\beta_{k}t_{k}(t_{k}-1)\|w^{k}\|^{2}\\ &&-{~}t_{k}\langle(\beta_{k}t_{k}+\gamma)w^{k+1}-\beta_{k}(t_{k}-1)w^{k},e^{k}\rangle.\end{array} By Young’s inequality, for any c_{1}>0 and c_{2}>0 , one can prove that \begin{array}[]{lcl}\mathcal{T}_{[3]}&:=&-\langle(\beta_{k}t_{k}+\gamma)w^{k+1}-\beta_{k}(t_{k}-1)w^{k},e^{k}\rangle\\ &=&-(\beta_{k}t_{k}+\gamma)\langle w^{k+1}-w^{k},e^{k}\rangle-(\gamma+\beta_{k})\langle w^{k},e^{k}\rangle\\ &\geq&-c_{1}(\beta_{k}t_{k}+\gamma)\|w^{k+1}-w^{k}\|^{2}-c_{2}(\gamma+\beta_{k})\|w^{k}\|^{2}-\big(\frac{\beta_{k}t_{k}+\gamma}{4c_{1}}+\frac{\gamma+\beta_{k}}{4c_{2}}\big)\|e^{k}\|^{2}\\ &=&-c_{1}(\beta_{k}t_{k}+\gamma)\|w^{k+1}\|^{2}-\big[c_{1}(\beta_{k}t_{k}+\gamma)+c_{2}(\gamma+\beta_{k})\big]\|w^{k}\|^{2}\\ &&+{~}2c_{1}(\beta_{k}t_{k}+\gamma)\langle w^{k+1},w^{k}\rangle-\big(\frac{\beta_{k}t_{k}+\gamma}{4c_{1}}+\frac{\gamma+\beta_{k}}{4c_{2}}\big)\|e^{k}\|^{2}.\end{array} Substituting \mathcal{T}_{[3]} into \mathcal{T}_{[2]} , we can derive that \begin{array}[]{lcl}\mathcal{T}_{[4]}&:=&(t_{k}-1)\langle w^{k},x^{k}-\bar{x}^{k}\rangle-t_{k}\langle w^{k+1},x^{k+1}-\bar{x}^{k+1}\rangle\\ &\geq&\eta t_{k}^{2}\langle\hat{w}^{k+1},w^{k+1}\rangle-\eta t_{k}(t_{k}-1)\langle\hat{w}^{k+1},w^{k}\rangle\\ &&-{~}t_{k}\big[\rho(t_{k}-1)+c_{1}(\beta_{k}t_{k}+\gamma)+c_{2}(\gamma+\beta_{k})-\beta_{k}(t_{k}-1)\big]\|w^{k}\|^{2}\\ &&-{~}t_{k}\big[\rho(t_{k}-1)+c_{1}(\beta_{k}t_{k}+\gamma)\big]\|w^{k+1}\|^{2}\\ &&-{~}t_{k}\big[(1-2c_{1})(\beta_{k}t_{k}+\gamma)-2\rho(t_{k}-1)\big]\langle w^{k+1},w^{k}\rangle\\ &&-{~}\frac{t_{k}}{4}\big(\frac{\beta_{k}t_{k}+\gamma}{c_{1}}+\frac{\gamma+\beta_{k}}{c_{2}}\big)\|e^{k}\|^{2}.\end{array} For any c_{1}\in(0,1/2) , since \beta_{k} is chosen as in ( 46 ), we have (1-2c_{1})(\beta_{k}t_{k}+\gamma)-2\rho(t_{k}-1)=0 . Substituting this expression into \mathcal{T}_{[4]} and using \beta_{k}=\frac{2\rho(t_{k}-1)}{(1-2c_{1})t_{k}}-\frac{\gamma}{t_{k}} from ( 46 ), we can simplify it as \begin{array}[]{lcl}\mathcal{T}_{[4]}&\geq&\eta t_{k}^{2}\langle\hat{w}^{k+1},w^{k+1}\rangle-\eta t_{k}(t_{k}-1)\langle\hat{w}^{k+1},w^{k}\rangle\\ &&-{~}\big[\gamma(1+c_{2})(t_{k}-1)-\frac{\rho(t_{k}-1)[t_{k}-2(1+c_{2})]}{1-2c_{1}}\big]\|w^{k}\|^{2}\\ &&-{~}\frac{\rho t_{k}(t_{k}-1)}{1-2c_{1}}\|w^{k+1}\|^{2}-\frac{(t_{k}-1)}{4(1-2c_{1})}\big[\frac{2\rho t_{k}}{c_{1}}+\frac{2\rho+\gamma(1-2c_{1})}{c_{2}}\big]\|e^{k}\|^{2}.\end{array} Since t_{k-1}-t_{k}+1=1-\mu due to ( 46 ), adding (1-\mu)\langle w^{k},x^{k}-\bar{x}^{k}\rangle to both sides of \mathcal{T}_{[4]} and using the identity \begin{array}[]{lcl}\eta t_{k}(t_{k}-1)\langle\hat{w}^{k+1},w^{k}\rangle&=&\frac{\eta t_{k}^{2}}{2}\|\hat{w}^{k+1}\|^{2}+\frac{\eta(t_{k}-1)^{2}}{2}\|w^{k}\|^{2}-\frac{\eta}{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}\end{array} into \mathcal{T}_{[4]} , we can show that \begin{array}[]{lcl}\mathcal{T}_{[5]}&:=&t_{k-1}\langle w^{k},x^{k}-\bar{x}^{k}\rangle-t_{k}\langle w^{k+1},x^{k+1}-\bar{x}^{k+1}\rangle\\ &\geq&\frac{\eta}{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}+(1-\mu)\langle w^{k},x^{k}-\bar{x}^{k}\rangle\\ &&-{~}\big[\frac{\eta(t_{k}-1)^{2}}{2}+\gamma(1+c_{2})(t_{k}-1)-\frac{\rho(t_{k}-1)[t_{k}-2(1+c_{2})]}{1-2c_{1}}\big]\|w^{k}\|^{2}\\ &&-{~}\frac{\rho t_{k}(t_{k}-1)}{1-2c_{1}}\|w^{k+1}\|^{2}-\frac{(t_{k}-1)}{4(1-2c_{1})}\big[\frac{2\rho t_{k}}{c_{1}}+\frac{2\rho+\gamma(1-2c_{1})}{c_{2}}\big]\|e^{k}\|^{2}\\ &&+{~}\eta t_{k}^{2}\langle\hat{w}^{k+1},w^{k+1}\rangle-\frac{\eta t_{k}^{2}}{2}\|\hat{w}^{k+1}\|^{2}.\end{array} Now, since \hat{\eta}_{k}=\eta(1-\tau_{k})=\frac{\eta(t_{k}-1)}{t_{k}} due to ( 46 ), from the first and second lines of ( GFEG+ ), we can show that \begin{array}[]{lcl}t_{k}(x^{k+1}-y^{k})=-\eta t_{k}\hat{w}^{k+1}+t_{k}\hat{\eta}_{k}z^{k}=-\eta\left[t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}-(t_{k}-1)e^{k}\right].\end{array} By the L -Lipschitz continuity of F , for any \omega\geq 0 , c>0 , and M^{2}:=(1+\omega)(1+c)L^{2} , applying Young’s inequality, the last expression leads to \begin{array}[]{lcl}(1+\omega)t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}&=&(1+\omega)t_{k}^{2}\|Fx^{k+1}-Fy^{k}\|^{2}\\ &\leq&(1+\omega)L^{2}\|t_{k}(x^{k+1}-y^{k})\|^{2}\\ &=&(1+\omega)L^{2}\eta^{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}-(t_{k}-1)e^{k}\|^{2}\\ &\leq&M^{2}\eta^{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}+\frac{M^{2}\eta^{2}(t_{k}-1)^{2}}{c}\|e^{k}\|^{2}.\end{array} This inequality implies that \begin{array}[]{lcl}0&\geq&\frac{\eta t_{k}^{2}}{2}\|w^{k+1}\|^{2}+\frac{\eta t_{k}^{2}}{2}\|\hat{w}^{k+1}\|^{2}-\eta t_{k}^{2}\langle w^{k+1},\hat{w}^{k+1}\rangle+\frac{\eta\omega t_{k}^{2}}{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&-{~}\frac{M^{2}\eta^{3}}{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}-\frac{M^{2}\eta^{3}(t_{k}-1)^{2}}{2c}\|e^{k}\|^{2}.\end{array} Adding this inequality to \mathcal{T}_{[5]} above, we get \begin{array}[]{lcl}\mathcal{T}_{[5]}&:=&t_{k-1}\langle w^{k},x^{k}-\bar{x}^{k}\rangle-t_{k}\langle w^{k+1},x^{k+1}-\bar{x}^{k+1}\rangle\\ &\geq&\frac{\omega\eta t_{k}^{2}}{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}+\frac{\eta(1-M{{}^{2}}\eta^{2})}{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}\\ &&+{~}(1-\mu)\langle w^{k},x^{k}-\bar{x}^{k}\rangle+\frac{t_{k}}{2}\big[\eta t_{k}-\frac{2\rho(t_{k}-1)}{1-2c_{1}}\big]\|w^{k+1}\|^{2}\\ &&-{~}\big[\frac{\eta(t_{k}-1)^{2}}{2}+\gamma(1+c_{2})(t_{k}-1)-\frac{\rho(t_{k}-1)[t_{k}-2(1+c_{2})]}{1-2c_{1}}\big]\|w^{k}\|^{2}\\ &&-{~}\Big\{\frac{M^{2}\eta^{3}(t_{k}-1)^{2}}{2c}+\frac{(t_{k}-1)}{4(1-2c_{1})}\big[\frac{2\rho t_{k}}{c_{1}}+\frac{2\rho+\gamma(1-2c_{1})}{c_{2}}\big]\Big\}\|e^{k}\|^{2}.\end{array} Next, since \bar{x}^{k+1}-\bar{x}^{k}=-\gamma z^{k}=-\gamma(w^{k}+e^{k}) from the third line of ( GFEG+ ), for any \hat{c}>0 , by Young’s inequality, we can prove that \begin{array}[]{lcl}\mathcal{T}_{[6]}&:=&\frac{(1-\mu)}{2\gamma}\|\bar{x}^{k}-x^{\star}\|^{2}-\frac{(1-\mu)}{2\gamma}\|\bar{x}^{k+1}-x^{\star}\|^{2}\\ &=&-\frac{(1-\mu)}{\gamma}\langle\bar{x}^{k+1}-\bar{x}^{k},\bar{x}^{k}-x^{\star}\rangle-\frac{(1-\mu)}{2\gamma}\|\bar{x}^{k+1}-\bar{x}^{k}\|^{2}\\ &=&(1-\mu)\langle w^{k}+e^{k},\bar{x}^{k}-x^{\star}\rangle-\frac{(1-\mu)\gamma}{2}\|w^{k}+e^{k}\|^{2}\\ &\geq&(1-\mu)\langle w^{k},\bar{x}^{k}-x^{\star}\rangle+(1-\mu)\langle e^{k},\bar{x}^{k}-x^{\star}\rangle\\ &&-{~}\frac{(1+\hat{c})(1-\mu)\gamma}{2}\|w^{k}\|^{2}-\frac{(1+\hat{c})(1-\mu)\gamma}{2\hat{c}}\|e^{k}\|^{2}.\end{array} Summing up the last two expressions \mathcal{T}_{[5]} and \mathcal{T}_{[6]} , we get \begin{array}[]{lcl}\mathcal{T}_{[7]}&:=&t_{k-1}\langle w^{k},x^{k}-\bar{x}^{k}\rangle-t_{k}\langle w^{k+1},x^{k+1}-\bar{x}^{k+1}\rangle\\ &&+{~}\frac{(1-\mu)}{2\gamma}\|\bar{x}^{k}-x^{\star}\|^{2}-\frac{(1-\mu)}{2\gamma}\|\bar{x}^{k+1}-x^{\star}\|^{2}\\ &\geq&\frac{\omega\eta t_{k}^{2}}{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}+\frac{{\color[rgb]{0,0,0}\eta}(1-M^{2}\eta^{2})}{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}\\ &&+{~}\frac{t_{k}}{2}\big[\eta t_{k}-\frac{2\rho(t_{k}-1)}{1-2c_{1}}\big]\|w^{k+1}\|^{2}-\frac{\hat{a}_{k}}{2}\|w^{k}\|^{2}\\ &&-{~}\frac{1}{2}\Big\{\frac{M^{2}\eta^{3}(t_{k}-1)^{2}}{c}+\frac{(t_{k}-1)}{2(1-2c_{1})}\big[\frac{2\rho t_{k}}{c_{1}}+\frac{2\rho+\gamma(1-2c_{1})}{c_{2}}\big]+\frac{(1+\hat{c})(1-\mu)\gamma}{\hat{c}}\Big\}\|e^{k}\|^{2}\\ &&+{~}(1-\mu)\langle e^{k},\bar{x}^{k}-x^{\star}\rangle+(1-\mu)\langle w^{k},x^{k}-x^{\star}\rangle.\end{array} where \hat{a}_{k}:=\eta(t_{k}-1)^{2}+2\gamma(1+c_{2})(t_{k}-1)-\frac{2\rho(t_{k}-1)[t_{k}-2(1+c_{2})]}{1-2c_{1}}+(1+\hat{c})(1-\mu)\gamma . Rearranging this expression and using \mathcal{V}_{k} from ( 45 ), we obtain ( 47 ). \square D.2 The Proof of Lemma 6 From the \rho -co-hypomonotonicity of \Phi and x^{\star}\in\mathrm{zer}(\Phi) , we can show that \langle w^{k},x^{k}-x^{\star}\rangle\geq-\rho\|w^{k}\|^{2} . Using this inequality and \mathcal{V}_{k} from ( 45 ), for any \tilde{c}>0 , we have \begin{array}[]{lcl}\mathcal{V}_{k}&:=&\frac{a_{k}}{2}\|w^{k}\|^{2}+t_{k-1}\langle w^{k},x^{k}-\bar{x}^{k}\rangle+\frac{(1-\mu)}{2\gamma}\|\bar{x}^{k}-x^{\star}\|^{2}\\ &=&\frac{\tilde{c}\eta t_{k-1}^{2}}{2}\|w^{k}\|^{2}-t_{k-1}\langle w^{k},\bar{x}^{k}-x^{\star}\rangle+\frac{1}{2\tilde{c}\eta}\|\bar{x}^{k}-x^{\star}\|^{2}+t_{k-1}\langle w^{k},x^{k}-x^{\star}\rangle\\ &&+\frac{(a_{k}-\tilde{c}\eta t_{k-1}^{2})}{2}\|w^{k}\|^{2}+\frac{[(1-\mu)\tilde{c}\eta-\gamma]}{2\tilde{c}\gamma\eta}\|\bar{x}^{k}-x^{\star}\|^{2}\\ &\geq&\frac{1}{2\tilde{c}\eta}\|\bar{x}^{k}-x^{\star}-\tilde{c}\eta t_{k-1}w^{k}\|^{2}+\frac{[(1-\mu)\tilde{c}\eta-\gamma]}{2\tilde{c}\gamma\eta}\|\bar{x}^{k}-x^{\star}\|^{2}\\ &&+{~}\frac{(a_{k}-\tilde{c}\eta t_{k-1}^{2}-2\rho t_{k-1})}{2}\|w^{k}\|^{2}\\ &\geq&\frac{[(1-\mu)\tilde{c}\eta-\gamma]}{2\tilde{c}\gamma\eta}\|\bar{x}^{k}-x^{\star}\|^{2}+\frac{(a_{k}-\tilde{c}\eta t_{k-1}^{2}-2\rho t_{k-1})}{2}\|w^{k}\|^{2},\end{array} which proves ( 49 ). \square D.3 The Proof of Theorem 5.1 For simplicity of our analysis, let us fix c_{1}:=\frac{1}{4} , c_{2}:=\frac{1}{2} , c:=1 , and \hat{c}:=1 in Lemma 5 . We also choose \gamma:=\zeta(1-\mu)\eta for some \zeta\in(0,1] . First, from ( 46 ), we have \begin{array}[]{ll}&\tau_{k}:=\frac{1}{t_{k}}=\frac{1}{\mu(k+r)},\quad\hat{\eta}_{k}:=\eta(1-\tau_{k})=\frac{\eta(t_{k}-1)}{t_{k}},\\ &\beta_{k}:=-\gamma\tau_{k}+4\rho(1-\tau_{k})=-\frac{\zeta(1-\mu)\eta}{t_{k}}+\frac{4\rho(t_{k}-1)}{t_{k}},\end{array} as stated in ( 50 ). Next, using c_{1}=\frac{1}{4} , c_{2}=\frac{1}{2} , c=1 , \hat{c}=1 , \gamma=\zeta(1-{\color[rgb]{0,0,0}\mu})\eta , and t_{k}=t_{k-1}+\mu in ( 47 ), we can show that \hskip-2.15277pt\left\{\begin{array}[]{lcl}a_{k}&=&\eta t_{k-1}^{2}-4\rho t_{k-1}(t_{k-1}-1),\\ \hat{a}_{k}&=&a_{k}-\psi_{k},\\ b_{k}&=&M^{2}\eta^{3}(t_{k}-1)^{2}+(t_{k}-1)\big[8\rho t_{k}+4\rho+\zeta(1-\mu)\eta\big]+2\zeta(1-\mu)^{2}\eta,\end{array}\right.\hskip-12.91663pt (124) where \begin{array}[]{lcl}\psi_{k}&:=&\big[(2-3\zeta)(1-\mu)\eta-4\rho(3-2\mu)\big]t_{k-1}\\ &&-{~}(1-\mu)[(1-\zeta)(1-\mu)\eta-4\rho(3-\mu)].\end{array} (125) Let us choose \zeta:=\frac{1-2\mu}{6(1-\mu)} , provided that 0<\mu<\frac{1}{2} . Then, we obtain \gamma=\frac{(1-2\mu)\eta}{6} as stated in ( 51 ). Moreover, from ( 125 ), to guarantee \psi_{k}\geq 0 , we first need to choose \eta>8\rho as stated in ( 51 ). Moreover, we can simplify \psi_{k} as \begin{array}[]{lcl}\psi_{k}&=&\frac{(3-2\mu)(\eta-8\rho)}{2}t_{k-1}-\frac{(1-\mu)[(5-4\mu)\eta-24\rho(3-\mu)]}{6}\\ &\geq&\frac{(3-2\mu)(\eta-8\rho)}{2}(t_{k-1}-1+\mu)=\frac{(3-2\mu)(\eta-8\rho)}{2}(t_{k}-1).\end{array} (126) Hence, if \mu r\geq 1 , then t_{k}\geq 1 , leading to \psi_{k}\geq 0 . Now, utilizing ( 124 ), we can simplify ( 47 ) as \begin{array}[]{lcl}\mathcal{V}_{k}&\geq&\mathcal{V}_{k+1}+\frac{\psi_{k}}{2}\|w^{k}\|^{2}+\frac{{\color[rgb]{0,0,0}\eta}(1-M^{2}\eta^{2})}{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}\\ &&+{~}\frac{\omega\eta t_{k}^{2}}{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}+(1-\mu)\langle w^{k},x^{k}-x^{\star}\rangle\\ &&+{~}(1-\mu)\langle e^{k},\bar{x}^{k}-x^{\star}\rangle-\frac{b_{k}}{2}\|e^{k}\|^{2}.\end{array} (127) Since e^{k+1}=u^{k+1}-Fx^{k+1} , from ( 41 ), for any \lambda>0 , we have \begin{array}[]{lcl}\frac{\lambda\eta t_{k}^{2}}{2}\|e^{k+1}\|^{2}&\leq&\frac{\lambda\kappa\eta t_{k}^{2}}{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}+\frac{\lambda\hat{\kappa}\eta{{}^{3}}}{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}.\end{array} In addition, by the \rho -co-hypomonotonicity of \Phi and x^{\star}\in\mathrm{zer}(\Phi) , we have \langle w^{k},x^{k}-x^{\star}\rangle\geq-\rho\|w^{k}\|^{2} . Combining ( 127 ) and the last two inequalities, and then applying Young’s inequality to \langle e^{k},\bar{x}^{k}-x^{\star}\rangle , for \nu>0 , we can show that \hskip-2.15277pt\begin{array}[]{lcl}\mathcal{V}_{k}+\frac{(b_{k}+\eta\nu t_{k}^{2})}{2}\|e^{k}\|^{2}&\geq&\mathcal{V}_{k+1}+\frac{\lambda\eta t_{k}^{2}}{2}\|e^{k+1}\|^{2}+\frac{[\psi_{k}-2(1-\mu)\rho]}{2}\|w^{k}\|^{2}\\ &&+{~}\frac{{\color[rgb]{0,0,0}\eta}(1-M^{2}\eta^{2}-\lambda\hat{\kappa}\eta^{2})}{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}\\ &&+{~}\frac{\eta(\omega-\lambda\kappa)t_{k}^{2}}{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}-\frac{(1-\mu)^{2}}{2\nu\eta t_{k}^{2}}\|\bar{x}^{k}-x^{\star}\|^{2}.\end{array}\hskip-8.61108pt (128) Since \eta>8\rho by ( 51 ), \zeta=\frac{1-2\mu}{6(1-\mu)} , and t_{k}=t_{k-1}+\mu , we have from ( 124 ) that \begin{array}[]{lcl}b_{k}+\eta\nu t_{k}^{2}&\leq&M^{2}\eta^{3}(t_{k}-1)^{2}+\eta\nu t_{k}^{2}+(t_{k}-1)\big[\eta t_{k}+\frac{\eta}{2}+\frac{(1-2\mu)\eta}{6}\big]+2\zeta(1-\mu)^{2}\eta\\ &=&\eta(M^{2}\eta^{2}+\nu+1)t_{k-1}^{2}+\eta S_{k},\end{array} where \begin{array}[]{lcl}S_{k}&:=&\big(2\nu\mu-2(1-\mu)M^{2}\eta^{2}+\frac{5\mu-1}{3}\big)t_{k-1}\\ &&+{~}\big[(1-\mu)^{2}M^{2}\eta^{2}+\nu\mu^{2}-\frac{(1-\mu)(1+4\mu)}{3}\big].\end{array} (129) From the last expression, if we impose \begin{array}[]{lcl}\lambda&\geq&M^{2}\eta^{2}+2\nu+1\quad\textrm{and}\quad S_{k}\leq\nu t_{k-1}^{2},\end{array} (130) then we can show that b_{k}+\eta\nu t_{k}^{2}\leq\eta\lambda t_{k-1}^{2} . Since b_{k}+\eta\nu t_{k}^{2}\leq\eta\lambda t_{k-1}^{2} , utilizing \mathcal{L}_{k} from ( 45 ), ( 128 ) implies that \hskip-2.15277pt\begin{array}[]{lcl}\mathcal{L}_{k}-\mathcal{L}_{k+1}&\geq&\frac{[\psi_{k}-2(1-\mu)\rho]}{2}\|w^{k}\|^{2}+\frac{{\color[rgb]{0,0,0}\eta}(1-M^{2}\eta^{2}-\lambda\hat{\kappa}\eta^{2})}{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}\\ &&+{~}\frac{\eta(\omega-\lambda\kappa)t_{k}^{2}}{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}-\frac{(1-\mu)^{2}}{2\nu\eta t_{k}^{2}}\|\bar{x}^{k}-x^{\star}\|^{2}.\end{array}\hskip-8.61108pt (131) Next, from Lemma 6 , a_{k} from ( 124 ), and \gamma={\color[rgb]{0,0,0}\zeta(1-\mu)\eta=}\frac{(1-2\mu)\eta}{6} , we also have \begin{array}[]{lcl}\mathcal{L}_{k}&\geq&\mathcal{V}_{k}\geq\frac{[(1-\mu)\tilde{c}-\zeta(1-\mu)]}{2\tilde{c}\zeta(1-\mu)\eta}\|\bar{x}^{k}-x^{\star}\|^{2}+\frac{[\eta(1-\tilde{c})-4\rho]t_{k-1}^{2}+2\rho t_{k-1}}{2}\|w^{k}\|^{2}\\ &\geq&\frac{[6(1-\mu)\tilde{c}-(1-2\mu)]}{2\tilde{c}(1-2\mu)\eta}\|\bar{x}^{k}-x^{\star}\|^{2}+\frac{[\eta(1-\tilde{c})-4\rho]t_{k-1}^{2}}{2}\|w^{k}\|^{2}.\end{array} (132) Let us first choose \tilde{c}:=\frac{1}{2} . Since \eta>8\rho , ( 132 ) reduces to \mathcal{L}_{k}\geq\frac{(2-\mu)}{(1-2\mu)\eta}\|\bar{x}^{k}-x^{\star}\|^{2} . Substituting this estimate into ( 131 ), we can show that \hskip-8.61108pt\begin{array}[]{lcl}\mathcal{L}_{k+1}&\leq&\big[1+\frac{(1-\mu)^{2}(1-2\mu)}{2\nu(2-\mu)t_{k}^{2}}\big]\cdot\mathcal{L}_{k}-\frac{[\psi_{k}-2(1-\mu)\rho]}{2}\|w^{k}\|^{2}\\ &&-{~}\frac{\eta(\omega-\lambda\kappa)t_{k}^{2}}{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&-{~}\frac{{\color[rgb]{0,0,0}\eta}(1-M^{2}\eta^{2}-\lambda\hat{\kappa}\eta^{2})}{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}.\end{array}\hskip-21.52771pt (133) Suppose that \begin{array}[]{lcl}\omega\geq\lambda\kappa\quad\textrm{and}\quad(M^{2}+\lambda\hat{\kappa})\eta^{2}\leq 1.\end{array} (134) Then ( 133 ) reduces to \begin{array}[]{lcl}0\leq\mathcal{L}_{k+1}&\leq&\big[1+\frac{(1-2\mu)(1-\mu)^{2}}{2\nu(2-\mu)t_{k}^{2}}\big]\cdot\mathcal{L}_{k}-\frac{[\psi_{k}-2(1-\mu)\rho]}{2}\|w^{k}\|^{2}.\end{array} (135) By Lemma 20 , for \Lambda_{K}:=\prod_{k=0}^{K}\big(1+\frac{\theta\mu^{2}}{t_{k}^{2}}\big)=\prod_{k=0}^{K}\big(1+\frac{\theta}{(k+r)^{2}}\big)\leq\Lambda:=e^{\frac{\theta}{r}} , where \theta:=\frac{(1-2\mu)(1-\mu)^{2}}{2\nu(2-\mu)\mu^{2}} , we obtain from ( 135 ) and Lemma 19 that \begin{array}[]{lcl}\mathcal{L}_{K}&\leq&e^{\frac{\theta}{r}}\cdot\mathcal{L}_{0}\quad\textrm{and}\quad\sum_{k=0}^{\color[rgb]{0,0,0}\infty}{\color[rgb]{0,0,0}\frac{\psi_{k}-2(1-\mu)\rho}{2}}\|w^{k}\|^{2}\leq\big(1+\frac{\theta}{r}e^{\frac{\theta}{r}}\big)\cdot\mathcal{L}_{0}.\end{array} Note that since u^{0}:=Fx^{0} and x^{0}=\bar{x}^{0} , we have \mathcal{L}_{0}=\mathcal{V}_{0}=\frac{a_{0}}{2}\|w^{0}\|^{2}+\frac{(1-\mu)}{2\gamma}\|x^{0}-x^{\star}\|^{2}\leq\frac{\eta\mu^{2}(r-1)^{2}}{2}\|Fx^{0}+\xi^{0}\|^{2}+\frac{3(1-\mu)}{(1-2\mu)\eta}\|x^{0}-x^{\star}\|^{2}=:\mathcal{R}_{0}^{2} . Combining this bound and the last expressions, we get \begin{array}[]{lcl}\mathcal{L}_{K}&\leq&e^{\frac{\theta}{r}}\cdot\mathcal{R}_{0}^{2},\\ \sum_{k=0}^{\infty}[\psi_{k}-2(1-\mu)\rho]\|w^{k}\|^{2}&\leq&2\big(1+\frac{\theta}{r}e^{\frac{\theta}{r}}\big)\cdot\mathcal{R}_{0}^{2}.\end{array} (136) If we choose \tilde{c}=\frac{1-2\mu}{6(1-\mu)} in ( 132 ), then \mathcal{L}_{k}\geq\frac{[\eta(5-4\mu)-24\rho(1-\mu)]\mu^{2}(k+r-1)^{2}}{12(1-\mu)}\|w^{k}\|^{2}\geq\frac{\mu^{2}\eta(k+r-1)^{2}}{6}\|w^{k}\|^{2} since \eta\geq 8\rho and \mu\in(0,\frac{1}{2}) . Combining this bound and the first line of ( 136 ), we obtain ( 52 ). From ( 126 ), one can also show that \psi_{k}-2(1-\mu)\rho\geq\frac{(3-2\mu){\color[rgb]{0,0,0}(\eta-8\rho)}}{2}(t_{k}-1) . Using this bound into the second line of ( 136 ), we obtain ( 53 ). Finally, let us choose \lambda:=\frac{\omega}{\kappa} , \nu:=1 , and \omega:=4\nu\kappa=4\kappa . Then, the first condition of ( 134 ) automatically holds, while the second one becomes \big[2(1+4\kappa)L^{2}+4\hat{\kappa}\big]\eta^{2}\leq 1 . Combining this condition and M^{2}\eta^{2}+2\nu+1=2(1+4\kappa)L^{2}\eta^{2}+3\leq\lambda=4 from ( 130 ), we get \begin{array}[]{lcl}\eta\leq\bar{\eta}:=\min\left\{\frac{1}{\sqrt{2(1+4\kappa)L^{2}+4\hat{\kappa}}},\frac{1}{L\sqrt{2(1+4\kappa)}}\right\}={\color[rgb]{0,0,0}\frac{1}{\sqrt{2(1+4\kappa)L^{2}+4\hat{\kappa}}}}.\end{array} This condition is exactly the one in ( 51 ). In addition, since \nu=1 , M^{2}\eta^{2}<1 and \mu\in{\color[rgb]{0,0,0}(0,\frac{1}{2})} , the second condition of ( 130 ) holds if t_{k-1}\geq 2 , which is guaranteed if we choose r\geq 1+\frac{2}{\mu} . Combining this condition and r\geq\frac{1}{\mu} above, we get r\geq 1+\frac{2}{\mu} . \square D.4 The Proof of Theorem 5.2 First, the condition r\geq 1+\frac{2}{\mu} is sufficiently tight to guarantee S_{k}\leq(1-\alpha)t_{k-1}^{2} for some sufficiently small \alpha\in(0,1) instead of the second condition of ( 130 ). Hence, we have b_{k}+\nu\eta t_{k}^{2}\leq\lambda\eta t_{k-1}^{2}-\alpha\eta t_{k-1}^{2} . Similar to the proof of ( 133 ), but using this condition, we have \hskip-8.61108pt\begin{array}[]{lcl}\mathcal{L}_{k+1}&\leq&\big(1+\frac{\theta\mu^{2}}{t_{k}^{2}}\big)\mathcal{L}_{k}-\frac{[\psi_{k}-2(1-\mu)\rho]}{2}\|w^{k}\|^{2}-\frac{\eta(\omega-\lambda\kappa)t_{k}^{2}}{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&-{~}\frac{{\color[rgb]{0,0,0}\eta}(1-M^{2}\eta^{2}-\lambda\hat{\kappa}\eta^{2})}{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}-\frac{\alpha\eta t_{k-1}^{2}}{2}\|e^{k}\|^{2}.\end{array}\hskip-21.52771pt (137) Next, since we assume that \eta<\bar{\eta} in ( 51 ), we can show from ( 134 ) that \omega>\lambda\kappa and (M^{2}+\lambda\hat{\kappa})\eta^{2}<1 . Hence, we can derive from ( 137 ) that \begin{array}[]{lcl}\sum_{k=0}^{\infty}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}&<&+\infty,\\ \sum_{k=0}^{\infty}t_{k}^{2}\|\hat{w}^{k+1}-w^{k+1}\|^{2}&<&+\infty,\\ \sum_{k=0}^{\infty}t_{k}^{2}\|e^{k}\|^{2}&<&+\infty.\end{array} (138) The second and third bounds of ( 138 ) are respectively the first two lines of ( 54 ). Since t_{k}(x^{k+1}-y^{k})=-\eta\big(t_{k}\hat{w}^{k+1}-(t_{k}-1)z^{k}\big) due to ( GFEG+ ) and e^{k}=z^{k}-w^{k} , by Young’s inequality, we can show that \begin{array}[]{lcl}t_{k}\|z^{k}\|^{2}&\leq&2t_{k}\|w^{k}\|^{2}+2t_{k}\|z^{k}-w^{k}\|^{2}=2t_{k}\|w^{k}\|^{2}+2t_{k}\|e^{k}\|^{2},\\ t_{k}^{2}\|w^{k+1}-w^{k}\|^{2}&\leq&3\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}+3t_{k}^{2}\|\hat{w}^{k+1}-w^{k+1}\|^{2}+3\|w^{k}\|^{2},\\ t_{k}^{2}\|x^{k+1}-y^{k}\|^{2}&=&\eta^{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)z^{k}\|^{2}\\ &\leq&2\eta^{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}+2\eta^{2}(t_{k}-1)^{2}\|e^{k}\|^{2}.\end{array} All the terms on the right-hand side of these inequalities are summable due to ( 138 ) and ( 53 ), we get \begin{array}[]{lcl}\sum_{k=0}^{\infty}t_{k}\|z^{k}\|^{2}&<&+\infty,\\ \sum_{k=0}^{\infty}t_{k}^{2}\|w^{k+1}-w^{k}\|^{2}&<&+\infty,\\ \sum_{k=0}^{\infty}t_{k}^{2}\|x^{k+1}-y^{k}\|^{2}&<&+\infty.\end{array} (139) Now, from the first line of ( GFEG+ ) and \tau_{k}=\frac{1}{t_{k}} , we can easily derive that \begin{array}[]{lcl}v^{k}&:=&\bar{x}^{k}-x^{k}=t_{k}(y^{k}-x^{k})+[(\eta-4\rho)(t_{k}-1)+\gamma]z^{k}.\end{array} Using this expression, the third line of ( GFEG+ ), and x^{k+1}-y^{k}=-\frac{\eta}{t_{k}}(t_{k}\hat{w}^{k+1}-(t_{k}-1)z^{k}) , we can derive \begin{array}[]{lcl}v^{k+1}-\frac{(t_{k}-1)}{t_{k}}v^{k}&=&\bar{x}^{k+1}-\bar{x}^{k}-(x^{k+1}-y^{k})+\frac{[(\eta-4\rho)(t_{k}-1)+\gamma]}{t_{k}}z^{k}\\ &=&-\gamma z^{k}+\eta\hat{w}^{k+1}-\frac{\eta(t_{k}-1)}{t_{k}}z^{k}+\frac{[(\eta-4\rho)(t_{k}-1)+\gamma]}{t_{k}}z^{k}\\ &=&\eta\big(\hat{w}^{k+1}-\frac{t_{k}-1}{t_{k}}w^{k}\big)+\frac{(t_{k}-1)}{t_{k}}(\eta-4\rho-\gamma)z^{k}-\frac{{\color[rgb]{0,0,0}\eta}(t_{k}-1)}{t_{k}}e^{k}.\end{array} Let us denote d^{k}:=\eta(t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k})+(\eta-4\rho-\gamma)(t_{k}-1)w^{k}-{\color[rgb]{0,0,0}\eta}(t_{k}-1)e^{k} . Then, the last estimate is rewritten as v^{k+1}=\big(1-\frac{1}{t_{k}}\big)v^{k}+\frac{1}{t_{k}}d^{k} . By convexity of \|\cdot\|^{2} , \frac{1}{t_{k}}\in(0,1] , and Young’s inequality, we have \begin{array}[]{lcl}\|v^{k+1}\|^{2}&\leq&\frac{t_{k}-1}{t_{k}}\|v^{k}\|^{2}+\frac{1}{t_{k}}\|d^{k}\|^{2}\\ &\leq&\|v^{k}\|^{2}-\frac{1}{t_{k}}\|v^{k}\|^{2}+\frac{3\eta^{2}}{t_{k}}\|t_{k}w^{k+1}-(t_{k}-1)w^{k}\|^{2}\\ &&+{~}\frac{3(\eta-4\rho-\gamma)^{2}(t_{k}-1)^{2}}{t_{k}}\|w^{k}\|^{2}+\frac{3{\color[rgb]{0,0,0}\eta}^{2}(t_{k}-1)^{2}}{t_{k}}\|e^{k}\|^{2}.\end{array} Since the last three terms of this inequality are summable due to ( 138 ) and ( 53 ), one can easily prove that \lim_{k\to\infty}\|v^{k}\|^{2}=\lim_{k\to\infty}\|x^{k}-\bar{x}^{k}\|^{2}=0,\ \ \textrm{and}\ \ \sum_{k=0}^{\infty}\tfrac{1}{t_{k}}\|v^{k}\|^{2}<+\infty. (140) Next, applying Young’s inequality again, we have \begin{array}[]{lcl}t_{k}\|y^{k}-x^{k}\|^{2}&\leq&\frac{2}{t_{k}}\|v^{k}\|^{2}+\frac{2[(\eta-4\rho)(t_{k}-1)+\gamma]^{2}}{t_{k}}\|z^{k}\|^{2},\\ t_{k}\|x^{k+1}-x^{k}\|^{2}&\leq&2t_{k}\|x^{k+1}-y^{k}\|^{2}+2t_{k}\|y^{k}-x^{k}\|^{2}.\end{array} Since the two terms on the right-hand side of the first inequality are summable due to ( 140 ) and ( 139 ), we get \begin{array}[]{lcl}\sum_{k=0}^{\infty}t_{k}\|y^{k}-x^{k}\|^{2}<+\infty\quad\text{and}\quad\sum_{k=0}^{\infty}t_{k}\|x^{k+1}-x^{k}\|^{2}<+\infty.\end{array} (141) These results lead to the third and fourth lines of ( 54 ). We have proven in Theorem 5.1 that \frac{(2-\mu)}{(1-2\mu)\eta}\|\bar{x}^{k}-x^{\star}\|^{2}\leq\mathcal{L}_{k}\leq\mathcal{L}_{0} . Hence, \{\|\bar{x}^{k}-x^{\star}\|\} is bounded, i.e. there exists M>0 such that \|\bar{x}^{k}-x^{\star}\|\leq M for all k\geq 0 . Using this fact and \langle w^{k},x^{k}-x^{\star}\rangle\geq-\rho\|w^{k}\|^{2} , we can deduce that \begin{array}[]{lcl}\|\bar{x}^{k+1}-x^{\star}\|^{2}&=&\|\bar{x}^{k}-x^{\star}\|^{2}+2\langle\bar{x}^{k+1}-\bar{x}^{k},\bar{x}^{k}-x^{\star}\rangle+\|\bar{x}^{k+1}-\bar{x}^{k}\|^{2}\\ &=&\|\bar{x}^{k}-x^{\star}\|^{2}-2\gamma\langle z^{k},\bar{x}^{k}-x^{\star}\rangle+\gamma^{2}\|z^{k}\|^{2}\\ &=&\|\bar{x}^{k}-x^{\star}\|^{2}-2\gamma\langle e^{k},\bar{x}^{k}-x^{\star}\rangle-2\gamma\langle w^{k},x^{k}-x^{\star}\rangle\\ &&-{~}2\gamma\langle w^{k},\bar{x}^{k}-x^{k}\rangle+\gamma^{2}\|z^{k}\|^{2}\\ &\leq&\|\bar{x}^{k}-x^{\star}\|^{2}+\gamma t_{k}^{2}\|e^{k}\|^{2}+\frac{{\color[rgb]{0,0,0}\gamma}}{t_{k}^{2}}\|\bar{x}^{k}-x^{\star}\|^{2}+2\gamma\rho\|w^{k}\|^{2}\\ &&+{~}\gamma t_{k}\|w^{k}\|^{2}+\frac{\gamma}{t_{k}}\|\bar{x}^{k}-x^{k}\|^{2}+\gamma^{2}\|z^{k}\|^{2}\\ &\leq&\|\bar{x}^{k}-x^{\star}\|^{2}+\gamma t_{k}^{2}\|e^{k}\|^{2}+\frac{M^{2}{\color[rgb]{0,0,0}\gamma}}{t_{k}^{2}}+\gamma(t_{k}+2\rho)\|w^{k}\|^{2}\\ &&+{~}\frac{\gamma}{t_{k}}\|v^{k}\|^{2}+\gamma^{2}\|z^{k}\|^{2}.\end{array} Since the last five terms on the right-hand side are summable, from this inequality, we conclude that \lim_{k\to\infty}\|\bar{x}^{k}-x^{\star}\|^{2} exists. However, since \lim_{k}\|x^{k}-\bar{x}^{k}\|^{2}=0 by ( 140 ), using the triangle inequality |\|x^{k}-x^{\star}\|-\|\bar{x}^{k}-x^{\star}\||\leq\|x^{k}-\bar{x}^{k}\| , we conclude that \lim_{k\to\infty}\|x^{k}-x^{\star}\|^{2} also exists. Next, from ( 135 ), we can easily show that \lim_{k\to\infty}\mathcal{L}_{k} exists. However, since \mathcal{L}_{k}=\mathcal{V}_{k}+\frac{\lambda\eta t_{k}^{2}}{2}\|e^{k}\|^{2} due to ( 45 ) and \lim_{k\to\infty}t_{k}^{2}\|e^{k}\|^{2}=0 by the last line of ( 138 ), we conclude that \lim_{k\to\infty}\mathcal{V}_{k} also exists. By ( 52 ), there exists \bar{M}>0 such that t_{k-1}\|w^{k}\|\leq\bar{M} . Using this inequality and \lim_{k\to\infty}\|v^{k}\|=0 , we can show that \begin{array}[]{lcl}{\color[rgb]{0,0,0}t_{k-1}}|\langle w^{k},x^{k}-\bar{x}^{k}\rangle|&\leq&t_{k-1}\|w^{k}\|\|v^{k}\|\leq\bar{M}\|v^{k}\|\to 0\quad\text{as}~k\to\infty.\end{array} From ( 45 ), we have \begin{array}[]{lcl}\mathcal{V}_{k}=\frac{a_{k}}{2}\|w^{k}\|^{2}+t_{k-1}\langle w^{k},x^{k}-\bar{x}^{k}\rangle+\frac{(1-\mu)}{2\gamma}\|\bar{x}^{k}-x^{\star}\|^{2}.\end{array} Using the above limits, we can say that \lim_{k\to\infty}a_{k}\|w^{k}\|^{2} exists. Since a_{k}=(\eta-4\rho)t_{k-1}^{2}+4\rho t_{k-1} , we conclude that \lim_{k\to\infty}t_{k}^{2}\|w^{k}\|^{2} exists. However, since \sum_{k=0}^{\infty}t_{k}\|w^{k}\|^{2}<+\infty due to ( 53 ), we can easily argue that \lim_{k\to\infty}t_{k}^{2}\|w^{k}\|^{2}=0 as stated in the first line of ( 55 ). Note that \begin{array}[]{lcl}t_{k}^{2}\|y^{k}-x^{k}\|^{2}&\leq&3\|v^{k}\|^{2}+3[(\eta-4\rho)(t_{k}-1)+\gamma]^{2}\|w^{k}\|^{2}\\ &&+{~}3[(\eta-4\rho)(t_{k}-1)+\gamma]^{2}\|e^{k}\|^{2}.\end{array} The limit of three terms on the right-hand side are zero, leading to \lim_{k\to\infty}t_{k}^{2}\|y^{k}-x^{k}\|^{2}=0 , which is the second line of ( 55 ). The third line of ( 55 ) follows from Young’s inequality, the second line of ( 55 ), and \lim_{k\to\infty}t_{k}^{2}\|x^{k+1}-y^{k}\|^{2}=0 . Finally, since \lim_{k\to\infty}\|x^{k}-x^{\star}\|^{2} exists and \mathrm{gra}(\Phi) is closed (see the proof of Theorem F.8 below), and one can easily show that any cluster point of \{x^{k}\} is a solution of ( NI ), we conclude that \{x^{k}\} converges to x^{\star} , a solution of ( NI ). Since \lim_{k\to\infty}\|x^{k}-\bar{x}^{k}\|=0 and \lim_{k\to\infty}\|x^{k}-y^{k}\|=0 , both \{\bar{x}^{k}\} and \{y^{k}\} also converge to x^{\star} . \square D.5 The Proof of Theorem 5.3 First, since u^{k}:=Fx^{k} , we have e^{k}=0 . Thus we can choose c_{1}=c_{2}=\hat{c}=c=0 and \omega=0 in Lemma 5 . In this case, from ( 46 ), we have t_{k} and \beta_{k} are updated as in ( 56 ), i.e.: \begin{array}[]{lcl}t_{k}:=\mu(k+t_{0})=\mu(k+r),\quad\textrm{and}\quad\beta_{k}:=\frac{2\rho(t_{k}-1)-\gamma}{t_{k}}=-\gamma\tau_{k}+2\rho(1-\tau_{k}).\end{array} In addition, a_{k} and \hat{a}_{k} in ( 47 ) respectively reduce to \begin{array}[]{lcl}a_{k}&:=&t_{k-1}\big[\eta t_{k-1}-2\rho(t_{k-1}-1)\big]\quad\textrm{and}\quad\hat{a}_{k}=a_{k}-\psi_{k},\end{array} (142) where \begin{array}[]{lcl}\psi_{k}&:=&2[(1-\mu)(\eta-2\rho)-\gamma](t_{k}-1)+{\color[rgb]{0,0,0}(1-\mu)[(1-\mu)\eta+2\mu\rho-\gamma]}.\end{array} (143) Second, since c=0 , \omega=0 , and e^{k}=0 , we have M^{2}=(1+c)(1+\omega)L^{2}=L^{2} and \mathcal{L}_{k}=\mathcal{V}_{k}+\frac{\lambda\eta t_{k}^{2}}{2}\|e^{k}\|^{2}=\mathcal{V}_{k} in ( 45 ). Moreover, due to the \rho -co-hypomonotonicity of \Phi and x^{\star}\in\mathrm{zer}(\Phi) , we have \langle w^{k},x^{k}-x^{\star}\rangle\geq-\rho\|w^{k}\|^{2} . Using these facts and ( 142 ), ( 47 ) reduces to \begin{array}[]{lcl}\mathcal{L}_{k}-\mathcal{L}_{k+1}&\geq&\frac{{\color[rgb]{0,0,0}\eta}(1-L^{2}\eta^{2})}{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}+\frac{\psi_{k}}{2}\|w^{k}\|^{2}\\ &&+{~}(1-\mu)\langle w^{k},x^{k}-x^{\star}\rangle\\ &\geq&\frac{{\color[rgb]{0,0,0}\eta}(1-L^{2}\eta^{2})}{2}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}+\frac{(\psi_{k}-2(1-\mu)\rho)}{2}\|w^{k}\|^{2}.\end{array} (144) To guarantee the nonnegativity of the right-hand side of ( 144 ), we require L\eta\leq 1 and \psi_{k}\geq{\color[rgb]{0,0,0}2(1-\mu)\rho} . The first condition holds if \eta\leq\frac{1}{L} , while the second one holds if \eta>2\rho , \gamma<(1-\mu)(\eta-2\rho) , and t_{k}\geq 1 . The first two conditions are stated in ( 57 ), while t_{k}\geq 1 holds if \mu r\geq 1 as stated in Theorem 5.3 . Since 2L\rho<1 , the condition 2\rho<\eta\leq\frac{1}{L} is well-defined. Third, by induction, we obtain from ( 144 ) that \left\{\begin{array}[]{lcl}\mathcal{L}_{k}&\leq&\mathcal{L}_{0},\quad\forall k\geq 0,\\ {\color[rgb]{0,0,0}\eta}(1-L^{2}\eta^{2})\sum_{k=0}^{\infty}\|t_{k}\hat{w}^{k+1}-(t_{k}-1)w^{k}\|^{2}&\leq&2\mathcal{L}_{0},\\ \sum_{k=0}^{\infty}[\psi_{k}-2(1-\mu)\rho]\|w^{k}\|^{2}&\leq&2\mathcal{L}_{0}.\end{array}\right. (145) By Lemma 6 and ( 45 ), with \tilde{c}:=\frac{\gamma}{\eta(1-\mu)} , we have \begin{array}[]{lcl}\mathcal{L}_{k}&\geq&\frac{[(1-\mu)(\eta-2\rho)-\gamma]t_{k-1}^{2}}{{\color[rgb]{0,0,0}2(1-\mu)}}\|w^{k}\|^{2},\\ \mathcal{L}_{0}&=&\frac{\mu(r-1)[\mu(r-1)(\eta-2\rho)+2\rho]}{2}\|w^{0}\|^{2}+\frac{(1-\mu)}{2\gamma}\|x^{0}-x^{\star}\|^{2}.\end{array} (146) Combining the first line of ( 145 ) and ( 146 ), we obtain the first line of ( 58 ). Finally, since \psi_{k}-2(1-\mu)\rho=[(1-\mu)(\eta-2\rho)-\gamma]{\color[rgb]{0,0,0}(2t_{k}-1-\mu)} due to ( 143 ), we obtain the second line of ( 58 ) from the third line of ( 145 ) and the second line of ( 146 ). \square Appendix E The Proof of Technical Results in Section 6 This appendix presents the full proof of technical results in Section 6 . E.1 The Proof of Lemma 7 First, by inserting a zero term (t_{k}-1)\hat{x}^{k+1}-(t_{k}-1)\hat{x}^{k+1} , we can expand \mathcal{T}_{[1]}:=\|\hat{x}^{k}-x^{\star}+t_{k}(y^{k}-\hat{x}^{k})\|^{2} as follows: \begin{array}[]{lcl}\mathcal{T}_{[1]}&:=&\|\hat{x}^{k}-x^{\star}+t_{k}(y^{k}-\hat{x}^{k})\|^{2}\\ &=&\|\hat{x}^{k+1}-x^{\star}+(t_{k}-1)(\hat{x}^{k+1}-\hat{x}^{k})+t_{k}(y^{k}-\hat{x}^{k+1})\|^{2}\\ &=&\|\hat{x}^{k+1}-x^{\star}\|^{2}+(t_{k}-1)^{2}\|\hat{x}^{k+1}-\hat{x}^{k}\|^{2}+t_{k}^{2}\|y^{k}-\hat{x}^{k+1}\|^{2}\\ &&+{~}2(t_{k}-1)\langle\hat{x}^{k+1}-\hat{x}^{k},\hat{x}^{k+1}-x^{\star}\rangle+2t_{k}\langle y^{k}-\hat{x}^{k+1},\hat{x}^{k+1}-x^{\star}\rangle\\ &&+{~}2(t_{k}-1)t_{k}\langle y^{k}-\hat{x}^{k+1},\hat{x}^{k+1}-\hat{x}^{k}\rangle.\end{array} Similarly, using y^{k+1}-\hat{x}^{k+1}=\theta_{k}(\hat{x}^{k+1}-\hat{x}^{k})+\nu_{k}(y^{k}-\hat{x}^{k+1}) from the last line of ( GAEG ), we can expand \mathcal{T}_{[2]}:=\|\hat{x}^{k+1}-x^{\star}+t_{k+1}(y^{k+1}-\hat{x}^{k+1})\|^{2} as \begin{array}[]{lcl}\mathcal{T}_{[2]}&:=&\|\hat{x}^{k+1}-x^{\star}+t_{k+1}(y^{k+1}-\hat{x}^{k+1})\|^{2}\\ &=&\|\hat{x}^{k+1}-x^{\star}+t_{k+1}\theta_{k}(\hat{x}^{k+1}-\hat{x}^{k})+t_{k+1}\nu_{k}(y^{k}-\hat{x}^{k+1})\|^{2}\\ &=&\|\hat{x}^{k+1}-x^{\star}\|^{2}+t_{k+1}^{2}\theta_{k}^{2}\|\hat{x}^{k+1}-\hat{x}^{k}\|^{2}+t_{k+1}^{2}\nu_{k}^{2}\|y^{k}-\hat{x}^{k+1}\|^{2}\\ &&+{~}2t_{k+1}\theta_{k}\langle\hat{x}^{k+1}-\hat{x}^{k},\hat{x}^{k+1}-x^{\star}\rangle+2t_{k+1}\nu_{k}\langle y^{k}-\hat{x}^{k+1},\hat{x}^{k+1}-x^{\star}\rangle\\ &&+{~}2t_{k+1}^{2}\nu_{k}\theta_{k}\langle y^{k}-\hat{x}^{k+1},\hat{x}^{k+1}-\hat{x}^{k}\rangle.\end{array} Combining the last two expressions, we can show that \begin{array}[]{lcl}\mathcal{T}_{[3]}&:=&\|\hat{x}^{k}-x^{\star}+t_{k}(y^{k}-\hat{x}^{k})\|^{2}-\|\hat{x}^{k+1}-x^{\star}+t_{k+1}(y^{k+1}-\hat{x}^{k+1})\|^{2}\\ &=&\left[(t_{k}-1)^{2}-t_{k+1}^{2}\theta_{k}^{2}\right]\|\hat{x}^{k+1}-\hat{x}^{k}\|^{2}+(t_{k}^{2}-t_{k+1}^{2}\nu_{k}^{2})\|y^{k}-\hat{x}^{k+1}\|^{2}\\ &&+{~}2(t_{k}-1-t_{k+1}\theta_{k})\langle\hat{x}^{k+1}-\hat{x}^{k},\hat{x}^{k+1}-x^{\star}\rangle\\ &&+{~}2(t_{k}-t_{k+1}\nu_{k})\langle y^{k}-\hat{x}^{k+1},\hat{x}^{k+1}-x^{\star}\rangle\\ &&+{~}2\left[t_{k}(t_{k}-1)-t_{k+1}^{2}\theta_{k}\nu_{k}\right]\langle y^{k}-\hat{x}^{k+1},\hat{x}^{k+1}-\hat{x}^{k}\rangle.\end{array} Now, if we define the following function: \begin{array}[]{lcl}\mathcal{Q}_{k}&:=&b_{k}\langle w^{k},\hat{x}^{k}-y^{k}\rangle+\|\hat{x}^{k}-x^{\star}+t_{k}(y^{k}-\hat{x}^{k})\|^{2},\end{array} then utilizing this function \mathcal{Q}_{k} and \hat{x}^{k+1}-y^{k+1}=-\theta_{k}(\hat{x}^{k+1}-\hat{x}^{k})-\nu_{k}(y^{k}-\hat{x}^{k+1}) from ( GAEG ), we can further derive from \mathcal{T}_{[3]} that \hskip-4.30554pt\begin{array}[]{lcl}\mathcal{Q}_{k}-\mathcal{Q}_{k+1}&=&\left[(t_{k}-1)^{2}-t_{k+1}^{2}\theta_{k}^{2}\right]\|\hat{x}^{k+1}-\hat{x}^{k}\|^{2}\\ &&+{~}(t_{k}^{2}-t_{k+1}^{2}\nu_{k}^{2})\|y^{k}-\hat{x}^{k+1}\|^{2}+b_{k}\langle w^{k+1}-w^{k},\hat{x}^{k+1}-\hat{x}^{k}\rangle\\ &&+{~}b_{k+1}\langle\nu_{k}w^{k+1}-\theta_{k}w^{k},y^{k}-\hat{x}^{k+1}\rangle\\ &&+{~}\left(b_{k+1}\theta_{k}-b_{k}\right)\big[\langle w^{k+1},\hat{x}^{k+1}-\hat{x}^{k}\rangle+\langle w^{k},y^{k}-\hat{x}^{k+1}\rangle\big]\\ &&+{~}2(t_{k}-1-t_{k+1}\theta_{k})\langle\hat{x}^{k+1}-\hat{x}^{k},\hat{x}^{k+1}-x^{\star}\rangle\\ &&+{~}2(t_{k}-t_{k+1}\nu_{k})\langle y^{k}-\hat{x}^{k+1},\hat{x}^{k+1}-x^{\star}\rangle\\ &&+{~}2\left[t_{k}(t_{k}-1)-t_{k+1}^{2}\theta_{k}\nu_{k}\right]\langle y^{k}-\hat{x}^{k+1},\hat{x}^{k+1}-\hat{x}^{k}\rangle.\end{array}\hskip-8.61108pt (147) Since \theta_{k}=\frac{t_{k}-1}{t_{k+1}} and \nu_{k}=\frac{t_{k}}{t_{k+1}} from ( 71 ), and we choose b_{k+1}=\frac{b_{k}}{\theta_{k}} , we have \left\{\begin{array}[]{lclclcl}t_{k}-t_{k+1}\nu_{k}&=&0,&&\quad t_{k}(t_{k}-1)-t_{k+1}^{2}\theta_{k}\nu_{k}&=&0,\\ t_{k}-1-t_{k+1}\theta_{k}&=&0,&\text{and}&\quad b_{k+1}\theta_{k}-b_{k}&=&0.\end{array}\right. (148) Substituting ( 148 ) into ( 147 ), it reduces to \begin{array}[]{lcl}\mathcal{Q}_{k}-\mathcal{Q}_{k+1}&=&b_{k}\langle w^{k+1}-w^{k},\hat{x}^{k+1}-\hat{x}^{k}\rangle\\ &&+{~}b_{k+1}\langle\nu_{k}w^{k+1}-\theta_{k}w^{k},y^{k}-\hat{x}^{k+1}\rangle.\end{array} (149) Applying \hat{x}^{k}=x^{k}-\lambda z^{k} from ( GAEG ) to {\tiny\textcircled{1}} , the \rho -co-hypomonotonicity of \Phi to {\tiny\textcircled{2}} , and Young’s inequality to {\tiny\textcircled{3}} , for any \hat{c}>0 , we can derive that \begin{array}[]{lcl}\mathcal{T}_{[4]}&:=&\langle w^{k+1}-w^{k},\hat{x}^{k+1}-\hat{x}^{k}\rangle\\ &\overset{\tiny{\tiny\textcircled{1}}}{=}&\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle-\lambda\langle w^{k+1}-w^{k},z^{k+1}-z^{k}\rangle\\ &\overset{\tiny{\tiny\textcircled{2}}}{\geq}&-(\lambda+\rho)\|w^{k+1}-w^{k}\|^{2}-\lambda\langle w^{k+1}-w^{k},z^{k+1}-w^{k+1}\rangle\\ &&+{~}\lambda\langle w^{k+1}-w^{k},z^{k}-w^{k}\rangle\\ &\overset{\tiny{\tiny\textcircled{3}}}{\geq}&-\big[(1+\hat{c})\lambda+\rho\big]\|w^{k+1}-w^{k}\|^{2}-\frac{\lambda}{2\hat{c}}\|z^{k+1}-w^{k+1}\|^{2}-\frac{\lambda}{2\hat{c}}\|z^{k}-w^{k}\|^{2}\\ &=&-\big[(1+\hat{c})\lambda+\rho\big]\big[\|w^{k+1}\|^{2}+\|w^{k}\|^{2}-2\langle w^{k+1},w^{k}\rangle\big]\\ &&-{~}\frac{\lambda}{2\hat{c}}\|u^{k+1}-Fx^{k+1}\|^{2}-\frac{\lambda}{2\hat{c}}\|u^{k}-Fx^{k}\|^{2}.\end{array} Alternatively, combining the first two lines of ( GAEG ) we get \begin{array}[]{lcl}y^{k}-\hat{x}^{k+1}&=&\lambda z^{k+1}+\eta\hat{w}^{k+1}-\eta\gamma_{k}z^{k}\\ &=&\lambda w^{k+1}+\eta\hat{w}^{k+1}-\eta\gamma_{k}w^{k}+\lambda(z^{k+1}-w^{k+1})-\eta\gamma_{k}(z^{k}-w^{k})\\ &=&\lambda w^{k+1}+\eta\hat{w}^{k+1}-\eta\gamma_{k}w^{k}+\lambda(u^{k+1}-Fx^{k+1})-\eta\gamma_{k}(u^{k}-Fx^{k}).\end{array} Utilizing this expression and Young’s inequality in {\tiny\textcircled{1}} , for any \beta>0 , we obtain \begin{array}[]{lcl}\mathcal{T}_{[5]}&:=&\langle\nu_{k}w^{k+1}-\theta_{k}w^{k},y^{k}-\hat{x}^{k+1}\rangle\\ &=&\langle\nu_{k}w^{k+1}-\theta_{k}w^{k},\lambda w^{k+1}+\eta\hat{w}^{k+1}-\eta\gamma_{k}w^{k}\rangle\\ &&+{~}\lambda\langle\nu_{k}w^{k+1}-\theta_{k}w^{k},u^{k+1}-Fx^{k+1}\rangle-\eta\gamma_{k}\langle\nu_{k}w^{k+1}-\theta_{k}w^{k},u^{k}-Fx^{k}\rangle\\ &\overset{\tiny{\tiny\textcircled{1}}}{\geq}&\lambda\nu_{k}\|w^{k+1}\|^{2}+\eta\gamma_{k}\theta_{k}\|w^{k}\|^{2}-\left(\eta\gamma_{k}\nu_{k}+\lambda\theta_{k}\right)\langle w^{k+1},w^{k}\rangle\\ &&-{~}\eta\theta_{k}\langle w^{k},\hat{w}^{k+1}\rangle+\eta\nu_{k}\langle w^{k+1},\hat{w}^{k+1}\rangle-\frac{\beta}{\nu_{k}}\|\nu_{k}w^{k+1}-\theta_{k}w^{k}\|^{2}\\ &&-{~}\frac{\lambda^{2}\nu_{k}}{2\beta}\|u^{k+1}-Fx^{k+1}\|^{2}-\frac{\eta^{2}\gamma_{k}^{2}\nu_{k}}{2\hat{\beta}}\|u^{k}-Fx^{k}\|^{2}\\ &=&\big(\lambda-\beta\big)\nu_{k}\|w^{k+1}\|^{2}+\theta_{k}\big(\eta\gamma_{k}-\frac{\beta\theta_{k}}{\nu_{k}}\big)\|w^{k}\|^{2}+\eta(\nu_{k}-\theta_{k})\langle\hat{w}^{k+1},w^{k+1}\rangle\\ &&-{~}\big[\eta\gamma_{k}\nu_{k}+\lambda\theta_{k}-2\beta\theta_{k}\big]\langle w^{k+1},w^{k}\rangle-\frac{\lambda^{2}\nu_{k}}{2\beta}\|u^{k+1}-Fx^{k+1}\|^{2}\\ &&-{~}\frac{\eta^{2}\gamma_{k}^{2}\nu_{k}}{2\beta}\|u^{k}-Fx^{k}\|^{2}+\eta\theta_{k}\langle\hat{w}^{k+1},w^{k+1}-w^{k}\rangle.\end{array} Substituting \mathcal{T}_{[4]} and \mathcal{T}_{[5]} above into ( 149 ), and using the facts that b_{k}=b_{k+1}\theta_{k} and \gamma_{k}=\frac{t_{k}-1}{t_{k}}=\frac{\theta_{k}}{\nu_{k}} from ( 71 ), we can prove that \hskip-8.61108pt\begin{array}[]{lcl}\mathcal{Q}_{k}-\mathcal{Q}_{k+1}&\geq&-{~}\frac{b_{k}}{2}\big(\frac{\lambda}{\hat{c}}+\frac{\eta^{2}\gamma_{k}}{\beta}\big)\|u^{k}-Fx^{k}\|^{2}-\frac{b_{k}}{2}\big(\frac{\lambda}{\hat{c}}+\frac{\lambda^{2}}{\beta\gamma_{k}}\big)\|u^{k+1}-Fx^{k+1}\|^{2}\\ &&-{~}b_{k}\big[(1+\hat{c})\lambda+\rho+\beta\gamma_{k}-\eta\gamma_{k}\big]\|w^{k}\|^{2}\\ &&+{~}b_{k}\big[\frac{\lambda-\beta}{\gamma_{k}}-(1+\hat{c})\lambda-\rho\big]\|w^{k+1}\|^{2}\\ &&+{~}b_{k}\big[(1+2\hat{c})\lambda+2\rho+2\beta-\eta\big]\langle w^{k+1},w^{k}\rangle\\ &&+{~}\eta b_{k}\langle\hat{w}^{k+1},w^{k+1}-w^{k}\rangle+\frac{\eta b_{k}(1-\gamma_{k})}{\gamma_{k}}\langle\hat{w}^{k+1},w^{k+1}\rangle.\end{array}\hskip-30.1388pt (150) Next, by the Lipschitz continuity of F , the first line of ( GAEG ), Young’s inequality, and z^{k}-w^{k}=u^{k}-Fx^{k} , for any c>0 , one can show that \begin{array}[]{lcl}\|w^{k+1}-\hat{w}^{k+1}\|^{2}&=&\|Fx^{k+1}-Fy^{k}\|^{2}\leq L^{2}\|x^{k+1}-y^{k}\|^{2}\\ &=&L^{2}\eta^{2}\|\hat{w}^{k+1}-\gamma_{k}w^{k}-\gamma_{k}(z^{k}-w^{k})\|^{2}\\ &\leq&(1+c)L^{2}\eta^{2}\|\hat{w}^{k+1}-\gamma_{k}w^{k}\|^{2}+\frac{(1+c)L^{2}\eta^{2}\gamma_{k}^{2}}{c}\|u^{k}-Fx^{k}\|^{2}.\end{array} Denote M_{c}:=(1+c)L^{2} . Partially expanding the last expression yields \begin{array}[]{lcl}0&\geq&\|w^{k+1}\|^{2}+(1-M_{c}\eta^{2})\|\hat{w}^{k+1}\|^{2}-M_{c}\eta^{2}\gamma_{k}^{2}\|w^{k}\|^{2}-\frac{M_{c}\eta^{2}\gamma_{k}^{2}}{c}\|u^{k}-Fx^{k}\|^{2}\\ &&-{~}2(1-M_{c}\eta^{2}\gamma_{k})\langle w^{k+1},\hat{w}^{k+1}\rangle-2M_{c}\eta^{2}\gamma_{k}\langle\hat{w}^{k+1},w^{k+1}-w^{k}\rangle.\end{array} Multiplying both sides of this inequality by \frac{b_{k}}{2M_{c}\eta\gamma_{k}} and adding the result to ( 150 ), we arrive at \begin{array}[]{lcl}\mathcal{Q}_{k}-\mathcal{Q}_{k+1}&\geq&-\frac{b_{k}}{2}\big(\frac{\lambda}{\hat{c}}+\frac{\eta^{2}\gamma_{k}}{\beta}+\frac{\eta\gamma_{k}}{c}\big)\|u^{k}-Fx^{k}\|^{2}\\ &&-{~}\frac{b_{k}}{2}\big(\frac{\lambda}{\hat{c}}+\frac{\lambda^{2}}{\beta\gamma_{k}}\big)\|u^{k+1}-Fx^{k+1}\|^{2}\\ &&-{~}b_{k}\big[(1+\hat{c})\lambda+\rho+\beta\gamma_{k}-\frac{\eta\gamma_{k}}{2}\big]\|w^{k}\|^{2}\\ &&+{~}b_{k}\big[\frac{1}{2M_{c}\eta\gamma_{k}}+\frac{\lambda-\beta}{\gamma_{k}}-(1+\hat{c})\lambda-\rho\big]\|w^{k+1}\|^{2}\\ &&-{~}b_{k}\big[\eta-(1+2\hat{c})\lambda-2\rho-2\beta\big]\langle w^{k+1},w^{k}\rangle\\ &&+{~}\frac{b_{k}(1-M_{c}\eta^{2})}{2M_{c}\eta\gamma_{k}}\big[\|\hat{w}^{k+1}\|^{2}-2\langle\hat{w}^{k+1},w^{k+1}\rangle\big].\end{array} Assume that we choose \mu\geq 0 such that \eta-(1+2\hat{c})\lambda-2\rho-2\beta=2\mu . This choice leads to \eta:=(1+2\hat{c})\lambda+2\rho+2\beta+2\mu as stated in ( 71 ). Moreover, utilizing this relation and the following two identities \begin{array}[]{lcl}2\langle w^{k+1},w^{k}\rangle&=&\|w^{k+1}\|^{2}+\|w^{k}\|^{2}-\|w^{k+1}-w^{k}\|^{2},\\ \|\hat{w}^{k+1}\|^{2}-2\langle\hat{w}^{k+1},w^{k+1}\rangle&=&\|w^{k+1}-\hat{w}^{k+1}\|^{2}-\|w^{k+1}\|^{2},\end{array} the last inequality becomes \begin{array}[]{lcl}\mathcal{Q}_{k}-\mathcal{Q}_{k+1}&\geq&-\frac{b_{k}}{2}\big(\frac{\lambda}{\hat{c}}+\frac{\eta^{2}\gamma_{k}}{\beta}+\frac{\eta\gamma_{k}}{c}\big)\|u^{k}-Fx^{k}\|^{2}\\ &&-{~}\frac{b_{k}}{2}\big(\frac{\lambda}{\hat{c}}+\frac{\lambda^{2}}{\beta\gamma_{k}}\big)\|u^{k+1}-Fx^{k+1}\|^{2}\\ &&-{~}\frac{a_{k}}{2}\|w^{k}\|^{2}+\frac{\hat{a}_{k}}{2}\|w^{k+1}\|^{2}+\mu b_{k}\|w^{k+1}-w^{k}\|^{2}\\ &&+{~}\frac{b_{k}(1-M_{c}\eta^{2})}{2M_{c}\eta\gamma_{k}}\|w^{k+1}-\hat{w}^{k+1}\|^{2},\end{array} (151) where \left\{\begin{array}[]{lcl}a_{k}&:=&2b_{k}\big[(1+\hat{c})\lambda+\rho+\beta\gamma_{k}-\frac{\eta\gamma_{k}}{2}+\mu\big],\\ \hat{a}_{k}&:=&2b_{k}\big[\frac{\eta}{2\gamma_{k}}+\frac{\lambda-\beta}{\gamma_{k}}-(1+\hat{c})\lambda-\rho-\mu\big].\end{array}\right. Finally, using ( 71 ), one can easily show that a_{k}=\frac{b_{k}}{t_{k}}\big[\lambda t_{k}+(1+2\hat{c})\lambda+2\rho+2\mu\big] and \hat{a}_{k}=\frac{b_{k+1}}{t_{k+1}}\big[\lambda t_{k+1}+(1+2\hat{c})\lambda+2\rho+2\mu\big]=a_{k+1} . In this case, substituting \mathcal{P}_{k} from ( 65 ) into ( 151 ), we obtain ( 66 ). \square E.2 The Proof of Lemma 8 Substituting ( 71 ) and b_{k}=b_{k+1}\theta_{k} into ( 66 ), we get \begin{array}[]{lcl}\mathcal{P}_{k}-\mathcal{P}_{k+1}&\geq&\frac{c_{k+1}}{2}\|u^{k+1}-Fx^{k+1}\|^{2}-\frac{c_{k}}{2}\|u^{k}-Fx^{k}\|^{2}\\ &&-{~}\frac{1}{2}\big[c_{k+1}+b_{k}\big(\frac{\lambda}{\hat{c}}+\frac{\lambda^{2}}{\beta\gamma_{k}}\big)\big]\|u^{k+1}-Fx^{k+1}\|^{2}\\ &&+{~}\mu b_{k}\|w^{k+1}-w^{k}\|^{2}+\frac{b_{k}(1-M_{c}\eta^{2})}{2M_{c}\eta\gamma_{k}}\|w^{k+1}-\hat{w}^{k+1}\|^{2}.\end{array} Since b_{k+1}=\frac{b_{k}}{\theta_{k}}=\frac{b_{k}t_{k+1}}{t_{k}-1} and t_{k+1}=t_{k}+1 from ( 71 ), one has \begin{array}[]{lcl}c_{k+1}+b_{k}\big(\frac{\lambda}{\hat{c}}+\frac{\lambda^{2}}{\beta\gamma_{k}}\big)=\frac{b_{k}t_{k}}{t_{k-1}}\big(\frac{2\lambda}{\hat{c}}+\frac{\eta^{2}}{\beta}+\frac{\eta}{c}+\frac{\lambda}{\beta}\big).\end{array} Using this relation, ( 60 ), and \mathcal{L}_{k} from ( 67 ) into the last inequality and noting that Fx^{k+1}-Fy^{k}=w^{k+1}-\hat{w}^{k+1} and \frac{t_{0}-1}{t_{0}}\leq\frac{t_{k-1}}{t_{k}}\leq 1 , we can show that \begin{array}[]{lcl}\mathcal{L}_{k}-\mathcal{L}_{k+1}&\geq&\frac{b_{k}t_{k}}{2t_{k-1}}\big[\frac{1-M_{c}\eta^{2}}{M_{c}\eta\ }-\kappa\big(\frac{2\lambda}{\hat{c}}+\frac{\lambda^{2}}{\beta}+\frac{\eta^{2}}{\beta}+\frac{\eta}{c}\big)\big]\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\frac{b_{k}t_{k}}{2t_{k-1}}\big[\frac{2\mu(t_{0}-1)}{t_{0}}-\hat{\kappa}\big(\frac{2\lambda}{\hat{c}}+\frac{\lambda^{2}}{\beta}+\frac{\eta^{2}}{\beta}+\frac{\eta}{c}\big)\big]\|w^{k+1}-w^{k}\|^{2}.\end{array} This exactly proves ( 68 ). \square E.3 The Proof of Lemma 9 By Young’s inequality and z^{k}-w^{k}=u^{k}-Fx^{k} , for any \omega>0 , we get \begin{array}[]{lcl}-\langle w^{k},z^{k}\rangle&=&-\langle w^{k},z^{k}-w^{k}\rangle-\|w^{k}\|^{2}\geq-\frac{2+\omega}{2}\|w^{k}\|^{2}-\frac{1}{2\omega}\|Fx^{k}-u^{k}\|^{2}.\end{array} Utilizing this inequality, \hat{x}^{k}=x^{k}-\lambda z^{k} from ( GAEG ), and \langle w^{k},x^{k}-x^{\star}\rangle\geq-\rho\|w^{k}\|^{2} from the \rho -co-hypomonotonicity of \Phi , we can show from ( 67 ) that \begin{array}[]{lcl}\mathcal{L}_{k}&=&\|\hat{x}^{k}+t_{k}(y^{k}-\hat{x}^{k})-x^{\star}-\frac{b_{k}}{2t_{k}}w^{k}\|^{2}+\big(\frac{a_{k}}{2}-\frac{b_{k}^{2}}{4t_{k}^{2}}\big)\|w^{k}\|^{2}\\ &&+{~}\frac{c_{k}}{2}\|Fx^{k}-u^{k}\|^{2}+\frac{b_{k}}{t_{k}}\langle w^{k},x^{k}-x^{\star}\rangle-\frac{\lambda b_{k}}{t_{k}}\langle w^{k},z^{k}\rangle\\ &\geq&\frac{1}{2}\big[a_{k}-\frac{b_{k}^{2}}{2t_{k}^{2}}-\frac{[\lambda(2+\omega)+2\rho]b_{k}}{t_{k}}\big]\|w^{k}\|^{2}+\frac{1}{2}\big(c_{k}-\frac{\lambda b_{k}}{\omega t_{k}}\big)\|Fx^{k}-u^{k}\|^{2}.\end{array} (152) Since t_{k+1}=t_{k}+1=k+t_{0}+1 and b_{k+1}=\frac{b_{k}}{\theta_{k}}=\frac{b_{k}t_{k+1}}{t_{k}-1}=\frac{b_{k}t_{k+1}t_{k}}{t_{k}t_{k-1}} , by induction, we get b_{k}=\frac{b_{0}t_{k}t_{k-1}}{t_{0}t_{-1}}=\frac{b_{0}t_{k}t_{k-1}}{t_{0}(t_{0}-1)} . Moreover, we also have a_{k}=\frac{b_{k}}{t_{k}}\big(\lambda t_{k}+(1+2\hat{c})\lambda+2\rho+2\mu\big) and c_{k}:=b_{k}\big(\frac{\lambda}{\hat{c}}+\frac{\eta^{2}\gamma_{k}}{\beta}+\frac{\eta\gamma_{k}}{c}\big)=\frac{b_{k}}{t_{k}}\big[\frac{\lambda}{\hat{c}}t_{k}+(\frac{\eta}{\beta}+\frac{1}{c})\eta t_{k-1}\big] . Using these formulae, and choosing \omega:=\frac{1}{4} and b_{0}:=\frac{3\lambda t_{0}(t_{0}-1)}{2} , we can easily show that \left\{\begin{array}[]{lcl}a_{k}-\frac{b_{k}^{2}}{2t_{k}^{2}}-\frac{[\lambda(2+\omega)+2\rho]b_{k}}{t_{k}}&=&\frac{b_{k}}{t_{k}}\big[\frac{\lambda t_{k}}{4}+(2\hat{c}-\frac{1}{2})\lambda+2\mu\big],\\ c_{k}-\frac{\lambda b_{k}}{\omega t_{k}}&=&\frac{b_{k}}{t_{k}}\big[\lambda(\frac{t_{k}}{\hat{c}}-4)+(\frac{\eta}{\beta}+\frac{1}{c})\eta t_{k-1}\big].\end{array}\right. Substituting these expressions into ( 152 ), we obtain ( 69 ). \square E.4 The Proof of Theorem 6.1 Let us choose c:=1 , \hat{c}:=\frac{1}{4} , \beta:=\frac{\eta}{4} , and \mu:=\frac{\lambda+4\sigma\rho}{4} in Lemma 8 , where \sigma:=\frac{5}{24} . Then, we have \eta:=(1+2\hat{c})\lambda+2\rho+2\beta+2\mu=2\lambda+\frac{\eta}{2}+2(1+\sigma)\rho , leading to \eta:=4[\lambda+(1+\sigma)\rho] as shown in ( 71 ). Using these choices, we can also show that \Lambda:=\frac{2\lambda}{\hat{c}}+\frac{\lambda^{2}}{\beta}+\frac{\eta^{2}}{\beta}+\frac{\eta}{c}=28\lambda+20(1+\sigma)\rho+\frac{\lambda^{2}}{\lambda+(1+\sigma)\rho} . Since t_{k}:=k+r (i.e. t_{0}=r ), to guarantee the nonnegativity of the right-hand side of ( 68 ), we needs to enforce the following two conditions: \begin{array}[]{lcl}\frac{1}{M_{c}\eta}-\eta-\kappa\Lambda\geq 0\quad\textrm{and}\quad\frac{(r-1)(\lambda+4\sigma\rho)}{2r}-\hat{\kappa}\Lambda\geq 0.\end{array} (153) The first condition of ( 153 ) becomes \begin{array}[]{lcl}4(7\kappa+1)\lambda+4(1+\sigma)(5\kappa+1)\rho+\frac{\kappa\lambda^{2}}{\lambda+(1+\sigma)\rho}\leq\frac{1}{4M_{c}[\lambda+(1+\sigma)\rho]}.\end{array} Since M_{c}=2L^{2} and \frac{\lambda}{\lambda+(1+\sigma)\rho}\leq 1 , the last condition holds if \begin{array}[]{lcl}8L^{2}[\lambda+(1+\sigma)\rho]\big[(29\kappa+4)\lambda+4(1+\sigma)(5\kappa+1)\rho\big]\leq 1.\end{array} This condition holds if \begin{array}[]{lcl}4(1+\sigma)\sqrt{2(5\kappa+1)}L\rho<1\quad\text{and}\quad 0<\lambda\leq\bar{\lambda}:=\frac{2\bar{c}}{\bar{b}+\sqrt{\bar{b}^{2}+4\bar{c}}},\end{array} where \bar{b}:=\frac{(1+\sigma)(49\kappa+8)\rho}{29\kappa+4} and \bar{c}:=\frac{1-32(1+\sigma)^{2}(5\kappa+1)L^{2}\rho^{2}}{8L^{2}(29\kappa+4)} as given in ( 70 ). Since \sigma=\frac{5}{24} , the condition 4(1+\sigma)\sqrt{2(5\kappa+1)}L\rho<1 is equivalent to L\rho<\frac{6}{29\sqrt{2(5\kappa+1)}} as stated in ( 71 ). Using again \sigma=\frac{5}{24} , we can easily bound \Lambda=28\lambda+20(1+\sigma)\rho+\frac{\lambda^{2}}{\lambda+(1+\sigma)\rho}\leq 29\lambda+20(1+\sigma)\rho=29(\lambda+4\sigma\rho\big) . Thus if we choose 0\leq\hat{\kappa}\leq\frac{r-1}{58r} as stated in ( 71 ), then 0\leq\hat{\kappa}\leq\frac{(r-1)(\lambda+4\sigma\rho)}{2r\Lambda} . Hence, the second condition of ( 153 ) holds. Under the above conditions, ( 68 ) reduces to \mathcal{L}_{k+1}\leq\mathcal{L}_{k} for all k\geq 0 . Since t_{0}:=r>1 , b_{k}:=\frac{3\lambda}{2}t_{k}t_{k-1} as computed in Lemma 9 , c:=1 , \hat{c}:=\frac{1}{4} , \mu:=\frac{\lambda+4\sigma\rho}{4} , \beta:=\frac{\eta}{4} , and \eta:=4[\lambda+(1+\sigma)\rho] , from ( 69 ) we can prove that \begin{array}[]{lcl}\mathcal{L}_{k}&\geq&\frac{b_{k}}{8t_{k}}\big[\lambda(t_{k}+2)+8\sigma\rho\big]\|w^{k}\|^{2}+\frac{2[6\lambda+5(1+\sigma)\rho]b_{k}t_{k-1}}{t_{k}}\|u^{k}-Fx^{k}\|^{2}\\ &\geq&\frac{3\lambda^{2}(k+r-1)(k+r+2)}{16}\|w^{k}\|^{2}.\end{array} (154) Finally, since y^{-1}=\hat{x}^{-1}=\hat{x}^{0}:=x^{0} , using the last line of ( GAEG ) at k=-1 , we can easily show that y^{0}=\hat{x}^{0} , leading to \mathcal{L}_{0}=\frac{a_{0}}{2}\|Fx^{0}+\xi^{0}\|^{2}+\|x^{0}-x^{\star}\|^{2} . Since \mathcal{L}_{k+1}\leq\mathcal{L}_{k} , by induction, we have \mathcal{L}_{k}\leq\mathcal{L}_{0}=\frac{a_{0}}{2}\|Fx^{0}+\xi^{0}\|^{2}+\|x^{0}-x^{\star}\|^{2}=\frac{3\lambda(r-1)[(r+2)\lambda+2(1+\sigma)\rho]}{4}\|Fx^{0}+\xi^{0}\|^{2}+\|x^{0}-x^{\star}\|^{2}=:\frac{\mathcal{R}_{0}^{2}}{16} . Combining this bound and ( 154 ), we obtain \begin{array}[]{lcl}\|w^{k}\|^{2}&\leq&\frac{\mathcal{R}_{0}^{2}}{3\lambda^{2}(k+r-1)(k+r+2)},\end{array} which is exactly ( 72 ) after substituting w^{k}:=Fx^{k}+\xi^{k} . \square E.5 The Proof of Corollary 4 \mathrm{(i)} Since we choose u^{k}:=Fx^{k} , ( 60 ) holds with \kappa=\hat{\kappa}=0 . We can choose c:=0 , \hat{c}:=0 , \beta:=0 , and \mu:=0 in Lemma 8 . In this case, we get \eta:=\lambda+2\rho , M_{c}=L^{2} , and c_{k}=0 . Moreover, ( 68 ) reduces to \hskip-4.30554pt\begin{array}[]{lcl}\mathcal{L}_{k}-\mathcal{L}_{k+1}&\geq&\frac{b_{k}t_{k}}{2t_{k-1}}\big(\frac{1}{L^{2}\eta}-\eta\big)\|w^{k+1}-\hat{w}^{k+1}\|^{2}.\end{array} Therefore, we can choose 0<\lambda\leq\frac{1}{L}-2\rho , provided that 2L\rho<1 . Furtheremore, from ( 69 ), we also have \begin{array}[]{lcl}\frac{3\lambda^{2}(k+r-2)(k+r-1)}{16}\|w^{k}\|^{2}&\leq&\mathcal{L}_{k}\leq\mathcal{L}_{0}=\frac{a_{0}}{2}\|w^{0}\|^{2}+\|x^{0}-x^{\star}\|^{2}.\end{array} Since a_{0}:=\frac{3\lambda}{2}(r-1)[\lambda(r+1)+2\rho] , the last inequality implies ( 73 ). \mathrm{(ii)} Since we choose u^{k}:=Fy^{k-1} , ( 60 ) holds with \kappa=1 and \hat{\kappa}=0 . In this case, we choose c:=1 , \hat{c}:=\frac{1}{4} , \beta:=\frac{\eta}{4} , and \mu:=0 . Then, we get \eta=3\lambda+4\rho and M_{c}=2L^{2} . Moreover, ( 68 ) reduces to \hskip-4.30554pt\begin{array}[]{lcl}\mathcal{L}_{k}-\mathcal{L}_{k+1}&\geq&\frac{b_{k}t_{k}}{2t_{k-1}}\big(\frac{1}{2L^{2}\eta}-6\eta-8\lambda-\frac{4\lambda^{2}}{\eta}\big)\|w^{k+1}-\hat{w}^{k+1}\|^{2}.\end{array} To guarantee \Gamma:=\frac{1}{2L^{2}\eta}-6\eta-8\lambda-\frac{4\lambda^{2}}{\eta}\geq 0 , we need to impose 8\sqrt{3}L\rho<1 and 0<\lambda\leq\bar{\lambda}:=\frac{2\bar{c}}{\bar{b}+\sqrt{\bar{b}^{2}+4\bar{c}}} as stated, where \bar{b}:=\frac{272\rho}{123} and \bar{c}:=\frac{1-129L^{2}\rho^{2}}{164L^{2}} . Moreover, we also have \mathcal{L}_{k+1}\leq\mathcal{L}_{k}\leq\mathcal{L}_{0} by induction. Using this relation and ( 69 ), we can conclude that \begin{array}[]{lcl}\frac{3\lambda^{2}(k+r-1)(k+r)}{16}\|w^{k}\|^{2}&\leq&\mathcal{L}_{k}\leq\mathcal{L}_{0}=\frac{a_{0}}{2}\|w^{0}\|^{2}+\|x^{0}-x^{\star}\|^{2}.\end{array} Since a_{0}:=\frac{3\lambda}{4}(r-1)[\lambda(2r+3)+4\rho] , the last inequality implies ( 74 ). \square Appendix F The Proof of Technical Results in Section 7 This appendix provides the proof of all results in Section 7 . F.1 The Proof of Lemma 10 First, from the first line y^{k}=x^{k+1}+\eta d^{k} of ( GAEG+ ), we have \begin{array}[]{lcl}\mathcal{T}_{[1]}&:=&\|r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})\|^{2}\\ &=&\|r(x^{k}-x^{\star})+t_{k}(x^{k+1}-x^{k}+\eta d^{k})\|^{2}\\ &=&\|r(x^{k+1}-x^{\star})+(t_{k}-r)(x^{k+1}-x^{k})+\eta t_{k}d^{k}\|^{2}\\ &=&r^{2}\|x^{k+1}-x^{\star}\|^{2}+(t_{k}-r)^{2}\|x^{k+1}-x^{k}\|^{2}+\eta^{2}t_{k}^{2}\|d^{k}\|^{2}\\ &&+{~}2r(t_{k}-r)\langle x^{k+1}-x^{k},x^{k+1}-x^{\star}\rangle+2\eta rt_{k}\langle d^{k},x^{k+1}-x^{\star}\rangle\\ &&+{~}2\eta t_{k}(t_{k}-r)\langle d^{k},x^{k+1}-x^{k}\rangle.\end{array} Second, from the second line y^{k+1}-x^{k+1}=\theta_{k}(x^{k+1}-x^{k})-p^{k} of ( GAEG+ ) and t_{k+1}\theta_{k}=t_{k}-r-\mu , we can also derive that \begin{array}[]{lcl}\mathcal{T}_{[2]}&:=&\|r(x^{k+1}-x^{\star})+t_{k+1}(y^{k+1}-x^{k+1})\|^{2}\\ &=&\|r(x^{k+1}-x^{\star})+t_{k+1}\theta_{k}(x^{k+1}-x^{k})-t_{k+1}p^{k}\|^{2}\\ &=&r^{2}\|x^{k+1}-x^{\star}\|^{2}+(t_{k}-r-\mu)^{2}\|x^{k+1}-x^{k}\|^{2}+t_{k+1}^{2}\|p^{k}\|^{2}\\ &&+{~}2r(t_{k}-r-\mu)\langle x^{k+1}-x^{k},x^{k+1}-x^{\star}\rangle\\ &&-{~}2rt_{k+1}\langle p^{k},x^{k+1}-x^{\star}\rangle-2t_{k+1}(t_{k}-r-\mu)\langle p^{k},x^{k+1}-x^{k}\rangle.\end{array} Third, combining \mathcal{T}_{[1]} and \mathcal{T}_{[2]} , and the identity \mu r\|x^{k}-x^{\star}\|^{2}-\mu r\|x^{k+1}-x^{\star}\|^{2}=\mu r\|x^{k+1}-x^{k}\|^{2}-2r\mu\langle x^{k+1}-x^{k},x^{k+1}-x^{\star}\rangle , we can prove that \hskip-8.61108pt\begin{array}[]{lcl}\mathcal{T}_{[3]}&:=&\|r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})\|^{2}-\|r(x^{k+1}-x^{\star})+t_{k+1}(y^{k+1}-x^{k+1})\|^{2}\\ &&+{~}\mu r\|x^{k}-x^{\star}\|^{2}-\mu r\|x^{k+1}-x^{\star}\|^{2}\\ &=&\mu(2t_{k}-r-\mu)\|x^{k+1}-x^{k}\|^{2}+\eta^{2}t_{k}^{2}\|d^{k}\|^{2}-t_{k+1}^{2}\|p^{k}\|^{2}\\ &&+{~}2r\langle\eta t_{k}d^{k}+t_{k+1}p^{k},x^{k+1}-x^{\star}\rangle\\ &&+{~}2\langle\eta(t_{k}-r)t_{k}d^{k}+t_{k+1}(t_{k}-r-\mu)p^{k},x^{k+1}-x^{k}\rangle.\end{array}\hskip-25.83325pt (155) Fourth, we process the last two terms ( 155 ) as follows. We first utilize d^{k} and p^{k} from ( GAEG+ ) and the update rule ( 77 ) to show that \begin{array}[]{lcl}\mathcal{T}_{[4]}&:=&\langle\eta t_{k}d^{k}+t_{k+1}p^{k},x^{k+1}-x^{\star}\rangle\\ &=&[(\eta-\beta)t_{k}-\delta]\langle z^{k+1},x^{k+1}-x^{\star}\rangle-[(\eta-\beta)t_{k}-\eta(r-1)]\langle z^{k},x^{k}-x^{\star}\rangle\\ &&-{~}[(\eta-\beta)t_{k}-\eta(r-1)]\langle z^{k},x^{k+1}-x^{k}\rangle.\end{array} Next, using again ( 77 ), we get \begin{array}[]{lcl}\mathcal{T}_{[5]}&:=&\langle\eta t_{k}(t_{k}-r)d^{k}+t_{k+1}(t_{k}-r-\mu)p^{k},x^{k+1}-x^{k}\rangle\\ &=&\big[((\eta-\beta)t_{k}-\delta)(t_{k}-r-\mu)+\mu\eta t_{k}\big]\langle z^{k+1},x^{k+1}-x^{k}\rangle\\ &&-{~}\big[\eta(t_{k}-r)(t_{k}-r+1)-\beta t_{k}(t_{k}-r-\mu)\big]\langle z^{k},x^{k+1}-x^{k}\rangle\\ &&+{~}\mu\eta t_{k}\langle\hat{w}^{k+1}-z^{k+1},x^{k+1}-x^{k}\rangle.\end{array} Then, substituting \mathcal{T}_{[4]} and \mathcal{T}_{[5]} into \mathcal{T}_{[3]} of ( 155 ), we arrive at \begin{array}[]{lcl}\mathcal{T}_{[3]}&=&\mu(2t_{k}-r-\mu)\|x^{k+1}-x^{k}\|^{2}+\eta^{2}t_{k}^{2}\|d^{k}\|^{2}-t_{k+1}^{2}\|p^{k}\|^{2}\\ &&+{~}2r[(\eta-\beta)t_{k}-\delta]\langle z^{k+1},x^{k+1}-x^{\star}\rangle\\ &&-{~}2r[(\eta-\beta)t_{k}-\eta(r-1)]\langle z^{k},x^{k}-x^{\star}\rangle\\ &&+{~}2\big[((\eta-\beta)t_{k}-\delta)(t_{k}-r-\mu)+\mu\eta t_{k}\big]\langle z^{k+1}-z^{k},x^{k+1}-x^{k}\rangle\\ &&-{~}2\big[((\eta-\beta)r+\delta)t_{k}-\eta(r-1)t_{k}-\delta(r+\mu)\big]\langle z^{k},x^{k+1}-x^{k}\rangle\\ &&+{~}2\mu\eta t_{k}\langle\hat{w}^{k+1}-z^{k+1},x^{k+1}-x^{k}\rangle.\end{array}\hskip-12.91663pt (156) Fifth, exploiting again y^{k}=x^{k+1}+\eta d^{k} and y^{k+1}-x^{k+1}=\theta_{k}(x^{k+1}-x^{k})-p^{k} from ( GAEG+ ), one can express \begin{array}[]{lcl}\mathcal{T}_{[6]}&=&2c_{k}\langle z^{k},y^{k}-x^{k}\rangle-2c_{k+1}\langle z^{k+1},y^{k+1}-x^{k+1}\rangle\\ &=&2c_{k}\langle z^{k},x^{k+1}-x^{k}+\eta d^{k}\rangle-2c_{k+1}\langle z^{k+1},\theta_{k}(x^{k+1}-x^{k})-p^{k}\rangle\\ &=&2c_{k}\langle z^{k},x^{k+1}-x^{k}\rangle-2c_{k+1}\theta_{k}\langle z^{k+1},x^{k+1}-x^{k}\rangle\\ &&+{~}2\eta c_{k}\langle z^{k},d^{k}\rangle+2c_{k+1}\langle z^{k+1},p^{k}\rangle.\end{array} Sixth, adding \mathcal{T}_{[6]} to \mathcal{T}_{[3]} in ( 156 ), and noticing that \mathcal{P}_{k}-\mathcal{P}_{k+1}=\mathcal{T}_{[3]}+\mathcal{T}_{[6]} , we further obtain \begin{array}[]{lcl}\mathcal{P}_{k}-\mathcal{P}_{k+1}&=&\eta^{2}t_{k}^{2}\|d^{k}\|^{2}-t_{k+1}^{2}\|p^{k}\|^{2}+2\eta c_{k}\langle z^{k},d^{k}\rangle+2c_{k+1}\langle z^{k+1},p^{k}\rangle\\ &&+{~}\mu(2t_{k}-r-\mu)\|x^{k+1}-x^{k}\|^{2}+2\mu\eta t_{k}\langle\hat{w}^{k+1}-z^{k+1},x^{k+1}-x^{k}\rangle\\ &&+{~}2r[(\eta-\beta)t_{k}-\delta]\langle z^{k+1},x^{k+1}-x^{\star}\rangle\\ &&-{~}2r[(\eta-\beta)t_{k}-\eta(r-1)]\langle z^{k},x^{k}-x^{\star}\rangle\\ &&+{~}2\big[\omega(t_{k}-\mu-r)+\mu\eta\big]t_{k}\langle z^{k+1}-z^{k},x^{k+1}-x^{k}\rangle.\end{array} Finally, substituting {\color[rgb]{0,0,0}\mathcal{E}_{k}} from ( 83 ) into the last expression, we get ( 84 ). \square F.2 The Proof of Lemma 11 Exploiting d^{k}:=\hat{w}^{k+1}-\gamma_{k}z^{k} and p^{k}:=\eta_{k}z^{k+1}-\lambda_{k}\hat{w}^{k+1}+\nu_{k}z^{k} from ( 76 ), we can expand \mathcal{E}_{k} in ( 83 ), grouping each term and using \lambda_{k}=\frac{\eta t_{k}}{t_{k+1}} yield \begin{array}[]{lcl}\mathcal{E}_{k}&\overset{\tiny\eqref{eq:NGEAG4NI_Pk_and_Ek}}{=}&\eta^{2}t_{k}^{2}\|d^{k}\|^{2}-t_{k+1}^{2}\|p^{k}\|^{2}+2\eta c_{k}\langle z^{k},d^{k}\rangle+2c_{k+1}\langle z^{k+1},p^{k}\rangle\\ &=&\eta_{k}\big(2c_{k+1}-t_{k+1}^{2}\eta_{k}\big)\|z^{k+1}\|^{2}+\big(\eta^{2}t_{k}^{2}\gamma_{k}^{2}-t_{k+1}^{2}\nu_{k}^{2}-2\eta c_{k}\gamma_{k}\big)\|z^{k}\|^{2}\\ &&+{~}2\lambda_{k}\big(t_{k+1}^{2}\eta_{k}-c_{k+1}\big)\langle z^{k+1},\hat{w}^{k+1}\rangle\\ &&-{~}2\nu_{k}\big(t_{k+1}^{2}\eta_{k}-c_{k+1}\big)\langle z^{k+1},z^{k}\rangle\\ &&-{~}2\big(\eta^{2}t_{k}^{2}\gamma_{k}-t_{k+1}^{2}\lambda_{k}\nu_{k}-\eta c_{k}\big)\langle\hat{w}^{k+1},z^{k}\rangle.\end{array} By ( 77 ) and ( 82 ), we can compute each coefficient of the last expression to get \begin{array}[]{lcl}\mathcal{E}_{k}&=&\eta^{2}t_{k}^{2}\|d^{k}\|^{2}-t_{k+1}^{2}\|p^{k}\|^{2}+2\eta c_{k}\langle z^{k},d^{k}\rangle+2c_{k+1}\langle z^{k+1},p^{k}\rangle\\ &=&\big[(1-\mu)\psi t_{k}-\delta\big][(\eta-\beta)t_{k}-\delta\big]\|z^{k+1}\|^{2}\\ &&+{~}\big\{\eta(t_{k}-r+1)\big[(\eta-2\psi)(t_{k}-r+1)+2(r-1)\beta\big]-\beta^{2}t_{k}^{2}\big\}\|z^{k}\|^{2}\\ &&+{~}2\eta\omega t_{k}^{2}\langle z^{k+1},\hat{w}^{k+1}\rangle-2\beta\omega t_{k}^{2}\langle z^{k+1},z^{k}\rangle\\ &&-{~}2\eta\omega t_{k}(t_{k}-r+1)\langle\hat{w}^{k+1},z^{k}\rangle.\end{array} This is exactly ( 85 ). \square F.3 The Proof of Lemma 12 First, by ( 75 ), ( GAEG+ ), and the L -Lipschitz continuity of F , we have \|w^{k+1}-\hat{w}^{k+1}\|=\|Fx^{k+1}-Fy^{k}\|\leq L\|x^{k+1}-y^{k}\|=L\eta\|d^{k}\| . Using this relation, Young’s inequality, for any c_{1}>0 , \phi>0 , and \hat{\phi}\geq 0 , we get \begin{array}[]{lcl}\bar{\mathcal{T}}_{[1]}&:=&\|z^{k+1}-\hat{w}^{k+1}\|^{2}+\phi\|z^{k+1}-\hat{w}^{k+1}\|^{2}+\hat{\phi}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &\leq&\big[(1+\phi)(1+c_{1})+\hat{\phi}\big]\|w^{k+1}-\hat{w}^{k+1}\|^{2}+\frac{(1+\phi)(1+c_{1})}{c_{1}}\|z^{k+1}-w^{k+1}\|^{2}\\ &\leq&\big[(1+\phi)(1+c_{1})+\hat{\phi}\big]L^{2}\eta^{2}\|\hat{w}^{k+1}-\gamma_{k}z^{k}\|^{2}+\frac{(1+\phi)(1+c_{1})}{c_{1}}\|z^{k+1}-w^{k+1}\|^{2}.\end{array} Let M^{2}:=\big[(1+\phi)(1+c_{1})+\hat{\phi}\big]L^{2} as stated. Expanding the first term on the left-hand side, and the first term on the right-hand side of \bar{\mathcal{T}}_{[1]} above, and then using \gamma_{k}:=\frac{t_{k}-r+1}{t_{k}} from ( 77 ), rearranging the result, we get \begin{array}[]{lcl}0&\geq&t_{k}^{2}\|z^{k+1}\|^{2}+(1-M^{2}\eta^{2})t_{k}^{2}\|\hat{w}^{k+1}\|^{2}-M^{2}\eta^{2}(t_{k}-r+1)^{2}\|z^{k}\|^{2}\\ &&-{~}2t_{k}^{2}\langle z^{k+1},\hat{w}^{k+1}\rangle+2M^{2}\eta^{2}t_{k}(t_{k}-r+1)\langle\hat{w}^{k+1},z^{k}\rangle+\hat{\phi}t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\phi t_{k}^{2}\|z^{k+1}-\hat{w}^{k+1}\|^{2}-\frac{(1+\phi)(1+c_{1})t_{k}^{2}}{c_{1}}\|z^{k+1}-w^{k+1}\|^{2}.\end{array} Multiplying this inequality by \eta\omega with \omega:=\frac{\mu(\eta-\beta)}{\mu+1} , and adding the result to \mathcal{E}_{k} in ( 85 ) of Lemma 11 , we can show that \begin{array}[]{lcl}\mathcal{E}_{k}&\geq&\big\{\big[(1-\mu)\psi t_{k}-\delta\big][(\eta-\beta)t_{k}-\delta]+\eta\omega t_{k}^{2}\big\}\|z^{k+1}\|^{2}\\ &&+{~}\big\{\eta(t_{k}\!-\!r\!+\!1)\big[\big(\eta-2\psi-\omega M^{2}\eta^{2}\big)(t_{k}\!-\!r\!+\!1)+2(r-1)\beta\big]-\beta^{2}t_{k}^{2}\big\}\|z^{k}\|^{2}\\ &&-{~}2\beta\omega t_{k}^{2}\langle z^{k+1},z^{k}\rangle+\eta\omega(1-M^{2}\eta^{2})t_{k}^{2}\|\hat{w}^{k+1}\|^{2}\\ &&-{~}2\eta\omega(1-M^{2}\eta^{2})t_{k}(t_{k}-r+1)\langle\hat{w}^{k+1},z^{k}\rangle+\eta\omega\hat{\phi}t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\eta\omega\phi t_{k}^{2}\|z^{k+1}-\hat{w}^{k+1}\|^{2}-\frac{\eta\omega(1+\phi)(1+c_{1})t_{k}^{2}}{c_{1}}\|z^{k+1}-w^{k+1}\|^{2}.\end{array} Since t_{k}\hat{w}^{k+1}-(t_{k}-r+1)z^{k}=t_{k}d^{k} due to ( 76 ), we can easily show that \begin{array}[]{ll}&2\langle z^{k+1},z^{k}\rangle=\|z^{k}\|^{2}+\|z^{k+1}\|^{2}-\|z^{k+1}-z^{k}\|^{2},\\ &t_{k}^{2}\|\hat{w}^{k+1}\|^{2}-2t_{k}(t_{k}-r+1)\langle\hat{w}^{k+1},z^{k}\rangle+(t_{k}-r+1)^{2}\|z^{k}\|^{2}=t_{k}^{2}\|d^{k}\|^{2}.\end{array} Utilizing these two identities into the last inequality, and rearranging the result, we can further lower bound \begin{array}[]{lcl}\mathcal{E}_{k}&\geq&\hat{\Lambda}_{k+1}\|z^{k+1}\|^{2}-\Lambda_{k}\|z^{k}\|^{2}+\beta\omega t_{k}^{2}\|z^{k+1}-z^{k}\|^{2}\\ &&+{~}\eta\omega(1-M^{2}\eta^{2})t_{k}^{2}\|d^{k}\|^{2}+\eta\omega\hat{\phi}t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\eta\omega\phi t_{k}^{2}\|z^{k+1}-\hat{w}^{k+1}\|^{2}-\frac{\eta\omega(1+\phi)(1+c_{1})t_{k}^{2}}{c_{1}}\|z^{k+1}-w^{k+1}\|^{2},\end{array} (157) where \hat{\Lambda}_{k+1} and \Lambda_{k} are respectively given by \left\{\begin{array}[]{lcl}\hat{\Lambda}_{k+1}&:=&\big[(1-\mu)\psi t_{k}-\delta\big][(\eta-\beta)t_{k}-\delta]+\omega(\eta-\beta)t_{k}^{2},\\ \Lambda_{k}&:=&2\eta(t_{k}-r+1)\big[\psi(t_{k}-r+1)-(r-1)\beta\big]-\beta^{2}t_{k}^{2}.\end{array}\right. If we simplify \Lambda_{k} , then we obtain it as in ( 87 ). Furthermore, we can easily prove that \hat{\Lambda}_{k+1}=\Lambda_{k+1}+S_{k} for S_{k} defined by ( 87 ). Finally, using \hat{\Lambda}_{k+1} , \Lambda_{k} , and S_{k} from ( 87 ), we obtain ( 86 ) from ( 157 ). \square F.4 The Proof of Lemma 13 First, since u^{k}:=Fx^{k} in ( GAEG+ ), \mathcal{P}_{k} and {\color[rgb]{0,0,0}\mathcal{E}_{k}} in ( 83 ) reduce to \begin{array}[]{lcl}\mathcal{P}_{k}&:=&\|r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})\|^{2}+r\mu\|x^{k}-x^{\star}\|^{2}+2c_{k}\langle w^{k},y^{k}-x^{k}\rangle,\\ {\color[rgb]{0,0,0}\mathcal{E}_{k}}&:=&\eta^{2}t_{k}^{2}\|d^{k}\|^{2}-t_{k+1}^{2}\|p^{k}\|^{2}+2\eta c_{k}\langle w^{k},d^{k}\rangle+2c_{k+1}\langle w^{k+1},p^{k}\rangle.\end{array} In this case, ( 84 ) becomes \begin{array}[]{lcl}\mathcal{P}_{k}-\mathcal{P}_{k+1}&=&\mathcal{E}_{k}+\mu(2t_{k}-r-\mu)\|x^{k+1}-x^{k}\|^{2}\\ &&+{~}2r[(\eta-\beta)t_{k}-\delta]\langle w^{k+1},x^{k+1}-x^{\star}\rangle\\ &&-{~}2r[(\eta-\beta)t_{k}-\eta(r-1)]\langle w^{k},x^{k}-x^{\star}\rangle\\ &&+{~}2t_{k}\big[\omega(t_{k}-\mu-r)+\mu\eta\big]\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle\\ &&+{~}2\mu\eta t_{k}\langle\hat{w}^{k+1}-w^{k+1},x^{k+1}-x^{k}\rangle.\end{array} (158) Next, since z^{k}=w^{k} , we can choose c_{1}:=0 and \phi:=0 and ( 86 ) reduces to \begin{array}[]{lcl}{\color[rgb]{0,0,0}\mathcal{E}_{k}}&\geq&\Lambda_{k+1}\|w^{k+1}\|^{2}-\Lambda_{k}\|w^{k}\|^{2}+S_{k}\|w^{k+1}\|^{2}+\beta\omega t_{k}^{2}\|w^{k+1}-w^{k}\|^{2}\\ &&+{~}\eta\omega(1-M^{2}\eta^{2})t_{k}^{2}\|d^{k}\|^{2}+\eta\omega\hat{\phi}t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2},\end{array} (159) where M^{2}:=\big(1+\hat{\phi})L^{2} for any \hat{\phi}\geq 0 . Now, substituting ( 159 ) into ( 158 ), and using \langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle\geq-\rho\|w^{k+1}-w^{k}\|^{2} from the \rho -co-hypomonotonicity of \Phi , we get \hskip-4.30554pt\begin{array}[]{lcl}\mathcal{P}_{k}-\mathcal{P}_{k+1}&\geq&\Lambda_{k+1}\|w^{k+1}\|^{2}-\Lambda_{k}\|w^{k}\|^{2}+\mu(2t_{k}-r-\mu)\|x^{k+1}-x^{k}\|^{2}\\ &&+{~}2r[(\eta-\beta)t_{k+1}-\eta(r-1)]\langle w^{k+1},x^{k+1}-x^{\star}\rangle\\ &&-{~}2r[(\eta-\beta)t_{k}-\eta(r-1)]\langle w^{k},x^{k}-x^{\star}\rangle\\ &&+{~}2r(r-2)\omega\langle w^{k+1},x^{k+1}-x^{\star}\rangle+S_{k}\|w^{k+1}\|^{2}\\ &&+{~}t_{k}\big\{\beta\omega t_{k}-2\rho\big[\omega(t_{k}-\mu-r)+\mu\eta\big]\big\}\|w^{k+1}-w^{k}\|^{2}\\ &&+{~}\eta\omega(1-M^{2}\eta^{2})t_{k}^{2}\|d^{k}\|^{2}+\eta\omega\hat{\phi}t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}2\mu\eta t_{k}\langle\hat{w}^{k+1}-w^{k+1},x^{k+1}-x^{k}\rangle.\end{array}\hskip-17.22217pt (160) Utilizing \mathcal{L}_{k} from ( 88 ), and Young’s inequality for the last term of ( 160 ), we can show from this expression that \begin{array}[]{lcl}\mathcal{L}_{k}-\mathcal{L}_{k+1}&\geq&\mu\big(2t_{k}-r-\mu-\frac{\mu\eta}{\hat{\phi}\omega}\big)\|x^{k+1}-x^{k}\|^{2}+\eta\omega(1-M^{2}\eta^{2})t_{k}^{2}\|d^{k}\|^{2}\\ &&+{~}2\omega r(r-2)\langle w^{k+1},x^{k+1}-x^{\star}\rangle+S_{k}\|w^{k+1}\|^{2}\\ &&+{~}t_{k}\big[\omega(\beta-2\rho)t_{k}+2\rho(\mu\omega+r\omega-\mu\eta)\big]\|w^{k+1}-w^{k}\|^{2}.\end{array} Substituting \langle w^{k+1},x^{k+1}-x^{\star}\rangle\geq-\rho\|w^{k+1}\|^{2} from the \rho -co-hypomonotonicity of \Phi into the last estimate, we obtain ( 89 ). \square F.5 The Proof of Lemma 14 From ( 88 ) and \langle w^{k},x^{k}-x^{\star}\rangle\geq-\rho\|w^{k}\|^{2} , we can show that \begin{array}[]{lcl}\mathcal{L}_{k}&:=&\|r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})\|^{2}+r\mu\|x^{k}-x^{\star}\|^{2}+\Lambda_{k}\|w^{k}\|^{2}\\ &&+{~}2\big[\psi(t_{k}-r+1)-(r-1)\beta\big]\langle w^{k},r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})\rangle\\ &&+{~}2\omega r(t_{k}-r+1)\langle w^{k},x^{k}-x^{\star}\rangle\\ &\geq&\|r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})+[\psi(t_{k}-r+1)-(r-1)\beta]w^{k}\|^{2}\\ &&+{~}\sigma_{k}\|w^{k}\|^{2}+r\mu\|x^{k}-x^{\star}\|^{2},\end{array} where \sigma_{k}:=\Lambda_{k}-[\psi(t_{k}-r+1)-(r-1)\beta]^{2}-2\rho\omega r(t_{k}-r+1) . Since \Lambda_{k} is given by ( 87 ), we can simplify \sigma_{k} as \sigma_{k}=\mu\big[\psi(t_{k}-r+1)-r\rho\big]^{2}-\mu\big[r^{2}\rho^{2}-\frac{(r-1)^{2}\beta(\eta-\beta)}{\mu+1}\big] . Substituting this value into the last estimate, we obtain ( 90 ). Finally, the proof of ( 91 ) directly follows from the definition of \mathcal{L}_{k} in ( 88 ), the Cauchy-Schwarz inequality and y^{0}=x^{0} . \square F.6 The Proof of Theorem 7.1 For simplicity of our presentation, we choose \mu:=1 . Since \mu=1 and \hat{\phi}:=\frac{1-L^{2}\eta^{2}}{2L^{2}\eta^{2}}>0 as in ( 93 ), we have 1-M^{2}\eta^{2}=\frac{1-L^{2}\eta^{2}}{2} . From this relation and the choice of t_{0} , we can derive from ( 89 ) that \begin{array}[]{lcl}\mathcal{L}_{k}-\mathcal{L}_{k+1}&\geq&\big(2t_{k}-r-1-\frac{\eta}{\hat{\phi}\omega}\big)\|x^{k+1}-x^{k}\|^{2}+\frac{\eta\omega(1-L^{2}\eta^{2})t_{k}^{2}}{2}\|d^{k}\|^{2}\\ &&+{~}t_{k}\big[\omega(\beta-2\rho)t_{k}+2\rho(\omega+r\omega-\eta)\big]\|w^{k+1}-w^{k}\|^{2}\\ &&+{~}\frac{(r-2)}{2}\big[(\eta-\beta)^{2}t_{k}-\frac{2\Gamma}{r-2}-4\rho\omega r\big]\|w^{k+1}\|^{2}.\end{array} Note that, since t_{k+1}=t_{k}+1 from ( 77 ), we get t_{k}=k+t_{0} . Summing up the last inequality from k:=0 to k:=K , then taking the limit as K\to\infty and noting that \mathcal{L}_{k}\geq 0 and t_{k}=k+t_{0} , we obtain ( 94 ). Here, the last summable estimate of ( 94 ) follows from the fact that \omega(\beta-2\rho)t_{k}+2\rho(\omega+r\omega-\eta)\geq\frac{\omega(\beta-2\rho)t_{k}}{2} due to the choice of t_{0} in ( 92 ). Finally, \sigma_{k} in Lemma 14 can be directly lower bounded by \begin{array}[]{lcl}\sigma_{k}&\geq&\frac{(\eta-\beta)^{2}(t_{k}-r+1)^{2}}{8}=\frac{(\eta-\beta)^{2}(k+t_{0}-r+1)^{2}}{8}.\end{array} Since \sigma_{k}\|w^{k}\|^{2}\leq\mathcal{L}_{k} from ( 90 ) and \mathcal{L}_{k}\leq\mathcal{L}_{0}\leq\mathcal{R}_{0}^{2} by induction and ( 91 ), combining these estimates, we obtain ( 95 ). \square F.7 The Proof of Theorem 7.2 From ( GAEG+ ), ( 76 ), and ( 77 ), since \mu=1 , we have \begin{array}[]{lcl}y^{k+1}-x^{k+1}+\eta_{k}w^{k+1}&=&\theta_{k}(y^{k}-x^{k}+\eta_{k-1}w^{k})-(\eta\theta_{k}-\lambda_{k})d^{k}\\ &&-{~}(\theta_{k}\eta_{k-1}+\nu_{k}-\lambda_{k}\gamma_{k})w^{k}\\ &=&\theta_{k}(y^{k}-x^{k}+\eta_{k-1}w^{k})+\frac{\eta(r+1)}{t_{k+1}}d^{k}+\frac{s_{k}}{t_{k+1}}w^{k},\end{array} where s_{k}:=\frac{(r+4)(\eta-\beta)t_{k}-2(r+1)(\eta-\beta+\delta)}{2t_{k}} . Let us denote v^{k}:=y^{k}-x^{k}+\eta_{k-1}w^{k} . Then, the last expression can be rewritten as follows: \begin{array}[]{lcl}v^{k+1}&=&\theta_{k}v^{k}+(1-\theta_{k})\big[\frac{\eta(r+1)}{(1-\theta_{k})t_{k+1}}d^{k}+\frac{s_{k}}{(1-\theta_{k})t_{k+1}}w^{k}\big].\end{array} Since \theta_{k}\in(0,1) , using convexity of \|\cdot\|^{2} , we have \begin{array}[]{lcl}\|v^{k+1}\|^{2}&\leq&\theta_{k}\|v^{k}\|^{2}+\frac{1}{1-\theta_{k}}\|\frac{\eta(r+1)}{t_{k+1}}d^{k}+\frac{s_{k}}{t_{k+1}}w^{k}\|^{2}\\ &\leq&\theta_{k}\|v^{k}\|^{2}+\frac{2\eta^{2}(r+1)^{2}}{(1-\theta_{k})t_{k+1}^{2}}\|d^{k}\|^{2}+\frac{2s_{k}^{2}}{(1-\theta_{k})t_{k+1}^{2}}\|w^{k}\|^{2}.\end{array} Noting that s_{k}\leq\bar{s}:=\frac{(r+4)(\eta-\beta)+1}{2} for any t_{k}\geq 0 . Multiplying both sides of the last expression by t_{k+1}^{2} and using this relation and \theta_{k}=\frac{t_{k}-r-1}{t_{k+1}} , we get \hskip-4.30554pt\begin{array}[]{lcl}t_{k+1}^{2}\|v^{k+1}\|^{2}&\leq&t_{k}^{2}\|v^{k}\|^{2}-(rt_{k}+r+1)\|v^{k}\|^{2}+\frac{2\eta^{2}(r+1)^{2}t_{k+1}}{(r+2)}\|d^{k}\|^{2}\\ &&+{~}\frac{2\bar{s}^{2}t_{k+1}}{r+2}\|w^{k}\|^{2}.\end{array}\hskip-8.61108pt (161) Since \sum_{k=0}^{\infty}\frac{2\eta^{2}(r+1)^{2}t_{k+1}}{(r+2)}\|d^{k}\|^{2}<+\infty and \sum_{k=0}^{\infty}\frac{2\bar{s}^{2}t_{k+1}}{r+2}\|w^{k}\|^{2}<+\infty due to ( 94 ), the limit \lim_{k\to\infty}t_{k}^{2}\|v^{k}\|^{2} exists. Moreover, since \sum_{k=0}^{\infty}[(r-2)t_{k}+r-1]\|v^{k}\|^{2}<+\infty , we can show that \lim_{k\to\infty}t_{k}^{2}\|y^{k}-x^{k}-\eta_{k-1}w^{k}\|^{2}=0 . Next, using again ( GAEG+ ), ( 76 ), and ( 77 ) with \mu=1 , we can also derive \begin{array}[]{lcl}y^{k+1}-x^{k+1}&=&\theta_{k}(y^{k}-x^{k})-p^{k}-\frac{\eta(t_{k}-r-1)}{t_{k+1}}d^{k}\\ &=&\theta_{k}(y^{k}-x^{k})+\frac{r+1}{t_{k+1}}d^{k}-\frac{r(\eta-\beta)}{2t_{k+1}}w^{k+1}\\[4.30554pt] &&-{~}\frac{[(\eta-\beta)t_{k}-(r-1)\eta]}{t_{k+1}}(w^{k+1}-w^{k})\\ &=&\theta_{k}(y^{k}-x^{k})+\frac{1}{t_{k+1}}\zeta_{k}-\frac{s_{k}}{t_{k+1}}(w^{k+1}-w^{k}),\end{array} (162) where \zeta^{k}:=(r+1)d^{k}-\frac{r(\eta-\beta)}{2}w^{k+1} and s_{k}:=(\eta-\beta)t_{k}-(r-1)\eta . Utilizing y^{k}-x^{k}=x^{k+1}-x^{k}+\eta d^{k} , the last expression leads to \begin{array}[]{lcl}\|y^{k+1}-x^{k+1}\|^{2}&=&\theta_{k}^{2}\|y^{k}-x^{k}\|^{2}+\frac{2\theta_{k}}{t_{k+1}}\langle\zeta^{k}-s_{k}(w^{k+1}-w^{k}),y^{k}-x^{k}\rangle\\ &&+{~}\frac{1}{t_{k+1}^{2}}\|\zeta^{k}-s_{k}(w^{k+1}-w^{k})\|^{2}\\ &=&\theta_{k}^{2}\|y^{k}-x^{k}\|^{2}+\frac{2\theta_{k}}{t_{k+1}}\langle\zeta^{k},y^{k}-x^{k}\rangle\\ &&-{~}\frac{2\theta_{k}s_{k}}{t_{k+1}}\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle-\frac{2\eta\theta_{k}s_{k}}{t_{k+1}}\langle w^{k+1}-w^{k},d^{k}\rangle\\ &&+{~}\frac{1}{t_{k+1}^{2}}\|\zeta^{k}-s_{k}(w^{k+1}-w^{k})\|^{2}.\end{array} By \langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle\geq-\rho\|w^{k+1}-w^{k}\|^{2} from the \rho -co-hypomonotonicity of \Phi and Young’s inequality, we can prove that \hskip-8.61108pt\begin{array}[]{lcl}\|y^{k+1}-x^{k+1}\|^{2}&\leq&\big(\theta_{k}^{2}+\frac{\theta_{k}}{t_{k+1}}\big)\|y^{k}-x^{k}\|^{2}+\frac{1}{t_{k+1}^{2}}\big(2+t_{k+1}\theta_{k}\big)\|\zeta^{k}\|^{2}\\ &&+{~}\frac{(\eta+2\rho)t_{k+1}s_{k}\theta_{k}+2s_{k}^{2}}{t_{k+1}^{2}}\|w^{k+1}-w^{k}\|^{2}+\frac{\eta\theta_{k}s_{k}}{t_{k+1}}\|d^{k}\|^{2}.\end{array}\hskip-17.22217pt (163) We can easily check that \theta_{k}^{2}+\frac{\theta_{k}}{t_{k+1}}=\frac{t_{k}^{2}-(2r+1)t_{k}+r(r+1)}{t_{k+1}^{2}} , and \begin{array}[]{ccl}(\eta+2\rho)t_{k+1}s_{k}\theta_{k}+2s_{k}^{2}&\leq&(3\eta-2\beta+2\rho)(\eta-\beta)t_{k}^{2},\\[8.61108pt] \eta\theta_{k}s_{k}&\leq&\eta(\eta-\beta)t_{k}^{2},\end{array} where the first inequality holds if t_{k}\geq\frac{(r^{2}-1)\eta+2(r-1)^{2}\eta^{2}}{(\eta+2\rho)[(\eta-\beta)(r+1)+(r-1)\eta]+4(r-1)\eta(\eta-\beta)} and the second one requires t_{k}\geq\frac{(r^{2}-1)\eta}{(r-1)\eta+(r+1)(\eta-\beta)} , which holds as t_{k}\to\infty . Multiplying ( 163 ) by t_{k+1}^{2} and using these bounds, we further obtain \hskip-8.61108pt\begin{array}[]{lcl}t_{k+1}^{2}\|y^{k+1}-x^{k+1}\|^{2}&\leq&t_{k}^{2}\|y^{k}-x^{k}\|^{2}-[(2r+1)t_{k}-r(r+1)]\|y^{k}-x^{k}\|^{2}\\ &&+{~}(t_{k}-r+1)\|\zeta^{k}\|^{2}+\eta(\eta-\beta)t_{k}^{2}\|d^{k}\|^{2}\\ &&+{~}(3\eta-2\beta+2\rho)(\eta-\beta)t_{k}^{2}\|w^{k+1}-w^{k}\|^{2}.\end{array}\hskip-17.22217pt (164) By Young’s inequality again, one has \begin{array}[]{lcl}\|\zeta^{k}\|^{2}&\leq&2(r+1)^{2}\|d^{k}\|^{2}+\frac{r^{2}(\eta-\beta)^{2}}{2}\|w^{k+1}\|^{2}.\end{array} Substituting this inequality into ( 164 ), we get \hskip-8.61108pt\begin{array}[]{lcl}t_{k+1}^{2}\|y^{k+1}-x^{k+1}\|^{2}&\leq&t_{k}^{2}\|y^{k}-x^{k}\|^{2}-[(2r+1)t_{k}-r(r+1)]\|y^{k}-x^{k}\|^{2}\\ &&+{~}\big[\eta(\eta-\beta)t_{k}^{2}+2(r+1)^{2}(t_{k}-r+1)\big]\|d^{k}\|^{2}\\ &&+{~}(3\eta-2\beta+2\rho)(\eta-\beta)t_{k}^{2}\|w^{k+1}-w^{k}\|^{2}\\ &&+{~}\frac{r^{2}(\eta-\beta)^{2}}{2}(t_{k}-r+1)\|w^{k+1}\|^{2}.\end{array}\hskip-17.22217pt Note that the last three terms are summable due to ( 94 ). This implies that \lim_{k\to\infty}t_{k}^{2}\|y^{k}-x^{k}\|^{2}=0 as before, which proves the last limit in ( 96 ). Now, utilizing Young’s inequality, one can show that \begin{array}[]{lcl}[(\eta-\beta)t_{k-1}-\delta]^{2}\|w^{k}\|^{2}&=&t_{k}^{2}\eta_{k-1}^{2}\|w^{k}\|^{2}\\ &\leq&2t_{k}^{2}\|y^{k}-x^{k}+\eta_{k-1}w^{k}\|^{2}+2t_{k}^{2}\|y^{k}-x^{k}\|^{2}.\end{array} Combining this inequality, \lim_{k\to\infty}t_{k}^{2}\|y^{k}-x^{k}-\eta_{k-1}w^{k}\|^{2}=0 , and \lim_{k\to\infty}t_{k}^{2}\|y^{k}-x^{k}\|^{2}=0 , we get \lim_{k\to\infty}[(\eta-\beta)t_{k-1}-\delta]^{2}\|w^{k}\|^{2}=0 , which also implies \lim_{k\to\infty}t_{k}^{2}\|w^{k}\|^{2}=0 . This proves the first line of ( 96 ). Finally, using Young’s inequality one more time, we get \begin{array}[]{lcl}t_{k}^{2}\|x^{k+1}-x^{k}\|^{2}=t_{k}^{2}\|y^{k}-x^{k}-\eta d^{k}\|^{2}\leq 2t_{k}^{2}\|y^{k}-x^{k}\|^{2}+2\eta^{2}t_{k}^{2}\|d^{k}\|^{2}.\end{array} Since \lim_{k\to\infty}t_{k}^{2}\|y^{k}-x^{k}\|^{2}=0 and \lim_{k\to\infty}t_{k}^{2}\|d^{k}\|^{2}=0 , this inequality implies \lim_{k\to\infty}t_{k}^{2}\|x^{k+1}-x^{k}\|^{2}=0 , which leads to the second line of ( 96 ). \square F.8 The Proof of Theorem 7.3 From the proof of Theorem 7.1 , we have r\|x^{k}-x^{\star}\|^{2}\leq\mathcal{L}_{k}\leq\mathcal{L}_{0}\leq\mathcal{R}_{0}^{2} , we conclude that \{x^{k}\} is bounded, and hence, it has a cluster point. Let x^{*} be a cluster point of \{x^{k}\} and \{x^{k_{i}}\} be a subsequence converging to x^{*} . Consider the subsequence \{z^{k_{i}}\} with z^{k_{i}}:=Fx^{k_{i}}+\xi^{k_{i}} for \xi^{k_{i}}\in Tx^{k_{i}} . By ( 96 ), we have \lim_{i\to\infty}\|Fx^{k_{i}}+\xi^{k_{i}}\|^{2}=\lim_{i\to\infty}\|z^{k_{i}}\|^{2}=0 , implies that z^{k_{i}}\to 0 . Since \mathrm{gra}(T) is closed and F is L -Lipschitz continuous, \mathrm{gra}(\Phi)=\mathrm{gra}(F+T) is also closed and (x^{k_{i}},z^{k_{i}})\in\mathrm{gra}(\Phi) converges to (x^{\star},0) . By the closedness of \mathrm{gra}(\Phi) , we have (x^{*},0)\in\mathrm{gra}(\Phi) , which means that 0\in\Phi{x^{*}}=Fx^{*}+Tx^{*} . Next, we prove that \lim_{k\to\infty}\|x^{k}-x^{\star}\|^{2} exists. By the \rho -co-hypomonotonicity of \Phi , the Cauchy-Schwarz inequality, and r\|x^{k}-x^{\star}\|^{2}\leq\mathcal{R}_{0}^{2} , we have \begin{array}[]{lcl}-\rho t_{k}\|w^{k}\|^{2}\leq t_{k}\langle w^{k},x^{k}-x^{\star}\rangle\leq t_{k}\|w^{k}\|\|x^{k}-x^{\star}\|\leq\frac{\mathcal{R}_{0}}{\sqrt{r}}t_{k}\|w^{k}\|.\end{array} Note that \lim_{k\to\infty}t_{k}\|w^{k}\|=0 due to the first line of ( 96 ), and \lim_{k\to\infty}t_{k}\|w^{k}\|^{2}=0 due to the third summable expression in ( 94 ), we conclude that \begin{array}[]{lcl}\lim_{k\to\infty}|t_{k}\langle w^{k},x^{k}-x^{\star}\rangle|=0.\end{array} (165) From ( 76 ) and ( 77 ), with z^{k}=w^{k} , we have \begin{array}[]{lcl}w^{k+1}-\hat{w}^{k+1}&=&w^{k+1}-w^{k}-d^{k}+\frac{r-1}{t_{k}}w^{k},\\ t_{k+1}p^{k}&=&[(\eta-\beta)t_{k}-\delta](w^{k+1}-w^{k})-\eta t_{k}d^{k}+[\eta(r-1)-\delta]w^{k}.\end{array} Applying Young’s inequality, these relations lead to \begin{array}[]{lclcl}0&\leq&t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}&\leq&3t_{k}^{2}\|w^{k+1}-w^{k}\|^{2}+3t_{k}^{2}\|d^{k}\|^{2}+3(r-1)^{2}t_{k}\|w^{k}\|^{2},\\ 0&\leq&t_{k+1}^{2}\|p^{k}\|^{2}&\leq&3[(\eta-\beta)t_{k}-\delta]^{2}\|w^{k+1}-w^{k}\|^{2}+3\eta^{2}t_{k}^{2}\|d^{k}\|^{2}\\ &&&&+{~}3[\eta(r-1)-\delta]^{2}\|w^{k}\|^{2}.\end{array} Each term on the right-hand side of each inequality is summable, we obtain \begin{array}[]{lcl}\sum_{k=0}^{+\infty}t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}<+\infty,\\ \sum_{k=0}^{+\infty}t_{k+1}^{2}\|p^{k}\|^{2}<+\infty.\end{array} (166) Let us define \begin{array}[]{lcl}\mathcal{Q}_{k}&:=&\|r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})\|^{2}+r\|x^{k}-x^{\star}\|^{2}\\ &&+{~}2r[(\eta-\beta)t_{k}-\eta(r-1)]\big[\langle w^{k},x^{k}-x^{\star}\rangle+\rho\|w^{k}\|^{2}].\end{array} (167) Then, similar to the proof of ( 156 ) in Lemma 10 , with \mu=1 , we can show that \begin{array}[]{lcl}\mathcal{Q}_{k}-\mathcal{Q}_{k+1}&=&(2t_{k}-r-1)\|x^{k+1}-x^{k}\|^{2}+\eta^{2}t_{k}^{2}\|d^{k}\|^{2}-t_{k+1}^{2}\|p^{k}\|^{2}\\ &&+{~}2r[\eta(r-2)-\delta]\langle w^{k+1},x^{k+1}-x^{\star}\rangle\\ &&+{~}2\big[((\eta-\beta)t_{k}-\delta)(t_{k}-r-1)+\eta t_{k}\big]\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle\\ &&-{~}2\big[((\eta-\beta)r+\delta)t_{k}-\eta(r-1)t_{k}-\delta(r+1)\big]\langle w^{k},x^{k+1}-x^{k}\rangle\\ &&+{~}2\eta t_{k}\langle\hat{w}^{k+1}-w^{k+1},x^{k+1}-x^{k}\rangle\\ &&+{~}2\rho r[(\eta-\beta)t_{k}-\eta(r-1)]\|w^{k}\|^{2}\\ &&-{~}2\rho r[(\eta-\beta)t_{k+1}-\eta(r-1)]\|w^{k+1}\|^{2}.\end{array} Next, by the \rho -co-hypomonotonicity of \Phi and Young’s inequality, we have \begin{array}[]{lcl}\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle\geq-\rho\|w^{k+1}-w^{k}\|^{2},\\ 2\eta t_{k}|\langle\hat{w}^{k+1}-w^{k+1},x^{k+1}-x^{k}\rangle|\geq-\eta t_{k}^{2}\|\hat{w}^{k+1}-w^{k+1}\|^{2}-\eta\|x^{k+1}-x^{k}\|^{2}.\end{array} Substituting these inequalities into \mathcal{Q}_{k}-\mathcal{Q}_{k+1} above, one can derive that \begin{array}[]{lcl}\mathcal{Q}_{k+1}-\mathcal{Q}_{k}&\leq&t_{k+1}^{2}\|p^{k}\|^{2}-\eta^{2}t_{k}^{2}\|d^{k}\|^{2}\\ &&+{~}2r[\eta(r-2)-\delta]|\langle w^{k+1},x^{k+1}-x^{\star}\rangle|\\ &&+{~}2\rho\big[((\eta-\beta)t_{k}-\delta)(t_{k}-r-1)+\eta t_{k}\big]\|w^{k+1}-w^{k}\|^{2}\\ &&+{~}\big[(\eta-\beta r+\delta)t_{k}-\delta(r+1)\big]\|w^{k}\|^{2}\\ &&+{~}\big[(\eta-\beta r+\delta-2)t_{k}-(\delta-1)(r+1)\big]\|x^{k+1}-x^{k}\|^{2}\\ &&+{~}2\rho r[(\eta-\beta)t_{k+1}-\eta(r-1)]\|w^{k+1}\|^{2}\\ &&+{~}\eta t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}+\eta\|x^{k+1}-x^{k}\|^{2}.\end{array} From ( 94 ), ( 165 ), and ( 166 ), we can see that all the terms on the right-hand side of this inequality are summable and \mathcal{Q}_{k}\geq 0 . We conclude that \lim_{k\to\infty}\mathcal{Q}_{k} exists due to ( 9 , Proposition 5.31) . By ( 167 ), the existence of \lim_{k\to\infty}\mathcal{Q}_{k} , and ( 165 ), we conclude that \begin{array}[]{lcl}\lim_{k\to\infty}\big[\|r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})\|^{2}+r\|x^{k}-x^{\star}\|^{2}\big]\ \text{exists}.\end{array} (168) Now, since \|x^{k}-x^{\star}\|\leq\frac{\mathcal{R}_{0}}{\sqrt{r}} and \lim_{k\to\infty}t_{k}\|y^{k}-x^{k}\|=0 due to ( 96 ), we get \begin{array}[]{lcl}|t_{k}\langle x^{k}-x^{\star},y^{k}-x^{k}\rangle|\leq t_{k}\|x^{k}-x^{\star}\|\|y^{k}-x^{k}\|\leq\frac{\mathcal{R}_{0}}{\sqrt{r}}t_{k}\|y^{k}-x^{k}\|\to 0,~k\to\infty.\end{array} As a consequence, we conclude that \lim_{k\to\infty}t_{k}\langle x^{k}-x^{\star},y^{k}-x^{k}\rangle=0 . Now, we consider \begin{array}[]{lcl}\bar{\mathcal{T}}_{[2]}&:=&\|r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})\|^{2}+r\|x^{k}-x^{\star}\|^{2}\\ &=&(r^{2}+r)\|x^{k}-x^{\star}\|^{2}+2rt_{k}\langle x^{k}-x^{\star},y^{k}-x^{k}\rangle+t_{k}^{2}\|y^{k}-x^{k}\|^{2}.\end{array} The limit on the left-hand side exists due to ( 168 ), while the limits of the last two terms t_{k}\langle x^{k}-x^{\star},y^{k}-x^{k}\rangle and t_{k}^{2}\|y^{k}-x^{k}\|^{2} on the right hand side are zero, we conclude that \lim_{k\to\infty}\|x^{k}-x^{\star}\|^{2} exists. Finally, since any cluster point of \{x^{k}\} is in \mathrm{zer}(\Phi) , we conclude that \{x^{k}\} converges to x^{\star}\in\mathrm{zer}(\Phi) . Combining the convergence of \{x^{k}\} and \lim_{k\to\infty}\|x^{k}-y^{k}\|=0 , we can say that \{y^{k}\} also converges to x^{\star}\in\mathrm{zer}(\Phi) due to the well-known Opial’s lemma. \square F.9 The Proof of Lemma 15 Denote \hat{a}_{k}:=\omega(t_{k}+1-r-\mu)+\mu\eta . First, since \omega=\frac{\mu(\eta-\beta)}{\mu+1} and \eta\geq\frac{(r+\mu-1)\beta}{r-2} , we have \hat{a}_{k}\leq\omega t_{k} . Similarly, since r>2 , we can show that \eta\geq\frac{(r+\mu-1)\beta}{r-2}\geq\frac{(r+\mu)\beta}{r-1} . Thus we also get a_{k}\leq\omega t_{k}^{2} . Next, utilizing ( GAEG+ ), we can express \mathcal{F}_{k} from ( 97 ) as \begin{array}[]{lcl}\mathcal{F}_{k}&=&2a_{k}\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle+2a_{k}\langle z^{k+1}-w^{k+1},x^{k+1}-x^{k}\rangle\\ &&-{~}2a_{k}\langle z^{k}-w^{k},x^{k+1}-x^{k}\rangle\\ &=&2a_{k}\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle+2a_{k+1}\langle z^{k+1}-w^{k+1},\theta_{k}(x^{k+1}-x^{k})\rangle\\ &&+{~}2(a_{k}-a_{k+1}\theta_{k})\langle z^{k+1}-w^{k+1},x^{k+1}-x^{k}\rangle\\ &&-{~}2a_{k}\langle z^{k}-w^{k},y^{k}-x^{k}-\eta d^{k}\rangle\\ &=&2a_{k}\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle+2a_{k+1}\langle z^{k+1}-w^{k+1},y^{k+1}-x^{k+1}\rangle\\ &&-{~}2a_{k}\langle z^{k}-w^{k},y^{k}-x^{k}\rangle+2(a_{k}-a_{k+1}\theta_{k})\langle z^{k+1}-w^{k+1},x^{k+1}-x^{k}\rangle\\ &&+{~}2a_{k+1}\langle z^{k+1}-w^{k+1},p^{k}\rangle+2\eta a_{k}\langle z^{k}-w^{k},d^{k}\rangle.\end{array} By the definition of p^{k} from ( 76 ), we have \begin{array}[]{lcl}t_{k+1}p^{k}&=&t_{k+1}\eta_{k}z^{k+1}-t_{k+1}\lambda_{k}\hat{w}^{k+1}+t_{k+1}\nu_{k}z^{k}\\ &=&[(\eta-\beta)t_{k}-\delta]z^{k+1}-\eta t_{k}\hat{w}^{k+1}+\beta t_{k}z^{k}\\ &=&\eta t_{k}(z^{k+1}-\hat{w}^{k+1})-\beta t_{k}(z^{k+1}-z^{k})-\delta z^{k+1}.\end{array} Therefore, applying Young’s inequality, for any \Delta>0 , we can derive that \begin{array}[]{lcl}\hat{\mathcal{T}}_{[1]}&:=&a_{k+1}\langle z^{k+1}-w^{k+1},p^{k}\rangle=\hat{a}_{k}\langle z^{k+1}-w^{k+1},t_{k+1}p^{k}\rangle\\ &=&\hat{a}_{k}\langle\eta t_{k}(z^{k+1}-\hat{w}^{k+1})-\beta t_{k}(z^{k+1}-z^{k})-\delta z^{k+1},z^{k+1}-w^{k+1}\rangle\\ &\geq&-\hat{a}_{k}[(\eta+\beta)t_{k}+\delta]\|z^{k+1}-w^{k+1}\|^{2}-\frac{\eta\hat{a}_{k}t_{k}}{4}\|z^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&-{~}\frac{\beta\hat{a}_{k}t_{k}}{4}\|z^{k+1}-z^{k}\|^{2}-\frac{\delta\hat{a}_{k}}{4}\|z^{k+1}\|^{2},\\ \hat{\mathcal{T}}_{[2]}&:=&\eta a_{k}\langle z^{k}-w^{k},d^{k}\rangle\geq-\frac{\Delta\eta ra_{k}}{2}\|z^{k}-w^{k}\|^{2}-\frac{\eta a_{k}}{2\Delta r}\|d^{k}\|^{2}.\end{array} Substituting these two inequalities of \hat{\mathcal{T}}_{[1]} and \hat{\mathcal{T}}_{[2]} into \mathcal{F}_{k} above, we get \begin{array}[]{lcl}\mathcal{F}_{k}&\geq&2a_{k+1}\langle z^{k+1}-w^{k+1},y^{k+1}-x^{k+1}\rangle-2a_{k}\langle z^{k}-w^{k},y^{k}-x^{k}\rangle\\ &&-{~}2\hat{a}_{k}[(\eta+\beta)t_{k}+\delta]\|z^{k+1}-w^{k+1}\|^{2}-\frac{\eta\hat{a}_{k}t_{k}}{2}\|z^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&-{~}\frac{\beta\hat{a}_{k}t_{k}}{2}\|z^{k+1}-z^{k}\|^{2}-\frac{\delta\hat{a}_{k}}{2}\|z^{k+1}\|^{2}-\frac{\eta a_{k}}{\Delta r}\|d^{k}\|^{2}\\ &&-{~}\Delta\eta ra_{k}\|z^{k}-w^{k}\|^{2}+2a_{k}\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle\\ &&+{~}2(a_{k}-a_{k+1}\theta_{k})\langle z^{k+1}-w^{k+1},x^{k+1}-x^{k}\rangle.\end{array} Finally, substituting b_{k}:=a_{k}-a_{k+1}\theta_{k}=\omega(r+\mu-1)(t_{k}-r-\mu)+\mu\eta(r+\mu) , \hat{a}_{k}\leq\omega t_{k} , and a_{k}\leq\omega t_{k}^{2} into the last inequality, we can further lower bound it and obtain ( 98 ). \square F.10 The Proof of Lemma 16 First, substituting \mathcal{F}_{k} from ( 97 ) and {\color[rgb]{0,0,0}\mathcal{E}_{k}} from ( 86 ) into ( 84 ), and choosing c_{1}:=1 in {\color[rgb]{0,0,0}\mathcal{E}_{k}} , we get \begin{array}[]{lcl}\mathcal{P}_{k}-\mathcal{P}_{k+1}&\geq&\big(S_{k}-\frac{\delta\omega t_{k}}{2}\big)\|z^{k+1}\|^{2}+\mu(2t_{k}-r-\mu)\|x^{k+1}-x^{k}\|^{2}\\ &&+{~}\Lambda_{k+1}\|z^{k+1}\|^{2}-\Lambda_{k}\|z^{k}\|^{2}+\frac{\beta\omega t_{k}^{2}}{2}\|z^{k+1}-z^{k}\|^{2}\\ &&+{~}\eta\omega\big(1-\frac{1}{\Delta r}-M^{2}\eta^{2}\big)t_{k}^{2}\|d^{k}\|^{2}+\eta\omega\hat{\phi}t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\eta\omega\big(\phi-\frac{1}{2}\big)t_{k}^{2}\|z^{k+1}-\hat{w}^{k+1}\|^{2}-\Delta r\eta\omega t_{k}^{2}\|z^{k}-w^{k}\|^{2}\\ &&-{~}2\omega t_{k}\big[(\eta(2+\phi)+\beta)t_{k}+\delta\big]\|z^{k+1}-w^{k+1}\|^{2}\\ &&+{~}2r[(\eta-\beta)t_{k+1}-\eta(r-1)]\langle z^{k+1},x^{k+1}-x^{\star}\rangle\\ &&-{~}2r[(\eta-\beta)t_{k}-\eta(r-1)]\langle z^{k},x^{k}-x^{\star}\rangle\\ &&+{~}2a_{k+1}\langle z^{k+1}-w^{k+1},y^{k+1}-x^{k+1}\rangle-2a_{k}\langle z^{k}-w^{k},y^{k}-x^{k}\rangle\\ &&+{~}2b_{k}\langle z^{k+1}-w^{k+1},x^{k+1}-x^{k}\rangle+2\mu\eta t_{k}\langle\hat{w}^{k+1}-z^{k+1},x^{k+1}-x^{k}\rangle\\ &&+{~}2a_{k}\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle+2\omega r(r-2)\langle z^{k+1},x^{k+1}-x^{\star}\rangle.\end{array} Next, applying Young’s inequality, we have \begin{array}[]{lcl}2\mu\eta t_{k}\langle\hat{w}^{k+1}-z^{k+1},x^{k+1}-x^{k}\rangle&\geq&-\frac{2\mu^{2}\eta}{\omega}\|x^{k+1}-x^{k}\|^{2}-\frac{\eta\omega t_{k}^{2}}{2}\|z^{k+1}-\hat{w}^{k+1}\|^{2}.\end{array} Since \eta\geq\frac{(r+\mu-1)\beta}{r-2} , we have b_{k}\leq(r+\mu-1)\omega t_{k} from Lemma 15 . Applying Young’s inequality again, and using b_{k}\leq(r+\mu-1)\omega t_{k} , we can show that \begin{array}[]{lcl}\hat{\mathcal{T}}_{[3]}&:=&2b_{k}\langle z^{k+1}-w^{k+1},x^{k+1}-x^{k}\rangle\\ &\geq&-\frac{b_{k}}{\omega(r+\mu-1)}\|x^{k+1}-x^{k}\|^{2}-\omega(r+\mu-1)b_{k}\|z^{k+1}-w^{k+1}\|^{2}\\ &\geq&t_{k}\|x^{k+1}-x^{k}\|^{2}-\omega^{2}(r+\mu-1)^{2}t_{k}\|z^{k+1}-w^{k+1}\|^{2}.\end{array} Substituting the last two inequalities into the first estimate, we obtain \begin{array}[]{lcl}\mathcal{P}_{k}-\mathcal{P}_{k+1}&\geq&\big(S_{k}-\frac{\delta\omega t_{k}}{2}\big)\|z^{k+1}\|^{2}+\mu\big(t_{k}-r-\mu-\frac{2\mu\eta}{\omega}\big)\|x^{k+1}-x^{k}\|^{2}\\ &&+{~}\Lambda_{k+1}\|z^{k+1}\|^{2}-\Lambda_{k}\|z^{k}\|^{2}+\frac{\beta\omega t_{k}^{2}}{2}\|z^{k+1}-z^{k}\|^{2}\\ &&+{~}\eta\omega\big(1-\frac{1}{\Delta r}-M^{2}\eta^{2}\big)t_{k}^{2}\|d^{k}\|^{2}+\eta\omega\hat{\phi}t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\eta\omega(\phi-1)t_{k}^{2}\|z^{k+1}-\hat{w}^{k+1}\|^{2}-\Delta r\eta\omega t_{k}^{2}\|z^{k}-w^{k}\|^{2}\\ &&-{~}\omega t_{k}\big[2(\eta(2+\phi)+\beta)t_{k}+2\delta+\omega(r+\mu-1)^{2}\big]\|z^{k+1}-w^{k+1}\|^{2}\\ &&+{~}2r[(\eta-\beta)t_{k+1}-\eta(r-1)]\langle z^{k+1},x^{k+1}-x^{\star}\rangle\\ &&-{~}2r[(\eta-\beta)t_{k}-\eta(r-1)]\langle z^{k},x^{k}-x^{\star}\rangle\\ &&+{~}2a_{k+1}\langle z^{k+1}-w^{k+1},y^{k+1}-x^{k+1}\rangle-2a_{k}\langle z^{k}-w^{k},y^{k}-x^{k}\rangle\\ &&+{~}2\omega r(r-2)\langle z^{k+1},x^{k+1}-x^{\star}\rangle+2a_{k}\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle.\end{array} Utilizing \mathcal{G}_{k} from ( 99 ), the last inequality leads to \hskip-12.91663pt\begin{array}[]{lcl}\mathcal{G}_{k}-\mathcal{G}_{k+1}&\geq&\big(S_{k}-\frac{\delta\omega t_{k}}{2}\big)\|z^{k+1}\|^{2}+\mu\big(t_{k}-r-\mu-\frac{2\mu\eta}{\omega}\big)\|x^{k+1}-x^{k}\|^{2}\\ &&+{~}\eta\omega\big(1-\frac{1}{\Delta r}-M^{2}\eta^{2}\big)t_{k}^{2}\|d^{k}\|^{2}+\eta\omega\hat{\phi}t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\eta\omega(\phi-1)t_{k}^{2}\|z^{k+1}-\hat{w}^{k+1}\|^{2}-\Delta r\eta\omega t_{k}^{2}\|z^{k}-w^{k}\|^{2}\\ &&+{~}2\omega r(r-2)\langle z^{k+1},x^{k+1}-x^{\star}\rangle+2a_{k}\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle\\ &&-{~}\omega t_{k}\big[2(\eta(2+\phi)+\beta)t_{k}+2\delta+\omega(r+\mu-1)^{2}\big]\|z^{k+1}-w^{k+1}\|^{2}\\ &&+{~}\frac{\beta\omega t_{k}^{2}}{2}\|z^{k+1}-z^{k}\|^{2}.\end{array}\hskip-25.83325pt (169) Now, applying again Young’s inequality, the \rho -co-hypomonotonicity of \Phi , and a_{k}\leq\omega t_{k}^{2} , we can prove that \begin{array}[]{lcl}\hat{\mathcal{T}}_{[4]}&:=&2a_{k}\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle\\ &\geq&2a_{k}\big[\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle+\rho\|w^{k+1}-w^{k}\|^{2}\big]\\ &&-{~}4\rho a_{k}\|z^{k+1}-z^{k}\|^{2}-8\rho a_{k}\|z^{k+1}-w^{k+1}\|^{2}-8\rho a_{k}\|z^{k}-w^{k}\|^{2}\\ &\geq&-4\rho\omega t_{k}^{2}\|z^{k+1}-z^{k}\|^{2}-8\rho\omega t_{k}^{2}\|z^{k+1}-w^{k+1}\|^{2}-8\rho\omega t_{k}^{2}\|z^{k}-w^{k}\|^{2},\\ \hat{\mathcal{T}}_{[5]}&:=&2\omega r(r-2)\langle z^{k+1},x^{k+1}-x^{\star}\rangle\\ &=&2\omega r(r-2)\langle w^{k+1},x^{k+1}-x^{\star}\rangle+2\omega r(r-2)\langle z^{k+1}-w^{k+1},x^{k+1}-x^{\star}\rangle\\ &\geq&-2\rho\omega r(r-2)\|w^{k+1}\|^{2}-\eta\omega t_{k}^{2}\|z^{k+1}-w^{k+1}\|^{2}-\frac{\omega r^{2}(r-2)^{2}}{\eta t_{k}^{2}}\|x^{k+1}-x^{\star}\|^{2}\\ &\geq&-4\rho\omega r(r-2)\|z^{k+1}\|^{2}-(\eta\omega t_{k}^{2}+4\rho\omega r(r-2))\|z^{k+1}-w^{k+1}\|^{2}\\ &&-{~}\frac{\omega r^{2}(r-2)^{2}}{\eta t_{k}^{2}}\|x^{k+1}-x^{\star}\|.\end{array} Substituting \hat{\mathcal{T}}_{[4]} and \hat{\mathcal{T}}_{[5]} into the expression \mathcal{G}_{k}-\mathcal{G}_{k+1} , we can further derive \begin{array}[]{lcl}\mathcal{G}_{k}-\mathcal{G}_{k+1}&\geq&\big[S_{k}-\frac{\delta\omega t_{k}}{2}-4\rho\omega r(r-2)\big]\|z^{k+1}\|^{2}-\frac{\omega r^{2}(r-2)^{2}}{\eta t_{k}^{2}}\|x^{k+1}-x^{\star}\|^{2}\\ &&+{~}\eta\omega\big(1-\frac{1}{\Delta r}-M^{2}\eta^{2}\big)t_{k}^{2}\|d^{k}\|^{2}+\eta\omega\hat{\phi}t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\eta\omega(\phi-1)t_{k}^{2}\|z^{k+1}-\hat{w}^{k+1}\|^{2}-\big(\Delta r\eta+8\rho\big)\omega t_{k}^{2}\|z^{k}-w^{k}\|^{2}\\ &&-{~}\hat{\Theta}_{k}\|z^{k+1}-w^{k+1}\|^{2}+\frac{(\beta-8\rho)\omega t_{k}^{2}}{2}\|z^{k+1}-z^{k}\|^{2}\\ &&+{~}\mu\big(t_{k}-r-\mu-\frac{2\mu\eta}{\omega}\big)\|x^{k+1}-x^{k}\|^{2},\end{array} where \hat{\Theta}_{k}:=\omega t_{k}\big[(\eta(5+2\phi)+2\beta+8\rho)t_{k}+2\delta+\omega(r+\mu-1)^{2}\big]+4\rho\omega r(r-2) . This exactly proves ( 100 ). \square F.11 The Proof of Lemma 17 Choosing \phi:=1 in ( 100 ), and \beta:=8\rho+2\epsilon , ( 100 ) reduces to \hskip-4.30554pt\begin{array}[]{lcl}\mathcal{G}_{k}-\mathcal{G}_{k+1}&\geq&\big[S_{k}-\frac{\delta\omega t_{k}}{2}-4\rho\omega r(r-2)\big]\|z^{k+1}\|^{2}-\frac{\omega r^{2}(r-2)^{2}}{\eta t_{k}^{2}}\|x^{k+1}-x^{\star}\|^{2}\\ &&+{~}\eta\omega\big(1-\frac{1}{\Delta r}-M^{2}\eta^{2}\big)t_{k}^{2}\|d^{k}\|^{2}-\big(\Delta r\eta+8\rho\big)\omega t_{k}^{2}\|z^{k}-w^{k}\|^{2}\\ &&+{~}\big(\Delta r\eta+8\rho\big)\omega t_{k+1}^{2}\|z^{k+1}-w^{k+1}\|^{2}\\ &&+{~}\mu\big(t_{k}-r-\mu-\frac{2\mu\eta}{\omega}\big)\|x^{k+1}-x^{k}\|^{2}\\ &&+{~}\eta\omega\hat{\phi}t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}-A_{k}\|z^{k+1}-w^{k+1}\|^{2}\\ &&+{~}\hat{\epsilon}\omega\eta t_{k+1}^{2}\|z^{k+1}-w^{k+1}\|^{2}+\epsilon\omega t_{k}^{2}\|z^{k+1}-z^{k}\|^{2},\end{array}\hskip-17.22217pt (170) where A_{k}:=\Theta_{k}+\big(\Delta r\eta+8\rho+\hat{\epsilon}\eta\big)\omega t_{k+1}^{2} for some \hat{\epsilon}\geq 0 . Now, since \beta=8\rho+2\epsilon\geq 8\rho , \phi=1 , and \eta\geq\frac{(r+\mu-1)\beta}{r-2} , we can show that \begin{array}[]{lcl}A_{k}&=&\omega t_{k}\big[(\eta(5+2\phi)+2\beta+8\rho)t_{k}+2\delta+\omega(r+\mu-1)^{2}\big]+4\rho\omega r(r-2)\\ &&+{~}\omega(\Delta r\eta+8\rho+\hat{\epsilon}\eta)(t_{k}^{2}+2t_{k}+1)\\ &\leq&\big[8+\frac{4(r-2)}{r+\mu-1}+\Delta r+\hat{\epsilon}\big]\eta\omega t_{k}^{2}\\ &\leq&\eta\omega\Theta t_{k}^{2},\end{array} where \Theta:=12+\Delta r+\hat{\epsilon} , provided that t_{k}\geq\frac{D+\sqrt{D^{2}+4\eta E}}{2\eta} with D:=2\delta+\omega(r+\mu-1)^{2}+2\Delta r\eta+2\beta+2\hat{\epsilon}\eta and E:=4\rho r(r-2)+\Delta r\eta+8\rho+\hat{\epsilon}\eta . We also have S_{k}-\frac{\delta\omega t_{k}}{2}-4\rho\omega r(r-2)=\frac{\omega[3(r-2)\psi-(r-1)\beta]t_{k}-\hat{\Gamma}}{2} , where \hat{\Gamma}:=2\Gamma+8\rho\omega r(r-2) . Moreover, from ( 78 ), and z^{k+1}:=u^{k+1}+\xi^{k+1} in ( 75 ), we can write \begin{array}[]{lcl}\|z^{k+1}-w^{k+1}\|^{2}&\leq&\kappa\|w^{k+1}-\hat{w}^{k+1}\|^{2}+\hat{\kappa}\|d^{k}\|^{2}.\end{array} Substituting the last three expressions into ( 170 ), rearranging the result, and using ( 101 ), we can prove that \hskip-8.61108pt\begin{array}[]{lcl}\hat{\mathcal{L}}_{k}-\hat{\mathcal{L}}_{k+1}&\geq&\frac{\omega[3(r-2)\psi-(r-1)\beta]t_{k}-\hat{\Gamma}}{2}\|z^{k+1}\|^{2}-\frac{\omega r^{2}(r-2)^{2}}{\eta t_{k}^{2}}\|x^{k+1}-x^{\star}\|^{2}\\ &&+{~}\eta\omega(\hat{\phi}-\kappa\Theta)t_{k}^{2}\|w^{k+1}-\hat{w}^{k+1}\|^{2}\\ &&+{~}\mu\big(t_{k}-r-\mu-\frac{2\mu\eta}{\omega}\big)\|x^{k+1}-x^{k}\|^{2}\\ &&+{~}\eta\omega\big(1-\frac{1}{\Delta r}-\hat{\kappa}\Theta-M^{2}\eta^{2}\big)t_{k}^{2}\|d^{k}\|^{2}\\ &&+{~}\hat{\epsilon}\omega\eta t_{k+1}^{2}\|z^{k+1}-w^{k+1}\|^{2}+\epsilon\omega t_{k}^{2}\|z^{k+1}-z^{k}\|^{2}.\end{array}\hskip-8.61108pt This is exactly ( 103 ). \square F.12 The Proof of Lemma 18 First, we can easily show that \begin{array}[]{lcl}\tilde{\mathcal{T}}_{[1]}&:=&\|r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})\|^{2}+\mu r\|x^{k}-x^{\star}\|^{2}\\ &=&(r^{2}+r\mu)\|x^{k}-x^{\star}\|^{2}+2rt_{k}\langle x^{k}-x^{\star},y^{k}-x^{k}\rangle+t_{k}^{2}\|y^{k}-x^{k}\|^{2}\\ &=&r\big(\mu-\frac{1}{2}\big)\|x^{k}{\!}-{\!}x^{\star}\|^{2}+\frac{2r+1}{2r}\|r(x^{k}{\!}-{\!}x^{\star})+\frac{2rt_{k}}{2r+1}(y^{k}{\!}-{\!}x^{k})\|^{2}+\frac{t_{k}^{2}}{2r+1}\|y^{k}{\!}-{\!}x^{k}\|^{2}.\end{array} Denote s_{k}:=(\eta-\beta)t_{k}-\eta(r-1) . Then, substituting \tilde{\mathcal{T}}_{[1]} into ( 101 ), and using Young’s inequality and a_{k}\leq\omega t_{k}^{2} , we can derive that \hskip-4.30554pt\begin{array}[]{lcl}\hat{\mathcal{L}}_{k}&:=&\|r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})\|^{2}+r\mu\|x^{k}-x^{\star}\|^{2}+\Lambda_{k}\|z^{k}\|^{2}\\ &&+{~}2rs_{k}\langle z^{k},x^{k}-x^{\star}\rangle+2c_{k}\langle z^{k},y^{k}-x^{k}\rangle\\ &&+{~}2a_{k}\langle z^{k}-w^{k},y^{k}-x^{k}\rangle+\alpha_{k}\|z^{k}-w^{k}\|^{2}\\ &\geq&r\big(\mu-\frac{1}{2}\big)\|x^{k}-x^{\star}\|^{2}+\frac{t_{k}^{2}}{2r+1}\|y^{k}-x^{k}\|^{2}+\Lambda_{k}\|z^{k}\|^{2}\\ &&+{~}\frac{2r+1}{2r}\|r(x^{k}-x^{\star})+\frac{2rt_{k}}{2r+1}(y^{k}-x^{k})\|^{2}\\ &&+{~}\frac{(2r+1)c_{k}}{rt_{k}}\langle z^{k},r(x^{k}-x^{\star})+\frac{2rt_{k}}{2r+1}(y^{k}-x^{k})\rangle\\ &&+{~}\big[2rs_{k}-\frac{(2r+1)c_{k}}{t_{k}}\big]\langle z^{k},x^{k}-x^{\star}\rangle\\ &&-{~}\frac{a_{k}}{(2r+1)\omega}\|y^{k}-x^{k}\|^{2}+\big[\alpha_{k}-(2r+1)\omega a_{k}\big]\|z^{k}-w^{k}\|^{2}\\ &=&r\big(\mu-\frac{1}{2}\big)\|x^{k}-x^{\star}\|^{2}+\big[\Lambda_{k}-\frac{(2r+1)c_{k}^{2}}{2rt_{k}^{2}}\big]\|z^{k}\|^{2}\\ &&+{~}\frac{2r+1}{2r}\|r(x^{k}-x^{\star})+\frac{2rt_{k}}{2r+1}(y^{k}-x^{k})+\frac{c_{k}}{t_{k}}z^{k}\|^{2}\\ &&+{~}\big[2rs_{k}-\frac{(2r+1)c_{k}}{t_{k}}\big]\langle z^{k},x^{k}-x^{\star}\rangle\\ &&+{~}\big[\alpha_{k}-(2r+1)\omega a_{k}\big]\|z^{k}-w^{k}\|^{2}.\end{array}\hskip-8.61108pt (171) We note that if t_{k}\geq\frac{\mu}{\omega}(r-1)[\eta-(2r+1)\omega] , then \hskip-4.30554pt\begin{array}[]{lcl}n_{k}&:=&2rs_{k}-\frac{(2r+1)c_{k}}{t_{k}}\leq 2r\omega t_{k}.\end{array}\hskip-17.22217pt (172) By Young’s inequality and \langle w^{k},x^{k}-x^{\star}\rangle\geq-\rho\|w^{k}\|^{2} , we have \begin{array}[]{lcl}\tilde{\mathcal{T}}_{[2]}&:=&n_{k}\langle z^{k},x^{k}-x^{\star}\rangle\\ &=&n_{k}\langle z^{k}-w^{k},x^{k}-x^{\star}\rangle+n_{k}\langle w^{k},x^{k}-x^{\star}\rangle\\ &\geq&-n_{k}\big(\frac{n_{k}}{2r}+2\rho)\|z^{k}-w^{k}\|^{2}-\frac{r}{2}\|x^{k}-x^{\star}\|^{2}-2\rho n_{k}\|z^{k}\|^{2}.\end{array} Substituting this inequality \tilde{\mathcal{T}}_{[2]} into ( 171 ), we can show that \begin{array}[]{lcl}\hat{\mathcal{L}}_{k}&\geq&r(\mu-1)\|x^{k}-x^{\star}\|^{2}+\big[\Lambda_{k}-\frac{(2r+1)c_{k}^{2}}{2rt_{k}^{2}}-2\rho n_{k}\big]\|z^{k}\|^{2}\\ &&+{~}\frac{2r+1}{2r}\|r(x^{k}-x^{\star})+\frac{2rt_{k}}{2r+1}(y^{k}-x^{k})+\frac{c_{k}}{t_{k}}z^{k}\|^{2}\\ &&+{~}\big[\alpha_{k}-(2r+1)\omega a_{k}-\frac{n_{k}^{2}}{2r}-2\rho n_{k}\big]\|z^{k}-w^{k}\|^{2}.\end{array} (173) One the one hand, we can lower bound \begin{array}[]{lcl}\tilde{\mathcal{T}}_{[3]}&:=&\Lambda_{k}-\frac{(2r+1)c_{k}^{2}}{2rt_{k}^{2}}-2\rho n_{k}\geq\frac{(\eta-\beta)^{2}t_{k}^{2}}{2(\mu+1)^{2}}.\end{array} provided that \mu\geq 1 , r>2 , and t_{k}\geq\hat{t}_{0} , where \hat{t}_{0} is defined as in ( 105 ). On the other hand, using ( 172 ) and a_{k}\leq\omega t_{k} , we can show that \begin{array}[]{lcl}\tilde{\mathcal{T}}_{[4]}&:=&\alpha_{k}-(2r+1)\omega a_{k}-\frac{n_{k}^{2}}{2r}-2\rho n_{k}\geq\big[\frac{\Delta r\eta}{\omega}-4r-1\big]\omega^{2}t_{k}^{2},\end{array} due to t_{k}\geq r>\frac{r}{2} . Substituting \tilde{\mathcal{T}}_{[3]} and \tilde{\mathcal{T}}_{[4]} into ( 173 ), we obtain ( 104 ). Finally, if we choose \Delta:=3\geq\frac{(\eta-\beta)[1+2(4r+1)\mu]}{2r\eta\mu(\mu+1)} , then we have \big[\frac{\Delta r\eta}{\omega}-4r-1\big]\omega^{2}\geq\frac{(\eta-\beta)^{2}}{2(\mu+1)^{2}} . Consequently, ( 104 ) leads to \begin{array}[]{lcl}\hat{\mathcal{L}}_{k}&\geq&r(\mu-1)\|x^{k}-x^{\star}\|^{2}+\frac{(\eta-\beta)^{2}t_{k}^{2}}{2(\mu+1)^{2}}\big[\|z^{k}\|^{2}+\|z^{k}-w^{k}\|^{2}\big]\\ &\geq&r(\mu-1)\|x^{k}-x^{\star}\|^{2}+\frac{(\eta-\beta)^{2}t_{k}^{2}}{4(\mu+1)^{2}}\|w^{k}\|^{2},\end{array} which proves ( 106 ). \square F.13 The Proof of Theorem 7.4 First, since \Delta=3 , we have \Theta=12+\Delta r+\hat{\epsilon}=12+3r+\hat{\epsilon} in Lemma 17 . Let us choose \hat{\phi}=\kappa\Theta=\kappa(12+3r+\hat{\epsilon}) in Lemma 17 . In this case, M^{2}=((1+\phi)(1+c_{1})+\hat{\phi})L^{2}=(4+12\kappa+3\kappa r+\kappa\hat{\epsilon})L^{2} in Lemma 12 . Next, to guarantee 1-\frac{1}{\Delta r}-\hat{\kappa}\Theta>0 , we need to impose 0\leq\hat{\kappa}<\frac{3r-1}{3r(12+3r+\hat{\epsilon})} , which is exactly the condition ( 107 ). Since we require 1-\frac{1}{\Delta r}-\hat{\kappa}\Theta-M^{2}\eta^{2}\geq 0 , we need to choose \begin{array}[]{lcl}\eta\leq\bar{\eta}:=\frac{\Psi}{L}=\frac{1}{L\sqrt{4+12\kappa+3\kappa r+\kappa\hat{\epsilon}}}\big[1-\frac{1}{3r}-\hat{\kappa}(12+3r+\hat{\epsilon})\big],\end{array} where \Psi is given in ( 108 ). Moreover, we have imposed \eta\geq\frac{\beta(r+\mu-1)}{r-2} in Lemma 17 , leading to the update rule of \eta as in ( 109 ). The choice of \beta:=8\rho+2\epsilon was also enforced in Lemma 17 for some \epsilon\geq 0 . Now, let us denote \hskip-8.61108pt\begin{array}[]{lcl}\mathcal{H}_{k}&:=&\frac{\omega[3(r-2)\psi-(r-1)\beta]t_{k}-\hat{\Gamma}}{2}\|z^{k+1}\|^{2}+\eta\omega M^{2}(\bar{\eta}^{2}-\eta^{2}\big)t_{k}^{2}\|d^{k}\|^{2}\\ &&+{~}\mu\big(t_{k}-r-\mu-\frac{2\mu\eta}{\omega}\big)\|x^{k+1}-x^{k}\|^{2}\\ &&+{~}\hat{\epsilon}\omega\eta t_{k+1}^{2}\|z^{k+1}-w^{k+1}\|^{2}+\epsilon\omega t_{k}^{2}\|z^{k+1}-z^{k}\|^{2}.\end{array}\hskip-8.61108pt (174) Then, under the above parameter selections, ( 103 ) reduces to \begin{array}[]{lcl}\hat{\mathcal{L}}_{k}-\hat{\mathcal{L}}_{k+1}&\geq&\mathcal{H}_{k}-\frac{\omega r^{2}(r-2)^{2}}{\eta t_{k}^{2}}\|x^{k+1}-x^{\star}\|^{2}\geq\mathcal{H}_{k}-\frac{\omega r(r-2)^{2}}{\eta(\mu-1)t_{k}^{2}}\hat{\mathcal{L}}_{k+1},\end{array} where we have used \hat{\mathcal{L}}_{k+1}\geq r(\mu-1)\|x^{k+1}-x^{\star}\|^{2} from ( 106 ) in the second inequality. Denote \tau_{k+1}:=\frac{\omega r(r-2)^{2}}{\eta(\mu-1)t_{k}^{2}}=\frac{\omega r(r-2)^{2}}{\eta(\mu-1)(k+t_{0})^{2}} . Then, by the choice of t_{0} in ( 108 ), we have \tau_{k+1}\in(0,1) . Thus the last inequality leads to \begin{array}[]{lcl}(1-\tau_{k+1})\hat{\mathcal{L}}_{k+1}\leq\big(1+\frac{\tau_{k}}{1-\tau_{k}}\big)(1-\tau_{k})\hat{\mathcal{L}}_{k}-\mathcal{H}_{k}\end{array} Since \sum_{k=0}^{\infty}\frac{\tau_{k}}{1-\tau_{k}}<\infty , (1-\tau_{k})\hat{\mathcal{L}}_{k}\geq 0 , and \mathcal{H}_{k}\geq 0 , applying ( 9 , Lemma 5.31) , we can show that \sum_{k=0}^{\infty}\mathcal{H}_{k}<+\infty and \lim_{k\to\infty}\hat{\mathcal{L}}_{k} exists. Since t_{k}-r-\mu-\frac{2\mu\eta}{\omega}\geq\frac{t_{k}}{2} due to the choice of t_{0} , the summable result \sum_{k=0}^{\infty}\mathcal{H}_{k}<+\infty implies the first five summable results in ( 110 ). The last summable result follows from \|w^{k}\|^{2}\leq 2\|z^{k}-w^{k}\|^{2}+2\|z^{k}\|^{2} . From the proof of ( 9 , Lemma 5.31) , we also have \begin{array}[]{lcl}\hat{\mathcal{L}}_{k}\leq\frac{1}{(1-\tau_{k})}\hat{\mathcal{L}}_{k-1}\leq\prod_{i=1}^{k}\frac{1}{1-\tau_{i}}\hat{\mathcal{L}}_{0}\leq\prod_{k=1}^{\infty}\frac{1}{1-\tau_{k}}\hat{\mathcal{L}}_{0}.\end{array} Denoting \tau:=\prod_{k=1}^{\infty}\frac{1}{1-\tau_{k}} . Then, using an elementary proof, we can show that 1\leq\tau\leq\Omega:=\exp\left(\frac{c^{2}}{(1+t_{0})^{2}-c^{2}}\right)\left(\frac{1+t_{0}+c}{1+t_{0}-c}\right)^{\frac{c}{2}} , where c^{2}:=\frac{\omega r(r-2)^{2}}{\eta(\mu-1)} . Hence, the last inequality leads to 0\leq\hat{\mathcal{L}}_{k}\leq\Omega\hat{\mathcal{L}}_{0} for all k\geq 0 . Finally, since z^{0}:=w^{0} and y^{0}:=x^{0} , it is obvious to show that \hat{\mathcal{L}}_{0}\leq\mathcal{R}_{0}^{2} for \mathcal{R}_{0}^{2} defined by ( 92 ). By ( 106 ) and \hat{\mathcal{L}}_{k}\leq\Omega\hat{\mathcal{L}}_{0}\leq\Omega\mathcal{R}_{0}^{2} , we have \frac{(\eta-\beta)^{2}t_{k}^{2}}{4(\mu+1)^{2}}\|w^{k}\|^{2}\leq\hat{\mathcal{L}}_{k}\leq\Omega\mathcal{R}_{0}^{2} . This relation implies ( 111 ). \square F.14 The Proof of Theorem 7.5 Denote \hat{v}^{k}:=y^{k}-x^{k}+\eta_{k-1}z^{k} . Similar to the proof of ( 161 ), we have \hskip-4.30554pt\begin{array}[]{lcl}t_{k+1}^{2}\|\hat{v}^{k+1}\|^{2}&\leq&t_{k}^{2}\|\hat{v}^{k}\|^{2}-[(r-2)t_{k}+r-1]\|\hat{v}^{k}\|^{2}+\frac{2\eta^{2}(r+1)^{2}t_{k+1}}{(r+2)}\|d^{k}\|^{2}\\ &&+{~}\frac{2[(r+4)(\eta-\beta)+1]^{2}t_{k+1}}{2(r+2)}\|z^{k}\|^{2}.\end{array}\hskip-8.61108pt This inequality together with ( 110 ) imply \begin{array}[]{ll}&\lim_{k\to\infty}t_{k}^{2}\|y^{k}-x^{k}+\eta_{k-1}z^{k}\|^{2}=0,\\ &\sum_{k=0}^{\infty}t_{k}\|y^{k}-x^{k}+\eta_{k-1}z^{k}\|^{2}<+\infty.\end{array} (175) Next, we write \begin{array}[]{lcl}\eta\theta_{k}d^{k}+p^{k}&=&(\eta\theta_{k}-\lambda_{k})\hat{w}^{k+1}+\eta_{k}z^{k+1}-(\eta\theta_{k}\gamma_{k}-\nu_{k})z^{k}\\ &=&-\frac{\eta(r+\mu)}{t_{k+1}}\hat{w}^{k+1}+\frac{[(\eta-\beta)t_{k}-\delta]}{t_{k+1}}z^{k+1}-\big[\frac{(\eta-\beta)\theta_{k}t_{k-1}}{t_{k}}-\frac{s_{k}}{t_{k+1}}\big]z^{k},\end{array} where s_{k}:=\frac{[\eta(r-2)+\beta(r+\mu+1)](t_{k}-r-\mu)+(r+\mu)^{2}\beta}{t_{k}} . Using this expression and ( GAEG+ ), we can show that \begin{array}[]{lcl}\zeta^{k+1}&:=&y^{k+1}-x^{k+1}+\eta_{k}(z^{k+1}-w^{k+1})\\ &=&\theta_{k}[(y^{k}-x^{k})+\eta_{k-1}(z^{k}-w^{k})]-\theta_{k}\eta_{k-1}(z^{k}-w^{k})-\eta\theta_{k}d^{k}\\ &&-{~}p^{k}+\eta_{k}(z^{k+1}-w^{k+1})\\ &=&\theta_{k}\zeta^{k}-\eta_{k}(w^{k+1}-w^{k})+\frac{\eta(r+\mu)}{t_{k+1}}d^{k}+\frac{\hat{s}_{k}}{t_{k+1}}(z^{k}-w^{k})-\frac{\tilde{s}_{k}}{t_{k+1}}z^{k},\end{array} where \hat{s}_{k}:=\frac{(r+\mu+1)(\eta-\beta)t_{k}-(r+\mu)(\eta-\beta+\delta)}{t_{k}}\leq\hat{s}:=(r+\mu+1)(\eta-\beta) and \tilde{s}_{k}:=\eta(r-1)-\delta:=\tilde{s}>0 . Denote h^{k}:=\eta(r+\mu)d^{k}+\hat{s}_{k}(z^{k}-w^{k})-\tilde{s}_{k}z^{k} . Using the last expression and Young’s inequality, we can derive \begin{array}[]{lcl}\|\zeta^{k+1}\|^{2}&=&\theta_{k}^{2}\|\zeta^{k}\|^{2}-2\eta_{k}\theta_{k}\langle w^{k+1}-w^{k},\zeta^{k}\rangle+\frac{2\theta_{k}}{t_{k+1}}\langle\zeta^{k},h^{k}\rangle-\frac{2\eta_{k}}{t_{k+1}}\langle h^{k},w^{k+1}-w^{k}\rangle\\ &&+{~}\eta_{k}^{2}\|w^{k+1}-w^{k}\|^{2}+\frac{1}{t_{k+1}^{2}}\|h^{k}\|^{2}\\ &\leq&\big(\theta_{k}^{2}+\frac{\theta_{k}}{t_{k+1}}\big)\|\zeta^{k}\|^{2}-2\eta_{k}\theta_{k}\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle\\ &&-{~}2\eta_{k}\theta_{k}\langle w^{k+1}-w^{k},\eta d^{k}+\eta_{k-1}(z^{k}-w^{k})\rangle\\ &&+{}\big(\frac{1}{t_{k+1}^{2}}+\frac{\theta_{k}}{t_{k+1}}+\frac{\eta_{k}}{t_{k+1}}\big)\|h^{k}\|^{2}+\left(\eta_{k}^{2}+\frac{\eta_{k}}{t_{k+1}}\right)\|w^{k+1}-w^{k}\|^{2}\\ &\leq&\big(\theta_{k}^{2}+\frac{\theta_{k}}{t_{k+1}}\big)\|\zeta^{k}\|^{2}+\big(2\rho\eta_{k}\theta_{k}+\eta_{k}\theta_{k}+\eta_{k}^{2}+\frac{\eta_{k}}{t_{k+1}}\big)\|w^{k+1}-w^{k}\|^{2}\\ &&+{~}2\eta^{2}\eta_{k}\theta_{k}\|d^{k}\|^{2}+2\eta_{k}\theta_{k}\eta_{k-1}^{2}\|z^{k}-w^{k}\|^{2}+\big(\frac{1}{t_{k+1}^{2}}+\frac{\theta_{k}}{t_{k+1}}+\frac{\eta_{k}}{t_{k+1}}\big)\|h^{k}\|^{2},\end{array} where we have used \langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle\geq-\rho\|w^{k+1}-w^{k}\|^{2} from the \rho -co-hypomonotonicity of \Phi in the last inequality. Let us denote \begin{array}[]{lcl}\Sigma_{k}&:=&\big(2\rho\eta_{k}\theta_{k}+\eta_{k}\theta_{k}+\eta_{k}^{2}+\frac{\eta_{k}}{t_{k+1}}\big)\|w^{k+1}-w^{k}\|^{2}+2\eta^{2}\eta_{k}\theta_{k}\|d^{k}\|^{2}\\ &&+{~}2\eta_{k}\theta_{k}\eta_{k-1}^{2}\|z^{k}-w^{k}\|^{2}+\big(\frac{1}{t_{k+1}^{2}}+\frac{\theta_{k}}{t_{k+1}}+\frac{\eta_{k}}{t_{k+1}}\big)\|h^{k}\|^{2}.\end{array} (176) Then, the last inequality can be rewritten as \begin{array}[]{lcl}\|\zeta^{k+1}\|^{2}&\leq&\big(\theta_{k}^{2}+\frac{\theta_{k}}{t_{k+1}}\big)\|\zeta^{k}\|^{2}+\Sigma_{k}.\end{array} Multiplying this inequality by t_{k+1}^{2} , we can show that \begin{array}[]{lcl}t_{k+1}^{2}\|\zeta^{k+1}\|^{2}&\leq&t_{k}^{2}\|\zeta^{k}\|^{2}+t_{k+1}^{2}\Sigma_{k}\\ &&-{~}[(2r+2\mu-1)t_{k}-(r+\mu)(r+\mu-1)]\|\zeta^{k}\|^{2}.\end{array} (177) Our next step is to prove that t_{k+1}^{2}\Sigma_{k} is summable. Indeed, from ( 176 ), by Young’s inequality, we have \begin{array}[]{lcl}\Sigma_{k}&\leq&3\big(2\rho\eta_{k}\theta_{k}+\eta_{k}\theta_{k}+\eta_{k}^{2}+\frac{\eta_{k}}{t_{k+1}}\big)\big[\|z^{k+1}-w^{k+1}\|^{2}+\|z^{k+1}-z^{k}\|^{2}\big]\\ &&+{~}\big[3\big(2\rho\eta_{k}\theta_{k}+\eta_{k}\theta_{k}+\eta_{k}^{2}+\frac{\eta_{k}}{t_{k+1}}\big)+2\eta_{k}\theta_{k}\eta_{k-1}^{2}\big]\|z^{k}-w^{k}\|^{2}\\ &&+{~}\big(\frac{1}{t_{k+1}^{2}}+\frac{\theta_{k}}{t_{k+1}}+\frac{\eta_{k}}{t_{k+1}}\big)\|\eta(r+\mu)d^{k}+\hat{s}_{k}(z^{k}-w^{k})-\tilde{s}_{k}z^{k}\|^{2}\\ &&+{~}2\eta^{2}\eta_{k}\theta_{k}\|d^{k}\|^{2}\\ &\leq&3\big(2\rho\eta_{k}\theta_{k}+\eta_{k}\theta_{k}+\eta_{k}^{2}+\frac{\eta_{k}}{t_{k+1}}\big)\big[\|z^{k+1}-w^{k+1}\|^{2}+\|z^{k+1}-z^{k}\|^{2}\big]\\ &&+{~}B_{k}\|z^{k}-w^{k}\|^{2}+3\tilde{s}_{k}^{2}\big(\frac{1}{t_{k+1}^{2}}+\frac{\theta_{k}}{t_{k+1}}+\frac{\eta_{k}}{t_{k+1}}\big)\|z^{k}\|^{2}\\ &&+{~}\Big[2\eta^{2}\eta_{k}\theta_{k}+3\eta^{2}(r+\mu)^{2}\big(\frac{1}{t_{k+1}^{2}}+\frac{\theta_{k}}{t_{k+1}}+\frac{\eta_{k}}{t_{k+1}}\big)\Big]\|d^{k}\|^{2},\end{array} where B_{k}:=3\big(2\rho\eta_{k}\theta_{k}+\eta_{k}\theta_{k}+\eta_{k}^{2}+\frac{\eta_{k}}{t_{k+1}}\big)+2\eta_{k}\theta_{k}\eta_{k-1}^{2}+3\hat{s}_{k}^{2}\big(\frac{1}{t_{k+1}^{2}}+\frac{\theta_{k}}{t_{k+1}}+\frac{\eta_{k}}{t_{k+1}}\big) . Using the facts that \eta_{k}\leq\frac{(\eta-\beta)t_{k}}{t_{k+1}}\leq\eta-\beta , \theta_{k}\leq\frac{t_{k}}{t_{k+1}} , \hat{s}_{k}\leq\hat{s}:=(r+\mu+1)(\eta-\beta) , and \tilde{s}_{k}=\tilde{s}:=\eta(r-1)-\delta , we can prove that \begin{array}[]{lcl}t_{k+1}^{2}\Sigma_{k}&\leq&3(\eta-\beta)\big[(2\rho+1+\eta-\beta)t_{k}^{2}+t_{k}\big]\big[\|z^{k+1}-w^{k+1}\|^{2}+\|z^{k+1}-z^{k}\|^{2}\big]\\ &&+{~}\Big[2\eta^{2}(\eta-\beta)t_{k}^{2}+3\eta^{2}(r+\mu)^{2}(\eta-\beta+1)t_{k}+3\eta^{2}(r+\mu)^{2}\Big]\|d^{k}\|^{2}\\ &&+{~}3\tilde{s}^{2}\big[(\eta-\beta+1)t_{k}+1\big]\|z^{k}\|^{2}+\tilde{B}_{k}\|z^{k}-w^{k}\|^{2},\end{array} where \tilde{B}_{k}:=(\eta-\beta)\big[(6\rho+3+3\eta-3\beta+2(\eta-\beta)^{2})t_{k}^{2}+3t_{k}\big]+3[\hat{s}^{2}(\eta-\beta+1)+\eta-\beta]t_{k}+3\hat{s}^{2}=\mathcal{O}\left(t_{k}^{2}\right) . Now, applying ( 110 ) to this inequality, one can establish that \begin{array}[]{lcl}\sum_{k=0}^{\infty}t_{k+1}^{2}\Sigma_{k}<+\infty.\end{array} (178) Then, applying ( 178 ), we can prove from ( 177 ) that \begin{array}[]{ll}&\lim_{k\to\infty}t_{k}^{2}\|y^{k}-x^{k}+\eta_{k-1}(z^{k}-w^{k})\|^{2}=0,\\ &\sum_{k=0}^{\infty}t_{k}\|y^{k}-x^{k}+\eta_{k-1}(z^{k}-w^{k})\|^{2}<+\infty.\end{array} (179) In addition, by Young’s inequality, we get \begin{array}[]{ll}\eta_{k-1}^{2}\|w^{k}\|^{2}&\leq 2\|y^{k}-x^{k}+\eta_{k-1}(z^{k}-w^{k})\|^{2}+2\|y^{k}-x^{k}+\eta_{k-1}z^{k}\|^{2}.\end{array} Using ( 176 ) and ( 179 ) into this inequality, we obtain \lim_{k\to\infty}t_{k}^{2}\eta_{k-1}^{2}\|w^{k}\|^{2}=0 . Since \eta_{k-1}=\frac{(\eta-\beta)t_{k-1}-\delta}{t_{k}}=\eta-\beta-\frac{(\eta-\beta-\delta)}{t_{k}}\geq\delta as t_{k}\geq 1 , the last limit leads to \lim_{k\to\infty}t_{k}^{2}\|w^{k}\|^{2}=0 , which proves the first line of ( 112 ). By Young’s inequality again, we also have \begin{array}[]{lcl}t_{k}^{2}\|y^{k}-x^{k}\|^{2}&\leq&2t_{k}^{2}\|y^{k}-x^{k}+\eta_{k-1}(z^{k}-w^{k})\|^{2}+2\eta_{k-1}^{2}t_{k}^{2}\|z^{k}-w^{k}\|^{2}.\end{array} This inequality together with ( 179 ) and the summability of t_{k}^{2}\|z^{k}-w^{k}\|^{2} imply the second line of ( 112 ). Finally, since t_{k}^{2}\|x^{k+1}-x^{k}\|^{2}\leq 2\eta^{2}t_{k}^{2}\|d^{k}\|^{2}+2t_{k}^{2}\|y^{k}-x^{k}\|^{2} , applying this relation, the second line of ( 112 ), and the summability of t_{k}^{2}\|d^{k}\|^{2} , we get the third line of ( 112 ). \square F.15 Technical Lemma 21 and Lemma 22 First, let us define the following quantities: \hskip-4.30554pt\begin{array}[]{lcl}\mathcal{A}_{k}&:=&\|r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})+t_{k}\eta_{k-1}(z^{k}-w^{k})\|^{2}+r\mu\|x^{k}-x^{\star}\|^{2},\\ \hat{\mathcal{Q}}_{k}&:=&\mathcal{A}_{k}+2rt_{k}\eta_{k-1}\big[\langle w^{k},x^{k}-x^{\star}\rangle+\rho\|w^{k}\|^{2}].\end{array}\hskip-8.61108pt (180) Clearly, by the \rho -co-hypomonotonicity of \Phi , we have \hat{\mathcal{Q}}_{k}\geq 0 . Lemma 21 Under the same settings as in Theorem 7.5 , \hat{\mathcal{Q}}_{k} defined by ( 180 ) satisfies the following inequality: \hskip-8.61108pt\begin{array}[]{lcl}\hat{\mathcal{Q}}_{k}-\hat{\mathcal{Q}}_{k+1}&\geq&\mu(2t_{k}-r-\mu)\|x^{k+1}-x^{k}\|^{2}-2[(\eta-\beta)t_{k}-\delta]^{2}\|z^{k+1}-w^{k+1}\|^{2}\\ &&-{~}2t_{k+1}^{2}\|p^{k}\|^{2}-2\rho rt_{k+1}\eta_{k}\|w^{k+1}\|^{2}\\ &&+{~}2[(\eta-\beta)t_{k}-\delta](t_{k}-r-\mu)\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle\\ &&+{~}2r[\eta(r-2)+\beta-\delta]\langle z^{k},x^{k+1}-x^{\star}\rangle+2\mu\eta t_{k}\langle\hat{w}^{k+1},x^{k+1}-x^{k}\rangle\\ &&+{~}2s^{3}_{k}\langle z^{k}-w^{k},x^{k+1}-x^{k}\rangle-2\hat{s}^{3}_{k}\langle z^{k},x^{k+1}-x^{k}\rangle,\end{array}\hskip-38.74988pt (181) where \left\{\begin{array}[]{lcl}s^{3}_{k}&:=&(\eta-\beta)(r+\mu-1)t_{k}-(r+\mu)\delta,\\ \hat{s}^{3}_{k}&:=&[(\mu+1)\eta+\delta-r\beta]t_{k}-r(2\eta-\beta+2\delta-r\eta)-\delta\mu.\end{array}\right. Proof Denote g^{k}:=\eta d^{k}+\eta_{k-1}(z^{k}-w^{k}) . Using this g^{k} and x^{k+1}=y^{k}-\eta d^{k} from ( GAEG+ ) and ( 76 ), similar to the proof of Lemma 10 , we have \begin{array}[]{lcl}\breve{\mathcal{T}}_{[1]}&:=&\|r(x^{k}-x^{\star})+t_{k}(y^{k}-x^{k})+t_{k}\eta_{k-1}(z^{k}-w^{k})\|^{2}\\ &=&r^{2}\|x^{k+1}-x^{\star}\|^{2}+(t_{k}-r)^{2}\|x^{k+1}-x^{k}\|^{2}+t_{k}^{2}\|g^{k}\|^{2}\\ &&+{~}2r(t_{k}-r)\langle x^{k+1}-x^{\star},x^{k+1}-x^{k}\rangle+2rt_{k}\langle g^{k},x^{k+1}-x^{\star}\rangle\\ &&+{~}2t_{k}(t_{k}-r)\langle g^{k},x^{k+1}-x^{k}\rangle.\end{array} Alternatively, denote h^{k}:=p^{k}-\eta_{k}(z^{k+1}-w^{k+1}) . Then, similar to the proof of Lemma 10 , we can show from ( GAEG+ ) that \begin{array}[]{lcl}\breve{\mathcal{T}}_{[2]}&:=&\|r(x^{k+1}-x^{\star})+t_{k+1}(y^{k+1}-x^{k+1})+t_{k+1}\eta_{k}(z^{k+1}-w^{k+1})\|^{2}\\ &=&r^{2}\|x^{k+1}-x^{\star}\|^{2}+t_{k+1}^{2}\theta_{k}^{2}\|x^{k+1}-x^{k}\|^{2}+t_{k+1}^{2}\|h^{k}\|^{2}\\ &&+{~}2rt_{k+1}\theta_{k}\langle x^{k+1}-x^{\star},x^{k+1}-x^{k}\rangle-2rt_{k+1}\langle h^{k},x^{k+1}-x^{\star}\rangle\\ &&-{~}2t_{k+1}^{2}\theta_{k}\langle h^{k},x^{k+1}-x^{k}\rangle.\end{array} Combining both expressions \breve{\mathcal{T}}_{[1]} and \breve{\mathcal{T}}_{[2]} , and the identity \mu r\|x^{k}-x^{\star}\|^{2}-\mu r\|x^{k+1}-x^{\star}\|^{2}=\mu r\|x^{k+1}-x^{k}\|^{2}-2r\mu\langle x^{k+1}-x^{k},x^{k+1}-x^{\star}\rangle , we can derive from ( 180 ) that \begin{array}[]{lcl}\mathcal{A}_{k}-\mathcal{A}_{k+1}&=&\mu(2t_{k}-r-\mu)\|x^{k+1}-x^{k}\|^{2}+t_{k}^{2}\|g^{k}\|^{2}-t_{k+1}^{2}\|h^{k}\|^{2}\\ &&+{~}2\langle t_{k}(t_{k}-r)g^{k}+t_{k+1}^{2}\theta_{k}h^{k},x^{k+1}-x^{k}\rangle\\ &&+{~}2r\langle t_{k}g^{k}+t_{k+1}h^{k},x^{k+1}-x^{\star}\rangle.\end{array} (182) Next, we note that \begin{array}[]{lcl}t_{k}g^{k}+t_{k+1}h^{k}&=&\eta t_{k}\hat{w}^{k+1}-\eta t_{k}\gamma_{k}z^{k}+t_{k}\eta_{k-1}(z^{k}-w^{k})+t_{k+1}\eta_{k}z^{k+1}\\ &&-{~}t_{k+1}\lambda_{k}\hat{w}^{k+1}+t_{k+1}\nu_{k}z^{k}-t_{k+1}\eta_{k}(z^{k+1}-w^{k+1})\\ &=&(\eta t_{k}-t_{k+1}\lambda_{k})\hat{w}^{k+1}+t_{k+1}\eta_{k}w^{k+1}-t_{k}\eta_{k-1}w^{k}\\ &&-{~}(\eta t_{k}\gamma_{k}-t_{k+1}\nu_{k}-t_{k}\eta_{k-1})z^{k}\\ &=&t_{k+1}\eta_{k}w^{k+1}-t_{k}\eta_{k-1}w^{k}+[\eta(r-2)+\beta-\delta]z^{k}.\end{array} Therefore, one can prove that \begin{array}[]{lcl}\breve{\mathcal{T}}_{[3]}&:=&\langle t_{k}g^{k}+t_{k+1}h^{k},x^{k+1}-x^{\star}\rangle\\ &=&t_{k+1}\eta_{k}\langle w^{k+1},x^{k+1}-x^{\star}\rangle-t_{k}\eta_{k-1}\langle w^{k},x^{k}-x^{\star}\rangle\\ &&-{~}t_{k}\eta_{k-1}\langle w^{k},x^{k+1}-x^{k}\rangle-(\eta r+\beta-\delta)\langle z^{k},x^{k+1}-x^{\star}\rangle\\ &=&t_{k+1}\eta_{k}\langle w^{k+1},x^{k+1}-x^{\star}\rangle-t_{k}\eta_{k-1}\langle w^{k},x^{k}-x^{\star}\rangle\\ &&+{~}t_{k}\eta_{k-1}\langle z^{k}-w^{k},x^{k+1}-x^{k}\rangle-t_{k}\eta_{k-1}\langle z^{k},x^{k+1}-x^{k}\rangle\\ &&+{~}[\eta(r-2)+\beta-\delta]\langle z^{k},x^{k+1}-x^{\star}\rangle.\end{array} Similarly, we also get \begin{array}[]{lcl}\breve{\mathcal{T}}_{[4]}&:=&t_{k}(t_{k}-r)g^{k}+t_{k+1}^{2}\theta_{k}h^{k}\\ &=&\eta t_{k}(t_{k}-r)\hat{w}^{k+1}-\eta t_{k}(t_{k}-r)\gamma_{k}z^{k}+t_{k}(t_{k}-r)\eta_{k-1}(z^{k}-w^{k})\\ &&+{~}t_{k+1}^{2}\theta_{k}\eta_{k}z^{k+1}-t_{k+1}^{2}\theta_{k}\lambda_{k}\hat{w}^{k+1}+t_{k+1}^{2}\theta_{k}\nu_{k}z^{k}-t_{k+1}^{2}\theta_{k}\eta_{k}(z^{k+1}-w^{k+1})\\ &=&\mu\eta t_{k}\hat{w}^{k+1}+[(\eta-\beta)t_{k}-\delta](t_{k}-r-\mu)(w^{k+1}-w^{k})\\ &&+{~}[(\mu-1)(\eta-\beta)t_{k}-\mu\delta+r(\eta-\beta)](z^{k}-w^{k})\\ &&+{~}\big\{[\eta(r-\mu-1)-\delta]t_{k}-\eta r(r-1)+\delta(r+\mu)\big\}z^{k}.\end{array} This expression leads to \begin{array}[]{lcl}\breve{\mathcal{T}}_{[5]}&:=&\langle t_{k}(t_{k}-r)g^{k}+t_{k+1}^{2}\theta_{k}h^{k},x^{k+1}-x^{k}\rangle\\ &=&\mu\eta t_{k}\langle\hat{w}^{k+1},x^{k+1}-x^{k}\rangle+[(\eta-\beta)t_{k}-\delta](t_{k}-r-\mu)\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle\\ &&+{~}[(\mu-1)(\eta-\beta)t_{k}-\mu\delta+r(\eta-\beta)]\langle z^{k}-w^{k},x^{k+1}-x^{k}\rangle\\ &&+{~}\big\{[\eta(r-\mu-1)-\delta]t_{k}-\eta r(r-1)+\delta(r+\mu)\big\}\langle z^{k},x^{k+1}-x^{k}\rangle.\end{array} Substituting \breve{\mathcal{T}}_{[3]} and \breve{\mathcal{T}}_{[5]} into ( 182 ), we can show that \begin{array}[]{lcl}\mathcal{A}_{k}-\mathcal{A}_{k+1}&=&\mu(2t_{k}-r-\mu)\|x^{k+1}-x^{k}\|^{2}+t_{k}^{2}\|g^{k}\|^{2}-t_{k+1}^{2}\|h^{k}\|^{2}\\ &&+{~}2rt_{k+1}\eta_{k}\langle w^{k+1},x^{k+1}-x^{\star}\rangle-2rt_{k}\eta_{k-1}\langle w^{k},x^{k}-x^{\star}\rangle\\ &&+{~}2r[\eta(r-2)+\beta-\delta]\langle z^{k},x^{k+1}-x^{\star}\rangle+2\mu\eta t_{k}\langle\hat{w}^{k+1},x^{k+1}-x^{k}\rangle\\ &&+{~}2[(\eta-\beta)t_{k}-\delta](t_{k}-r-\mu)\langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle\\ &&+{~}2s^{3}_{k}\langle z^{k}-w^{k},x^{k+1}-x^{k}\rangle-2\hat{s}^{3}_{k}\langle z^{k},x^{k+1}-x^{k}\rangle,\end{array} where \begin{array}[]{lcl}s^{3}_{k}&:=&(\eta-\beta)(r+\mu-1)t_{k}-(r+\mu)\delta\\ \hat{s}^{3}_{k}&:=&[(\mu+1)\eta+\delta-r\beta]t_{k}-r(2\eta-\beta+2\delta-r\eta)-\delta\mu.\end{array} Finally, using the definition of \hat{\mathcal{Q}}_{k} from ( 180 ) and t_{k+1}^{2}\|h^{k}\|^{2}\leq 2t_{k+1}^{2}\|p^{k}\|^{2}+2[(\eta-\beta)t_{k}-\delta]^{2}\|z^{k+1}-w^{k+1}\|^{2} , and neglecting the nonnegative terms t_{k}^{2}\|g^{k}\|^{2} and 2\rho rt_{k}\eta_{k-1}\|w^{k}\|^{2} , the last expression leads to ( 181 ). \square Lemma 22 Under the same settings as in Theorem 7.5 , we have \begin{array}[]{lcl}\sum_{k=0}^{\infty}t_{k}|\langle z^{k},x^{k+1}-x^{k}\rangle|&<&+\infty,\\ \sum_{k=0}^{\infty}t_{k}|\langle\hat{w}^{k+1},x^{k+1}-x^{k}\rangle|&<&+\infty,\\ \sum_{k=0}^{\infty}t_{k}|\langle z^{k}-w^{k},x^{k+1}-x^{k}\rangle|&<&+\infty,\\ \sum_{k=0}^{\infty}t_{k}^{2}\|w^{k+1}-w^{k}\|^{2}&<&+\infty,\\ \sum_{k=0}^{\infty}t_{k}^{2}\|p^{k}\|^{2}&<&+\infty.\end{array} (183) In addition, for \Omega given in Theorem 7.4 , and \mathcal{R}_{0}^{2} in ( 92 ), we also have \begin{array}[]{lcl}2\langle z^{k},x^{k+1}-x^{\star}\rangle&\geq&-2\rho\|w^{k}\|^{2}-\|z^{k}\|^{2}-\|x^{k+1}-x^{k}\|^{2}\\ &&-{~}t_{k}^{2}\|z^{k}-w^{k}\|^{2}-\frac{\Omega\mathcal{R}_{0}^{2}}{r(\mu-1)t_{k}^{2}}.\end{array} (184) Proof Since 2t_{k}|\langle z^{k},x^{k+1}-x^{k}\rangle|\leq t_{k}\|z^{k}\|^{2}+t_{k}\|x^{k+1}-x^{k}\|^{2} , we obtain the first line of ( 183 ) from this relation and the summable results in ( 110 ). Next, we have \|\hat{w}^{k+1}\|^{2}\leq 2\|d^{k}\|^{2}+2\gamma_{k}^{2}\|z^{k}\|^{2}\leq 2\|d^{k}\|^{2}+2\|z^{k}\|^{2} . Therefore, we can show that \begin{array}[]{lcl}2t_{k}|\langle\hat{w}^{k+1},x^{k+1}-x^{k}\rangle|&\leq&t_{k}\|\hat{w}^{k+1}\|^{2}+t_{k}\|x^{k+1}-x^{k}\|^{2}\\ &\leq&2t_{k}\|d^{k}\|^{2}+2t_{k}\|z^{k}\|^{2}+t_{k}\|x^{k+1}-x^{k}\|^{2}.\end{array} This inequality together with the summable results in ( 110 ) imply the second line of ( 183 ). The third line of ( 183 ) is proven similarly, and hence, we omit. Since \|w^{k+1}-w^{k}\|^{2}\leq 3\|z^{k+1}-w^{k+1}\|^{2}+3\|z^{k+1}-z^{k}\|^{2}+3\|z^{k}-w^{k}\|^{2} due to Young’s inequality, we obtain the fourth line of ( 183 ) by combining this inequality and ( 110 ). We also have \begin{array}[]{lcl}t_{k+1}^{2}\|p^{k}\|^{2}&\leq&3[(\eta-\beta)t_{k}-\delta]^{2}\|z^{k+1}-z^{k}\|^{2}\\ &&+{~}3\eta^{2}t_{k}^{2}\|d^{k}\|^{2}+3[\eta(r-1)-\delta]^{2}\|z^{k}\|^{2}.\end{array} Since all the terms on the right-hand side of this inequality are summable due to ( 110 ), we obtain the last line of ( 183 ). Finally, by Young’s inequality, we have \begin{array}[]{lcl}2\langle z^{k},x^{k+1}-x^{\star}\rangle&=&2\langle w^{k},x^{k}-x^{\star}\rangle+2\langle z^{k}-w^{k},x^{k}-x^{\star}\rangle+2\langle z^{k},x^{k+1}-x^{k}\rangle\\ &\geq&-2\rho\|w^{k}\|^{2}-2\|z^{k}-w^{k}\|\|x^{k}-x^{\star}\|-\|z^{k}\|^{2}-\|x^{k+1}-x^{k}\|^{2}\\ &\geq&-2\rho\|w^{k}\|^{2}-t_{k}^{2}\|z^{k}-w^{k}\|^{2}-\frac{\|x^{k}-x^{\star}\|^{2}}{t_{k}^{2}}\\ &&-{~}\|z^{k}\|^{2}-\|x^{k+1}-x^{k}\|^{2}.\end{array} Since r(\mu-1)\|x^{k}-x^{\star}\|^{2}\leq\hat{\mathcal{L}}_{k}\leq\Omega\hat{\mathcal{L}}_{0}\leq\Omega\mathcal{R}_{0}^{2} as in the proof of Theorem 7.4 , the last inequality implies ( 184 ). \square F.16 The Proof of Theorem 7.6 From the proof of Theorem 7.4 , we have r(\mu-1)\|x^{k}-x^{\star}\|^{2}\leq\mathcal{L}_{k}\leq\Omega\mathcal{R}_{0}^{2} , we conclude that \{x^{k}\} is bounded, and hence, it has a cluster point. Let x^{*} be a cluster point of x^{k} and x^{k_{i}} be a subsequence converging to x^{*} . Similar to the proof of Theorem 7.3 , we have 0\in Fx^{*}+Tx^{*} (i.e. x^{*}\in\mathrm{zer}(\Phi) ). Our next step is to show that \lim_{k\to\infty}\|x^{k}-x^{\star}\|^{2} exists. Combining ( 181 ), ( 183 ), and ( 184 ), we can derive that \begin{array}[]{lcl}\hat{\mathcal{Q}}_{k+1}-\hat{\mathcal{Q}}_{k}&\leq&t_{k+1}^{2}\|p^{k}\|^{2}+2[(\eta-\beta)t_{k}-\delta]^{2}\|z^{k+1}-w^{k+1}\|^{2}\\ &&+{~}2\rho[(\eta-\beta)t_{k}-\delta](t_{k}-r-\mu)\|w^{k+1}-w^{k}\|^{2}\\ &&+{~}r[\eta(r-2)+\beta-\delta]\Big[2\rho\|w^{k}\|^{2}+\|z^{k}\|^{2}+\|x^{k+1}-x^{k}\|^{2}\\ &&\qquad\qquad\qquad\qquad\qquad+{~}t_{k}^{2}\|z^{k}-w^{k}\|^{2}+\frac{\Omega\mathcal{R}_{0}^{2}}{r(\mu-1)t_{k}^{2}}\Big]\\ &&+{~}2\mu\eta t_{k}|\langle\hat{w}^{k+1},x^{k+1}-x^{k}\rangle|+2\rho rt_{k+1}\eta_{k}\|w^{k+1}\|^{2}\\ &&+{~}2s^{3}_{k}|\langle z^{k}-w^{k},x^{k+1}-x^{k}\rangle|+2\hat{s}^{3}_{k}|\langle z^{k},x^{k+1}-x^{k}\rangle|,\end{array} where we have used \langle w^{k+1}-w^{k},x^{k+1}-x^{k}\rangle\geq-\rho\|w^{k+1}-w^{k}\|^{2} . It is clear that since s_{k}^{3}=\mathcal{O}\left(t_{k}\right) and \hat{s}^{3}_{k}=\mathcal{O}\left(t_{k}\right) , all the terms on the right-hand sides of the last inequality are summable due to ( 110 ) and ( 183 ). Moreover, we have \hat{\mathcal{Q}}_{k}\geq 0 . Applying ( 9 , Lemma 5.31) , we can conclude that the limit \lim_{k\to\infty}\hat{\mathcal{Q}}_{k} exists. Similar to ( 165 ), we have \lim_{k\to\infty}|t_{k}\langle w^{k},x^{k}-x^{\star}\rangle|=0 . Since \hat{\mathcal{Q}}_{k}=\mathcal{A}_{k}+2rt_{k}\eta_{k-1}\big[\langle w^{k},x^{k}-x^{\star}\rangle+\rho\|w^{k}\|^{2}\big] , this limit also implies that \lim_{k\to\infty}\mathcal{A}_{k} exists. From the definition of \mathcal{A}_{k} in ( 180 ), we can expand it as \begin{array}[]{lcl}\mathcal{A}_{k}&=&(r^{2}+\mu r)\|x^{k}-x^{\star}\|^{2}+t_{k}^{2}\|y^{k}-x^{k}\|^{2}+t_{k}^{2}\eta_{k-1}^{2}\|z^{k}-w^{k}\|^{2}\\ &&+{~}2rt_{k}\langle x^{k}-x^{\star},y^{k}-x^{k}\rangle+2rt_{k}\eta_{k-1}\langle z^{k}-w^{k},x^{k}-x^{\star}\rangle\\ &&+{~}2t_{k}^{2}\eta_{k-1}\langle y^{k}-x^{k},z^{k}-w^{k}\rangle.\end{array} It is not hard to show that the limits of the last five terms are all zeros. Combining these facts and the existence of \lim_{k\to\infty}\mathcal{A}_{k} , we conclude that \lim_{k\to\infty}\|x^{k}-x^{\star}\|^{2} exists. Thus we have \lim_{k\to\infty}x^{k}=x^{\star}\in\mathrm{zer}(\Phi) due to the well-known Opial’s lemma. Finally, since \lim_{k\to\infty}\|y^{k}-x^{k}\|^{2}=0 , we also have \lim_{k\to\infty}y^{k}=x^{\star}\in\mathrm{zer}(\Phi) . \square