A Double Inertial Forward-Backward Splitting Algorithm With Applications to Regression and Classification Problems

İrfan IŞIK a , Ibrahim KARAHAN b and Okan ERKAYMAZ c · 2025

Read A Double Inertial Forward-Backward Splitting Algorithm With Applications to Regression and Classification Problems by İrfan IŞIK a , Ibrahim KARAHAN b and Okan ERKAYMAZ c (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03794.txt.

Introduction

İrfan Işık, Ibrahim Karahan, and Okan Erkaymaz present an improved splitting algorithm for hard optimization problems, in a 2025 paper. Many tasks in fitting and sorting data reduce to finding a point where two operators cancel out. Splitting methods attack that point by alternating steps, and adding momentum-like inertia can speed them up.

Their version uses two inertia terms at once. Under standard assumptions, they prove the method settles weakly toward a solution in the general space they study. Tests on regression and classification problems compared it with established rivals from the literature. The new method came out ahead in those trials. The authors offer it as a stronger default for a family of problems that appears across applied math.

Catalog record

CREATORİrfan IŞIK a , Ibrahim KARAHAN b and Okan ERKAYMAZ c
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-03794

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Double Inertial Forward-Backward Splitting Algorithm With Applications to Regression and Classification Problems İrfan IŞIK a , Ibrahim KARAHAN b and Okan ERKAYMAZ c Abstract This paper presents an improved forward-backward splitting algorithm with two inertial parameters. It aims to find a point in the real Hilbert space at which the sum of a co-coercive operator and a maximal monotone operator vanishes. Under standard assumptions, our proposed algorithm demonstrates weak convergence. We present numerous experimental results to demonstrate the behavior of the developed algorithm by comparing it with existing algorithms in the literature for regression and data classification problems. Furthermore, these implementations suggest our proposed algorithm yields superior outcomes when benchmarked against other relevant algorithms in existing literature. keywords Double inertial, monotone inclusion problem, regression, classification, weak convergence † † affiliation: a,b Department of Mathematics, Faculty of Science, Erzurum Technical University, Erzurum, 25100, Türkiye; c Department of Computer Engineering, Naval Academy National Defense University, Istanbul, Türkiye. 1 Introduction Let assume that H is a real Hilbert Space, A:H\rightarrow H is a co-coercive operator and B:H\rightarrow 2^{H} is a maximal monotone operator. In this study, we consider the monotone inclusion problem, which is to find z\in H such that 0\in\left(Az+Bz\right)\text{.} (1) Monotone inclusion problems serve as a fundamental tool in addressing a wide range of problems on machine-learning, linear inverse problems, image processing, variational inequality problems, convex minimization problems, convex-concave saddle point problems and equilibrium problems, see e.g. [ 11 , 19 , 21 , 22 , 35 , 38 ] . More specifically, it finds utility across various scientific domains, including but not limited to statistical regression, machine learning, image and signal processing see, e.g. [ 9 , 32 , 33 , 37 , 16 , 10 , 15 ] . Recognizing its significance, numerous authors have explored suitable algorithms to find solutions for it. The most popular algorithm to solve ( 1 ) is the forward-backward splitting algorithm given by Lions and Mercier [ 20 ] : x_{n+1}=\left(I+\lambda_{n}A\right)^{-1}\left(I-\lambda_{n}B\right)x_{n}\text{ \ for all }n\in\mathbb{N} (2) where A and B are monotone operators and \lambda_{n} is step size. In 2000, Tseng [ 38 ] introduced a two-step strategy as part of the forward-backward algorithm, necessitating adjustments to equation ( 1 ). Tseng’s approach is delineated by the following way with \rho,k\in(0,1) , \alpha>0 and \theta_{1}\in H , \displaystyle y_{n} \displaystyle= \displaystyle(I+\gamma_{n}A)^{-1}(I-\gamma_{n}B)x_{n} \displaystyle x_{n+1} \displaystyle= \displaystyle x_{n}-\gamma_{n}(Bx_{n}-By_{n}) where \gamma_{n}=\alpha k^{m_{n}} and m_{n} signify the smallest non-negative integer that satisfies the following condition: \gamma_{n}\left\|\nabla f(\theta_{n+1})-\nabla f(\theta_{n})\right\|\leq\rho\left\|\theta_{n+1}-\theta_{n}\right\|\text{.} (3) It has been demonstrated that \left\{x_{n}\right\} converges weakly to a solution of \left(A+B\right)^{-1}(0) . This algorithm, called forward–backward–forward, is one of the most basic algorithms. In 2009, Beck and Teboulle defined the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA) as follows: Let \alpha_{1}=1 and \rho_{0}=\rho_{1}\in H , y_{n}=x_{n}+\beta_{n}(x_{n}-x_{n-1})\text{,}\\ x_{n+1}=\left(I+\frac{1}{L}\partial f\right)^{-1}(I-\frac{1}{L}\nabla f)(y_{n})\text{,} (4) where \beta_{n}=\frac{t_{n}-1}{t_{n+1}} , t_{n+1}=\frac{1+\sqrt{1+4t_{n}^{2}}}{2} , and L is Lipschitz constant of \nabla f . They demonstrated the convergence rate and used it for image reconstruction. Following these studies, new algorithms were defined in order to increase the running speed of existing algorithms. Adding an inertial extropolation step to algorithms, which was was first suggested by Polyak [ 27 ] in 1964, is one way to increase the speed of algorithms. It was later used by Nesterov [ 25 ] in 1983 as an inertial step to solve minimization problems. In the past, many researchers have included an inertial extrapolation step in their algorithms, see, e.g., [ 5 , 26 , 36 , 2 , 3 , 31 , 1 ] . Nevertheless, subsequent to these favorable advancements, Poon and Liang [ 29 , 30 ] highlighted certain drawbacks associated with optimization methods that incorporate only a single-step inertial extrapolation. Although the inertial term has often been employed to accelerate the convergence of algorithms, it is demonstrated in [ [ 30 ] , Section 4] that the Douglas–Rachford splitting scheme without any inertial component outperformed its counterpart that incorporates a single-step inertial extrapolation. However, Polyak [ 28 ] , who argues that increasing the number of inertial steps in the algorithms will increase the current convergence speed of the algorithms, has not established both the speedup and convergence states of multi-step inertial algorithms. In 2018, Dong et al. [ 12 ] presented the double inertial Mann algorithm in their study. In this study, they also provided the convergence of their suggested algorithm under some mild conditions. Let \left\{\alpha_{n}\right\}\subset\left[0,\alpha\right] and \left\{\beta_{n}\right\}\subset\left[0,\beta\right] are nondecreasing with \alpha_{1}=\beta_{1}=0 and \alpha,\beta\in\left[0,1\right) . In addition T be a nonexpansive mapping. The algorithm given by Dong et al. is defined as follows: \displaystyle z_{n} \displaystyle= \displaystyle v_{n}+\alpha_{n}\left(v_{n}-v_{n-1}\right) \displaystyle y_{n} \displaystyle= \displaystyle v_{n}+\beta_{n}\left(v_{n}-v_{n-1}\right) \displaystyle v_{n+1} \displaystyle= \displaystyle\left(1-\gamma_{n}\right)z_{n}+\gamma_{n}T\left(y_{n}\right). Not long after, in 2022 Iyiola and Shehu [ 17 ] introduced the new two-step inertial Douglas-Rachford splitting method in their work. This approach essentially integrates the two-step inertial proximal point algorithm with the Douglas-Rachford splitting algorithm, (see [ 13 , 20 ] ). Let given 0\leq\alpha<\frac{1}{3} and \frac{3\alpha-1}{3+4\alpha}<\beta\leq 0 . Let \gamma_{0}=\theta_{0}=\theta_{-1}\in H , \rho>0 and n>0 . The two step inertial Douglas-Rachford splitting method is introduced by the following way: \displaystyle\theta_{n+1} \displaystyle= \displaystyle J_{n}^{A}\left(2J_{n}^{B}-I\right)\left(\gamma_{n}\right)+\left(I-J_{n}^{B}\right)\left(\gamma_{n}\right) \displaystyle\gamma_{n+1} \displaystyle= \displaystyle\theta_{n+1}+\alpha(\theta_{n+1}-\theta_{n})+\beta(\theta_{n}-\theta_{n-1}) where A,B:H\rightarrow 2^{H} be maximal monotone operator on H . In 2023 Suantai et al. [ 34 ] , inspired by previous work on solving ( 1 ), proposed the following Double inertial forward–backward–forward algorithm for convex minimization problems. Let us denote \Gamma by \left(A+B\right)^{-1}\left(0\right)\neq\emptyset and let v_{1}>0 , \rho\in(0,1) , \left\{\alpha_{n}\right\} and \left\{\beta_{n}\right\} real positive sequences. Again let \theta_{0} , \theta_{1}\in H and \left\{\lambda_{n}\right\} be nonnegative sequence such that \sum_{n=1}^{\infty}\lambda_{n}<\infty . Given \theta_{n} , \theta_{n-1} and compute \displaystyle p_{n} \displaystyle= \displaystyle\theta_{n}+\alpha_{n}(\theta_{n}-\theta_{n-1})\text{,} \displaystyle r_{n} \displaystyle= \displaystyle p_{n}+\beta_{n}(p_{n}-p_{n-1})\text{,} \displaystyle s_{n} \displaystyle= \displaystyle(I+v_{n}B)^{-1}(I-v_{n}A)r_{n}\text{.} If s_{n}=r_{n} then stop and s_{n}\in\Gamma . Else \theta_{n+1}=s_{n}-v_{n}(As_{n}-Ar_{n})\text{.} (5) Here v_{n+1}=\left\{\begin{array}[]{c}\min\left\{\frac{\rho\left\|r_{n}-s_{n}\right\|}{\left\|Ar_{n}-As_{n}\right\|},v_{n}+\lambda_{n}\right\}\text{ if }\left\|Ar_{n}-As_{n}\right\|\neq 0\text{;}\\ v_{n}+\lambda_{n}\text{ otherwise}\end{array}\right. (6) and the algorithm is updated again from the beginning. For Problem ( 1 ), it is shown in [ [ 30 ] , Chapter 4] that the Douglas–Rachford splitting method equipped with a two-step inertial extrapolation exhibits faster convergence than its variant utilizing a single-step inertial scheme. Inspired and motivated by the above studies, our paper offers the following significant contributions: We propose a novel splitting method, namely a forward-backward scheme enhanced with a two-step inertial extrapolation technique, and provide a comprehensive investigation of its weak convergence properties. We implement comprehensive computational experiments to illustrate the efficient of the proposed algorithm and also exhibit its superior performance in comparison to the certain algorithms existed in the literature. 2 Preliminaries In this section, we give the basic concepts and definitions which we need to prove the main results. We use the notations p_{k}\rightharpoonup p and p_{k}\to p in H to indicate weak and strong convergence of the sequence \{p_{k}\}\subset H to p\in H , respectively, as k\to\infty . For all p,y\in H and any \alpha\in\mathbb{R} , the following identities and inequality hold: \begin{array}[]{ll}\text{i)}&\|p+y\|^{2}\leq\|p\|^{2}+2\langle y,p+y\rangle\\ \text{ii)}&\|p+y\|^{2}=\|p\|^{2}+2\langle p,y\rangle+\|y\|^{2}\\ \text{iii)}&\|\alpha p+(1-\alpha)y\|^{2}=\alpha\|p\|^{2}+(1-\alpha)\|y\|^{2}-\alpha(1-\alpha)\|p-y\|^{2}.\end{array} Let \mathcal{T}:H\rightarrow H be an operator on a real Hilbert space H . The mapping \mathcal{T} is said to satisfy the following properties: (i) \mathcal{T} is called nonexpansive if \|\mathcal{T}p-\mathcal{T}y\|\leq\|p-y\|,\quad\forall p,y\in H. (7) (ii) \mathcal{T} is said to be firmly nonexpansive provided that \|\mathcal{T}p-\mathcal{T}y\|^{2}\leq\|p-y\|^{2}-\|(I-\mathcal{T})p-(I-\mathcal{T})y\|^{2},\quad\forall p,y\in H, (8) which is equivalent to \|\mathcal{T}p-\mathcal{T}y\|^{2}\leq\langle p-y,\mathcal{T}p-\mathcal{T}y\rangle. (9)

Read the complete text