Momentum-based Accelerated Algorithm for Distributed Optimization under Sector-Bound Nonlinearity Journal: Journal of The Franklin Institute
Mohammadreza Doostmohammadian, Hamid R. Rabiee · 2025
Read Momentum-based Accelerated Algorithm for Distributed Optimization under Sector-Bound Nonlinearity Journal: Journal of The Franklin Institute by Mohammadreza Doostmohammadian, Hamid R. Rabiee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-22855.txt.
Introduction
Mohammadreza Doostmohammadian and Hamid Rabiee designed a faster learning algorithm for networks of computers that must agree on a shared answer. In distributed optimization, each node sees part of the data and the group settles on one model. Real communication links distort the shared values through clipping or coarse quantization, which older algorithms rarely handle. Their method adds momentum, a heavy-ball push in the direction things were already moving, plus a gradient-tracking step.
Using perturbation theory and spectrum analysis, they prove the method still converges when links apply any sign-preserving sector-bound distortion, even for locally non-convex costs. The network design also tolerates directed, time-varying links, covering failures and dropped packets. Tests support the theory.
Catalog record
| CREATOR | Mohammadreza Doostmohammadian, Hamid R. Rabiee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-22855
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Momentum-based Accelerated Algorithm for Distributed Optimization under Sector-Bound Nonlinearity Journal: Journal of The Franklin Institute Mohammadreza Doostmohammadian, Hamid R. Rabiee Abstract Distributed optimization advances centralized machine learning methods by enabling parallel and decentralized learning processes over a network of computing nodes. This work provides an accelerated consensus-based distributed algorithm for locally non-convex optimization using the gradient-tracking technique. The proposed algorithm (i) improves the convergence rate by adding momentum towards the optimal state using the heavy-ball method, while (ii) addressing general sector-bound nonlinearities over the information-sharing network. The link nonlinearity includes any sign-preserving odd sector-bound mapping, for example, log-scale data quantization or clipping in practical applications. For admissible momentum and gradient-tracking parameters, using perturbation theory and eigen-spectrum analysis, we prove convergence even in the presence of sector-bound nonlinearity and for locally non-convex cost functions. Further, in contrast to most existing weight-stochastic algorithms, we adopt weight-balanced (WB) network design. This WB design and perturbation-based analysis allow to handle dynamic directed network of agents to address possible time-varying setups due to link failures or packet drops. † † graphicalabstract: † † highlights: Introducing a new accelerated algorithm for distributed optimization based on the gradient-tracking technique. Addressing general sector-bound nonlinearities in the information-sharing network, including sign-preserving odd mappings such as log-scale data quantization and clipping. Proposing a weight-balanced network structure, contrasting with existing weight-stochastic algorithms, to accommodate time-varying conditions such as link failures and packet drops. 1 Introduction Decentralized methods for data mining are of interest in general machine learning [ 1 ] , signal processing [ 2 ] , and control [ 3 ] applications. The idea is to parallelize and distribute the learning process and computation over a network of agents. This is motivated by recent advances in intelligent transportation systems (ITS) [ 4 , 5 ] , Internet-of-Things (IoT) [ 6 , 7 ] , cloud computing [ 8 ] , and distributed cyber-physical-systems (CPS) [ 9 ] . In these setups, data samples are distributed across multiple agents/nodes, and computation tasks are divided among them. The data sharing between agents only occurs through established communication links where, in reality, might be subject to some non-ideal conditions, for example, due to quantization [ 10 , 11 , 12 ] and saturation/clipping [ 13 ] . These nonlinearities make the existing linear methods impractical and inefficient. Such nonlinearities are required to be addressed in the optimization algorithm, otherwise, it may cause large optimality gap and even divergence in practice. Another concern is the convergence rate in real-world applications to reduce the residual as fast as possible. Moreover, in mobile multi-agent networks, the communications and linkings might change with data-sharing channels coming and going between different computing nodes. The change in the network might also be due to packet drops [ 14 ] . This necessitates algorithms adaptable to time-varying network structures. Addressing non-ideal data-sharing and dynamic network topology while improving the rate of convergence is both challenging and of great interest in real-world large-scale distributed setups. The existing literature on distributed optimization and machine learning rarely addresses the nonlinearity of the links over dynamic networks. Most existing literature on distributed optimisation (both constrained and unconstrained) considers ideal links [ 15 , 16 , 17 , 18 , 19 , 20 ] . These works give no possibility to address non-ideal linking, such as saturation and quantization. Similarly, the momentum-based strategies in [ 21 , 22 , 23 , 24 , 25 , 26 ] are designed over linear ideal channels, and the ones in [ 22 , 26 ] only converge over static time-invariant networks. There exist some accelerated algorithms that adopt sign-based nonlinearity to improve the convergence rate, e.g., see [ 27 ] and [ 28 , 29 , 30 ] for fixed/finite-time solutions. In contrast to these works, quantization and compression of data in networking setups are a must in real-world applications, which is not well-addressed in the mentioned literature. Quantization reduces the precision of data, thereby reducing the number of bits needed to represent it. By mapping a large set of input values to a smaller set of output values, quantization simplifies the data and makes it more compressible. Few works in the literature address distributed optimization under uniform quantization that results in certain optimality gap, for example, see [ 31 , 32 , 33 ] for distributed optimization and [ 34 , 35 ] for distributed machine learning. What is missing in the existing literature is a distributed algorithm to address log-scale quantization, clipping, and general sector-bound nonlinear models over data-sharing channels. The information sent over the real-world networks might be subject to different nonlinear mappings, such as clipping and quantization. Logarithmic quantization allocates more precision to smaller values and less to larger ones. This is beneficial because, from a system perspective, updates can have a wide dynamic range, but smaller updates are often more critical for fine-tuning the solution. By preserving the precision of smaller values better than larger ones, logarithmic quantization ensures that significant information is not lost [ 36 ] . To be more specific, for the training process in distributed optimization, the gradients can vary in magnitude. By logarithmic quantization one can represent a large range of values with relatively fewer bits; this helps to efficiently encode gradients of different magnitudes without severe loss of information, which is crucial for maintaining the accuracy of gradient-based methods. Example applications are in distributed resource allocation [ 37 ] , speech and audio encoding [ 38 ] , and image processing [ 36 ] . On the other hand, clipping and saturation are other concerns in real-world applications, for example, ramp-rate-limits for distributed optimization over energy networks [ 39 ] and signal clipping in cognitive radio networks [ 40 ] . In this paper, (i) a momentum-based distributed optimization algorithm is proposed. The momentum improves (and accelerates) the convergence rate toward the optimal point. The algorithm includes two parameters, one to tune the gradient tracking rate and the other to tune the momentum rate for faster convergence. We derive the trade-off relation for the admissible range of these two parameters and prove convergence under this relation using perturbation theory, eigen-spectrum analysis, and Lyapunov theory. (ii) Our proof analysis is irrespective of the convexity of the local cost functions. Therefore, the local costs might be non-convex similar to [ 18 , 22 ] . (iii) The proposed algorithm converges over time-varying WB directed networks. This finds application over volatile and unreliable networking setups, for example, in the presence of link failures, packet drops, congestion, or general non-ideal networking conditions. Most importantly, (iv) the proposed algorithm addresses general nonlinear mappings on the information-sharing channels representing, for example, log-scale quantization or clipping. This is not addressed in the existing literature, however, it exists in practical real-world distributed setups. Therefore, the existing methods may result in suboptimal solutions or a large optimality gap. Few mentioned works consider uniform quantization instead, which, in contrast to log-scale quantization, results in large optimality gap. Our distributed optimization setup shows convergence with the same level of optimality gap both in the presence and absence of sector-bound nonlinearity and advances the state-of-the-art in this regard. Applications of locally non-convex optimization, distributed support-vector-machine (SVM), and distributed linear