Efficiency Boost in Decentralized Optimization: Reimagining Neighborhood Aggregation with Minimal Overhead Conference: Proceedings of the 34th ACM International Conference on Information and Knowledge Management; November 10–14, 2025; Seoul, Republic of Korea Proceedings of the 34th ACM Internationa
Durgesh Kalwar, Mayank Baranwal, Harshad Khadilkar · 2025
Read Efficiency Boost in Decentralized Optimization: Reimagining Neighborhood Aggregation with Minimal Overhead Conference: Proceedings of the 34th ACM International Conference on Information and Knowledge Management; November 10–14, 2025; Seoul, Republic of Korea Proceedings of the 34th ACM Internationa by Durgesh Kalwar, Mayank Baranwal, Harshad Khadilkar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22174.txt.
Introduction
When learning is spread across many machines with no central server, the machines must blend their neighbors' updates wisely. Durgesh Kalwar, Mayank Baranwal, and Harshad Khadilkar present DYNAWEIGHT in this 2025 paper, a blending scheme for decentralized learning. Fixed blending weights, like the classic Metropolis rule, treat every neighbor the same forever. That wastes the fact that some neighbors hold more diverse data than others.
DYNAWEIGHT instead assigns weights dynamically, based on each neighbor's relative loss on its own data. Neighbors whose information differs more get favored, which matters most when data varies a lot between machines. The scheme adds little communication or memory cost. It also works as a plug-in on top of any local training algorithm. Experiments on three image data sets, across server counts and network shapes, showed clear gains in training speed. The authors present dynamic weighting as a cheap upgrade for decentralized learning systems.
Catalog record
| CREATOR | Durgesh Kalwar, Mayank Baranwal, Harshad Khadilkar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-22174
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficiency Boost in Decentralized Optimization: Reimagining Neighborhood Aggregation with Minimal Overhead Conference: Proceedings of the 34th ACM International Conference on Information and Knowledge Management; November 10–14, 2025; Seoul, Republic of Korea Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM ’25), November 10–14, 2025, Seoul, Republic of Korea DOI: 10.1145/3746252.3761288 ISBN: 979-8-4007-2040-6/2025/11 CCS: Computing methodologies Distributed algorithms CCS: Computing methodologies Multi-agent systems CCS: Computing methodologies Cooperation and coordination Durgesh Kalwar, Mayank Baranwal, Harshad Khadilkar Abstract . In today’s data-sensitive landscape, distributed learning emerges as a vital tool, not only fortifying privacy measures but also streamlining computational operations. This becomes especially crucial within fully decentralized infrastructures where local processing is imperative due to the absence of centralized aggregation. Here, we introduce DYNAWEIGHT , a novel framework to information aggregation in multi-agent networks. DYNAWEIGHT offers substantial acceleration in decentralized learning with minimal additional communication and memory overhead. Unlike traditional static weight assignments, such as Metropolis weights, DYNAWEIGHT dynamically allocates weights to neighboring servers based on their relative losses on local datasets. Consequently, it favors servers possessing diverse information, particularly in scenarios of substantial data heterogeneity. Our experiments on various datasets MNIST, CIFAR10, and CIFAR100 incorporating various server counts and graph topologies, demonstrate notable enhancements in training speeds. Notably, DYNAWEIGHT functions as an aggregation scheme compatible with any underlying server-level optimization algorithm, underscoring its versatility and potential for widespread integration. 1. Introduction Distributed and decentralized optimization 1 1 1 In the literature, “distributed” and “decentralized” are frequently used interchangeably. While they differ slightly, both terms denote a setup lacking a central server or compute. Each server processes private data locally, with information exchanged between neighboring servers. techniques have emerged as pivotal strategies in tackling large-scale computational challenges across diverse domains ( Li et al. 2019 ; Elgabli et al. 2020 ; Koloskova et al. 2020 ) . In distributed optimization, the computational task is divided among multiple processing units or nodes, each contributing to the overall solution. This approach offers advantages such as scalability, fault tolerance, and parallelism. Decentralized optimization takes this concept further by removing the need for a central coordinator, allowing individual nodes to collaboratively reach a solution through local interactions. The importance of decentralized learning is underscored by their applicability in various fields. In machine learning and artificial intelligence, these techniques enable the training of complex models on massive datasets distributed across different locations, while preserving data privacy and security ( Xin et al. 2020 ) . Decentralized optimization finds applications in decentralized control systems ( Inalhan et al. 2002 ) , sensor networks ( Rabbat and Nowak 2004 ) , and multi-agent systems ( Baranwal et al. 2020 ) , where local decision-making leads to emergent global behavior. Moreover, in fields like logistics and transportation, these approaches facilitate real-time decision-making in dynamic environments, optimizing resource allocation and routing. However, despite their immense potential, distributed and decentralized optimization methods come with several challenges. One primary concern is communication overhead, as exchanging information between nodes can lead to bottlenecks and increased latency ( Ballotta et al. 2023 ) . Ensuring convergence to a global optimum in decentralized settings without a central coordinator poses another challenge, requiring sophisticated algorithms that balance exploration and exploitation ( Taylor et al. 2011 ) . Moreover, maintaining consistency and synchronization across distributed systems in the presence of failures or network delays is a non-trivial task ( Hsieh et al. 2022 ) . Additionally, privacy and security concerns arise when dealing with sensitive data distributed across multiple nodes, necessitating robust encryption and access control mechanisms ( Zhang and Wang 2018 ) . Addressing these challenges requires a multidisciplinary approach, integrating techniques from optimization theory, distributed systems, machine learning, and cryptography. Advances in communication protocols, consensus algorithms, and optimization methods tailored for distributed and decentralized environments are crucial. Furthermore, developing robust frameworks for evaluating the performance and scalability of these techniques across diverse applications is essential for their widespread adoption. In this paper, we introduce DYNAWEIGHT , an adaptive weighting framework designed to address data heterogeneity across servers in decentralized learning. Unlike static weighting schemes that rely solely on network connectivity for server weighting during information aggregation, DYNAWEIGHT utilizes server performance on the datasets of neighboring servers (without explicitly sharing the data). We empirically demonstrate that DYNAWEIGHT achieves faster convergence compared to traditional non-adaptive weighting schemes while remaining computationally efficient. Moreover, DYNAWEIGHT is a versatile weighting scheme that can be integrated with any suitable optimization algorithm. 2. Distributed Learning in Multi-Agent Systems We consider a network comprising N interconnected servers (agents) over a communication graph \mathcal{G}(V,\mathcal{E}) collaboratively training a deep neural network in a decentralized fashion. Each server i possesses its private dataset \mathcal{D}_{i}\coloneqq(\mathbf{x}_{i}^{q},y_{i}^{q})_{q=1}^{n_{i}} comprising n_{i} samples and maintains its individual network copy with parameters \theta_{i}\in\mathbb{R}^{d} . While a server i can exchange parameter vectors with its neighbors \mathcal{N}_{i} , the confidentiality of local data is preserved. Homogeneity of data distribution across servers is not mandated. It’s crucial to note that this setup fundamentally differs from conventional federated learning (FL), which resembles a centralized framework where a central server communicates with a fraction of edge devices for information exchange. The collective objective is to minimize the team objective function: (1) \displaystyle F(\theta)\coloneqq\sum\limits_{i=1}^{N}\frac{1}{n_{i}}\sum\limits_{q=1}^{n_{i}}\ell\left(\texttt{NN}_{\theta}(\mathbf{x}_{i}^{q}),y_{i}^{q}\right), where \texttt{NN}_{\theta}(\mathbf{x}_{i}^{q}) denotes the output of the neural network of the i^{\text{th}} server given input \mathbf{x}_{i}^{q} , and \ell(\cdot) represents any sample-level loss function. However, ( 1 ) depicts a centralized framework for minimizing the cumulative loss function, assuming that the network parameters \theta are shared across servers. A distributed version of the above problem can instead be expressed as: \displaystyle\min\limits_{\{\theta_{1},\theta_{2},\dots,\theta_{N}\}} \displaystyle\sum\limits_{i=1}^{N}\frac{1}{n_{i}}\sum\limits_{q=1}^{n_{i}}\ell\left(\texttt{NN}_{\theta_{i}}(\mathbf{x}_{i}^{q}),y_{i}^{q}\right) (2) s.t. \displaystyle\theta_{1}=\theta_{2}=\dots=\theta_{N}. However, in the absence of a centralized server, servers aim to execute a consensus algorithm to ensure convergence of their parameters to a common vector. A typical decentralized optimization algorithm executes following steps at k^{\text{th}} iteration of each server: (Gradient step) \displaystyle\theta_{i}^{k+\frac{1}{2}} \displaystyle\leftarrow\theta_{i}^{k}-\eta\frac{1}{n_{i}}\sum\limits_{q=1}^{n_{i}}\nabla_{\theta_{i}}\ell\left(\texttt{NN}_{\theta_{i}}(\mathbf{x}_{i}^{q}),y_{i}^{q}\right), (Consensus) \displaystyle\theta_{i}^{k+1} \displaystyle\leftarrow\sum\limits_{j\in\mathcal{N}_{i}}w_{ij}\theta_{j}^{k+\frac{1}{2}}, where \eta>0 denotes the step-size, while weights w_{ij}\geq 0 , with \sum_{j\in\mathcal{N}i}w{ij}=1 , signify the relative importance of the i^{\text{th}} server and its neighbors in updating the parameter vector \theta_{i} . It’s important to recall that during the gradient step, each server computes the gradient of the loss function solely based on its local (private) data and executes a gradient descent. Furthermore, in practical implementations, this update process can be further accelerated by utilizing momentum-based optimization algorithms, such as AdaGrad ( Chen et al. 2023 ) or Adam ( Nazari et al. 2022 ) . The next phase involves executing a consensus update among neighboring servers utilizing weights w_{ij} . There exist various methods for selecting these consensus weights. The prevalent approach involves employing equal weighting across all servers, termed as the simple weighting scheme. Alternatively, another viable option is the utilization of Metropolis weights, closely associated with the Metropolis-Hastings algorithm. The Metropolis weights are given as: \displaystyle w_{ij}=\left\{\begin{array}[]{cl}\frac{1}{1+\max{(d_{i},d_{j})}},&\text{if}\ j\in\mathcal{N}_{i},i\neq j\\ 1-\sum_{l\in\mathcal{N}_{i}}w_{il},&\text{if}\ i=j\\ 0,&\text{else}\end{array}\right., where d_{i} represents the degree of i^{\text{th}} -server. It is worth noting that Metropolis weights can be derived through local information exchange, obviating the need for servers to possess knowledge of the entire communication topology. Heterogeneous data : A notable challenge with static (fixed) weighting schemes lies in their inability to accommodate data heterogeneity across servers. To