Zeus : An Efficient GPU Optimization Method Integrating PSO, BFGS, and Automatic Differentiation
Dominik Soós 1 , Marc Paterno 1 2 , Desh Ranjan 1 , Mohammad Zubair 1 · 2026
Read Zeus : An Efficient GPU Optimization Method Integrating PSO, BFGS, and Automatic Differentiation by Dominik Soós 1 , Marc Paterno 1 2 , Desh Ranjan 1 , Mohammad Zubair 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28770.txt.
Introduction
Soos and co-authors introduce ZEUS, a GPU optimization method that joins particle swarm optimization, BFGS, and automatic differentiation. High-dimensional, non-convex problems are hard to optimize well. ZEUS first uses particle swarm search to find good starting points. It then runs BFGS in parallel from those points on GPUs. Automatic differentiation supplies the gradients BFGS needs, so users need not derive them by hand. A few swarm iterations improved global convergence in their tests. The implementation is open source.
Catalog record
| CREATOR | Dominik Soós 1 , Marc Paterno 1 2 , Desh Ranjan 1 , Mohammad Zubair 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48956 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.28770
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Zeus : An Efficient GPU Optimization Method Integrating PSO, BFGS, and Automatic Differentiation Dominik Soós 1 , Marc Paterno 1 2 , Desh Ranjan 1 , Mohammad Zubair 1 Abstract We introduce a novel, efficient computational method, Zeus , for numerical optimization, and provide an open-source implementation. It has four key ingredients: (1) particle swarm optimization (PSO), (2) the use of the Broyden-Fletcher-Goldfarb-Shanno (BFGS) method, (3) automatic differentiation (AD), and (4) GPUs. Our approach addresses the computational challenges inherent in high-dimensional, non-convex optimization problems. In the first phase of the algorithm, we get a potentially good set of starting points using PSO. Thereafter, we run BFGS independently in parallel from these starting points. BFGS is one of the best-performing algorithms for numerical optimization. However, it requires the gradient of the function being optimized. Zeus integrates automatic differentiation into BFGS thus avoiding the need for the user to calculate derivatives explicitly. The use of GPUs allows Zeus to speed up the calculations substantially. We carry out systematic studies to explore the trade-offs between the number of PSO iterations taken, starting points, and BFGS iteration depth. We show that a handful of iterations of PSO can improve global convergence when combined with BFGS. We also present performance studies using common test functions. The source code can be found at https://github.com/fnal-numerics/global-optimizer-gpu . I Introduction A wide range of domains make use of numerical optimization, including particle physics simulations, machine learning, and financial modeling. Multidimensional non-convex global optimization can be difficult due to a number of reasons. One of these reasons is that the solution space grows exponentially with the dimensionality of the function being optimized. The classical sequential numerical optimization algorithms start from an initial “guess” and then use the function gradient to move in the direction that reduces the value of the objective function. However, such conventional optimization techniques often fail to navigate complex landscapes with many local minima or narrow valleys where the gradient is close to zero. Purely gradient-based methods like stochastic gradient descent [ 24 ] , mini-batching [ 26 , 39 ] , and stochastic variance-reduced gradient [ 35 ] get stuck in flat regions or local minima when dealing with non-convex landscapes. To tackle difficult landscapes with many local minima, the particle swarm optimization (PSO) method has been widely adopted for its ability to handle global searches by exploring multiple regions of the hyperspace. However, PSO by itself struggles with problems with flat regions. The use of graphical processing units (GPUs) has become popular in parallel computing, as they can run thousands of computations simultaneously. Parallel computing approaches have been proposed that distribute the computations of the gradients [ 43 ] . This paper investigates how parallel computing on GPUs can improve non-convex global optimization strategies. We propose a novel, GPU-based global optimization algorithm named Zeus that combines the particle swarm optimization (PSO) with popular quasi-Newton optimization (BFGS) [ 7 , 17 , 18 , 37 ] . Additionally, the method provides a built-in forward-mode automatic differentiation (AD) library, thus avoiding the explicit calculation of gradients needed in the BFGS method. We propose an approach that initializes the optimization from a multitude of random starting points in multidimensional solution space, utilizing GPUs to concurrently execute the BFGS optimization algorithm using forward-mode AD. Our contributions are summarized below. • We provide a parallel multistart optimization algorithm with automatic differentiation that combines the advantages of the PSO and BFGS methods as well as automatic computation of AD for accuracy and ease of use. • Our C++ CUDA implementation of the algorithm provides a 10- to 100-fold speedup compared to our serial implementation. • We provide insight into the trade-off between the number of PSO steps taken and the number of optimizations run concurrently through our extensive studies. Our extensive experiments not only confirm that a handful of PSO iterations can boost global convergence rates, as shown in Figure 3 , but also reveal hyperparameter trade-offs for different functions. We further discuss limitations like the failure condition on functions like the Ackley function with discontinuous gradients. We outline future plans to reduce computational complexity in the parts of the algorithm that dominate runtime. The rest of the paper is organized as follows: In the next section, we present some background and previous related work. In Section III , we present the sequential Zeus algorithm. Section IV presents the parallel version of the algorithm. Section V presents our experiments and results. The last section discusses our results and observations as well as future work. In the next section, we present the background information and the related work to our approach, where we discuss each component of the algorithm, including multistart and swarm-based algorithms, BFGS with AD, and GPUs. II Background and Related work Stochastic multistart and evolutionary algorithms, where the optimizer starts from many points, are not a new concept. The earliest strategies date back to the 1970s, the first multistart method, which proposed the idea of launching many optimizations from many different points to improve the chance of finding the global optimum [ 19 , 5 ] and was later implemented as GLOBAL [ 12 ] . The PSO method is widely used in the literature to improve global convergence. [ 27 ] combined PSO and BFGS, but run on a CPU and do not leverage automatic differentiation or parallel GPUs. Other approaches used a two-phase global-local scheme, but neither used PSO nor BFGS, nor AD [ 16 ] . Similarly, [ 3 ] implemented a GPU-based global search by space-filling, but no PSO, BFGS, or AD was used. Our unique combination is the first C++ CUDA-based library to integrate: • global search with PSO • local refinement via BFGS • automatic differentiation for gradient calculation • massively parallel GPU The well-established BFGS algorithm that originates from the works of Broyden [ 7 ] , Fletcher [ 17 ] , Goldfarb [ 18 ] , and Shanno [ 37 ] uses quasi-Newton updates to approximate the Hessian and typically converges in fewer iterations than first-order methods. [ 33 ] showed that BFGS combined with a Wolfe line search enjoys global convergence properties even in moderately high dimensions, further motivating its application to non-convex problems. A major challenge with gradient-based methods, however, is the need for accurate derivative information. Manual derivation is error-prone and often impractical for complex, high-dimensional functions. This is where AD becomes invaluable. II-A Automatic Differentiation In recent years, AD has gained widespread attention, not only in machine learning but also in scientific computing [ 21 , 4 , 8 , 29 ] . AD is attractive because it provides more accurate derivatives with minimal overhead without the need for the user to provide manual derivation of the gradient of the objective function. For instance, [ 44 ] implemented a forward-mode AD technique on GPUs for computational fluid dynamics, achieving performance close to the hardware’s peak throughput with higher accuracy than manual derivative calculation. Likewise, [ 20 ] used AD for 2D/3D registration on GPUs, addressing challenges in medical imaging. Recent advances have further accelerated AD on GPUs using modern compiler techniques. Google Brain’s JAX framework [ 6 ] offers a user-friendly, NumPy-like interface that supports just-in-time, or