Formulations and scalability of neural network surrogates in nonlinear optimization problems

Robert B. Parker, Oscar Dowson, Nicole LoGiudice, Manuel Garcia, Russell Bent · 2024

Read Formulations and scalability of neural network surrogates in nonlinear optimization problems by Robert B. Parker, Oscar Dowson, Nicole LoGiudice, Manuel Garcia, Russell Bent (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11403.txt.

Introduction

Robert Parker and four co-authors compare in 2024 ways to embed trained neural networks inside nonlinear optimization problems without choking the solver. Engineers increasingly use networks as stand-ins for hard physics, then optimize around them. Three embeddings are tested: full-space, reduced-space, and gray-box.

The testbed is a power-grid problem where a network judges transient stability, with networks up to 590 million parameters. Full-space runs stall in the linear solver, and reduced-space runs stall in model building and derivatives. The gray-box route scales best, and a graphics card speeds it further. The findings, plus a new Julia package, give practitioners a practical path to optimizing around very large learned models.

Catalog record

CREATORRobert B. Parker, Oscar Dowson, Nicole LoGiudice, Manuel Garcia, Russell Bent
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-11403

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Formulations and scalability of neural network surrogates in nonlinear optimization problems Robert B. Parker, Oscar Dowson, Nicole LoGiudice, Manuel Garcia, Russell Bent Abstract We compare full-space, reduced-space, and gray-box formulations for representing trained neural networks in nonlinear optimization problems. We test these formulations on a transient stability-constrained, security-constrained alternating current optimal power flow (SCOPF) problem where the transient stability criteria are represented by a trained neural network surrogate. Optimization problems are implemented in JuMP and trained neural networks are embedded using a new Julia package: MathOptAI.jl. To study the bottlenecks of the three formulations, we use neural networks with up to 590 million trained parameters. The full-space formulation is bottlenecked by the linear solver used by the optimization algorithm, while the reduced-space formulation is bottlenecked by the algebraic modeling environment and derivative computations. The gray-box formulation is the most scalable and is capable of solving with the largest neural networks tested. It is bottlenecked by evaluation of the neural network’s outputs and their derivatives, which may be accelerated with a graphics processing unit (GPU). Leveraging the gray-box formulation and GPU acceleration, we solve our test problem with our largest neural network surrogate in 2.5 × \times the time required for a simpler SCOPF problem without the stability constraint. 1 Introduction Nonlinear local optimization is a powerful tool for engineers and operations researchers for its ability to handle accurate physical models, respect explicit constraints, and solve large-scale problems [ 2 ] . However, it is often the case that practitioners wish to include components that do not easily fit into the differentiable and algebraic frameworks of nonlinear optimization. Examples include when a mechanistic model is not available [ 21 ] , is time-consuming to simulate [ 7 ] , or renders the optimization problem too complicated to solve reliably [ 5 ] . A recent trend has been to replace troublesome components by a trained neural network surrogate and then embed the trained neural network into a nonlinear optimization model [ 16 ] . Several open-source software packages, e.g., OMLT [ 7 ] and gurobi-machinelearning [ 12 ] , make it easy to embed neural network models in Python-based modeling environments for nonlinear optimization. A trained neural network may be represented in an optimization problem with different formulations, e.g., full-space and reduced-space formulations [ 7 , 20 ] , which have been compared by Kilwein [ 13 ] for a security-constrained AC optimal power flow (SCOPF) problem. A third approach, first suggested by Casas [ 6 ] , is called a gray-box formulation, in which function and derivative evaluations of the surrogate are handled by the neural network modeling software (here, PyTorch [ 18 ] ), rather than the algebraic modeling environment (here, JuMP [ 15 ] ). While full-space, reduced-space, and gray-box formulations have been compared in [ 6 ] , the bottlenecks of these formulations have not been carefully identified. This paper profiles these three formulations of a neural network predicting transient feasibility in an SCOPF problem. We demonstrate that the gray-box formulation is the most scalable, and that it can naturally take advantage of GPU acceleration built into PyTorch. We find that the full-space formulation is bottlenecked by the solution of a linear system of equations, and the reduced-space formulation is bottlenecked by JuMP and its automatic differentiation system. Our work provides a clear benchmark and direction for future work. Additionally, we provide MathOptAI.jl, a new open-source library for embedding trained machine learning predictors into optimization models built with JuMP [ 15 ] . MathOptAI.jl is available at https://github.com/lanl-ansi/MathOptAI.jl under a BSD-3 license. 2 Background 2.1 Nonlinear optimization We study nonlinear optimization problems in the form given by Equation ( 1 ): \min_{x}f(x)\text{ s.t. }g(x)=0;~x\geq 0. (1) We consider interior point methods, such as IPOPT [ 22 ] , for solving ( 1 ). These methods iteratively compute search directions d by solving the linear system ( 2 ): \begin{bmatrix}(\nabla^{2}\mathcal{L}(x)+\alpha)&\nabla g(x)^{T}\\ \nabla g(x)&0\end{bmatrix}d=-\begin{bmatrix}\nabla f(x)+\nabla g(x)^{T}\lambda+\beta\\ g(x)\end{bmatrix}, (2) where \alpha and \beta are additional terms that are not shown for simplicity. The matrix on the left-hand side is referred to as the Karush-Kuhn Tucker , or KKT , matrix. To construct this system, solvers rely on callbacks that provide the Jacobian \nabla g and, optionally, the Hessian of the Lagrangian function, \nabla^{2}\mathcal{L} . These are typically provided by the automatic differentiation system of an algebraic modeling environment. If the Hessian \nabla^{2}\mathcal{L} is not available, a limited-memory quasi-Newton approximation is used [ 17 ] . 2.2 Neural network predictors A neural network predictor is a function denoted y={\rm NN}(x) . We consider neural networks defined by repeated application of an affine transformation and a nonlinear activation function \sigma over L layers: \displaystyle y_{l} \displaystyle=\sigma_{l}(W_{l}y_{l-1}+b_{l}) \displaystyle l\in\{1,\dots,L\}, (3) where y_{0}=x and y=y_{L} . Weights W_{l} and biases b_{l} are parameters that are optimized to minimize error on a set of training data representing desired inputs and outputs of the neural network. To a nonlinear optimization solver using a trained neural network, W_{l} and b_{l} are constant. To fit assumptions made by these solvers, we consider only smooth activation functions, e.g., sigmoid and hyperbolic tangent functions. 2.3 Algebraic representations of a neural network In this section, we explain the three ways in which we encode pre-trained neural network predictors into the constraints of a nonlinear optimization model of the form ( 1 ). The three approaches are denominated full-space , reduced-space , and gray-box . Full-space In the full-space formulation, we add an intermediate vector-valued decision variable z_{l} to represent the output of the affine transformation in each layer l , and we add a vector-valued decision variable y_{l} to represent the output of each nonlinear activation function. We then add a linear equality constraint to enforce the relationship between y_{l-1} and z_{l} and a nonlinear equality constraint to enforce the relationship between z_{l} and y_{l} . Thus, the neural network in ( 3 ) is encoded by the constraints: \displaystyle{W_{l}}y_{l-1}-z_{l} \displaystyle=-b_{l} \displaystyle l\in\{1,\dots,L\} (4) \displaystyle y_{l}-\sigma_{l}(z_{l}) \displaystyle=0 \displaystyle l\in\{1,\dots,L\}. The full-space approach prioritizes small expressions and small, sparse nonlinear constraints at the cost of introducing additional variables and constraints for each layer of the neural network. This formulation conforms to the assumptions of JuMP’s reverse-mode automatic differentiation algorithm: 1) nonlinear constraints can be written as a set of scalar-valued functions, and 2) they are sparse in the sense that each scalar constraint contains relatively few variables and has a simple functional form. Reduced-space In the reduced-space formulation we add a single vector-valued decision variable y to represent the output of the final activation function, and we add a single vector-valued nonlinear equality constraint that encodes the complete network. Thus, the neural network in ( 3 ) is encoded as the vector-valued constraint: y-\sigma_{L}({W_{L}}(\ldots\sigma_{l}({W_{l}}(\ldots\sigma_{1}({W}_{1}x+b_{1})\ldots)+b_{l})\ldots)+b_{L})=0. (5) The benefit of the reduced-space approach is that we add only a single vector-valued decision variable y and a single vector-valued nonlinear equality constraint (each of dimension of the output size of the last layer). The downside is that the nonlinear constraint is a complicated expression with a very large number of terms. This is made worse by the fact that JuMP scalarizes vector-valued expressions in nonlinear constraints. Thus, instead of efficiently representing the affine relationship {W_{1}}x+b_{1}

Read the complete text