Code Generation for Solving and Differentiating through Convex Optimization Problems
Maximilian Schaller, Stephen Boyd · 2025
Read Code Generation for Solving and Differentiating through Convex Optimization Problems by Maximilian Schaller, Stephen Boyd (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14099.txt.
Introduction
Maximilian Schaller and Stephen Boyd add, in a 2025 paper, a code mill that both solves and grades bent-shape tune tasks. Such tasks, convex by form, sit in learn, control, and trade code. Their tool grows out of CVXPYgen, which writes fast C for a set task in the CVXPY tongue. The new part writes C that also gives how the best point shifts when a task knob shifts. A Python wrap serves desk tests at the same time. Three uses show the path: tune learn knobs, set a control plan's knobs, and tune a trade rule on past prices.
Speed ran near ten-fold past the broad tool for the same job. It is the first mill to write such slope code in C.
Catalog record
| CREATOR | Maximilian Schaller, Stephen Boyd |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-14099
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Code Generation for Solving and Differentiating through Convex Optimization Problems Maximilian Schaller, Stephen Boyd Abstract We introduce custom code generation for parametrized convex optimization problems that supports evaluating the derivative of the solution with respect to the parameters, i.e. , differentiating through the optimization problem. We extend the open source code generator CVXPYgen, which itself extends CVXPY, a Python-embedded domain-specific language with a natural syntax for specifying convex optimization problems, following their mathematical description. Our extension of CVXPYgen adds a custom C implementation to differentiate the solution of a convex optimization problem with respect to its parameters, together with a Python wrapper for prototyping and desktop (non-embedded) applications. We give three representative application examples: Tuning hyper-parameters in machine learning; choosing the parameters in an approximate dynamic programming (ADP) controller; and adjusting the parameters in an optimization based financial trading engine via back-testing, i.e. , simulation on historical data. While differentiating through convex optimization problems is not new, CVXPYgen is the first tool that generates custom C code for the task, and increases the computation speed by about an order of magnitude in most applications, compared to CVXPYlayers, a general-purpose tool for differentiating through convex optimization problems. 1 Introduction A convex optimization problem, parametrized by \theta\in\Theta\subseteq{\mbox{\bf R}}^{d} , can be written as \begin{array}[]{ll}\mbox{minimize}&f_{0}(x,\theta)\\ \mbox{subject to}&f_{i}(x,\theta)\leq 0,\quad i=1,\ldots,m\\ &A(\theta)x=b(\theta),\end{array} (1) where x\in{\mbox{\bf R}}^{n} is the optimization variable, f_{0} is the objective function to be minimized, which is convex in x , and f_{1},\ldots,f_{m} are inequality constraint functions that are convex in x [ BV04 ] . The parameter \theta specifies data that can change, but is constant and given (or chosen) when we solve an instance of the problem. We refer to the parametrized problem ( 1 ) as a problem family ; when we specify a fixed value of \theta\in\Theta , we refer to it as a problem instance . We let x^{\star} denote an optimal point for problem ( 1 ), assuming it exists. To emphasize its dependence on \theta , we write it as x^{\star}(\theta) . Convex optimization is used in many domains, including signal and image processing [ MB10 , ZE10 ] , machine learning [ Mur12 , BN06 , ZH05 , Tib96 , Cor95 , HK70 , Cox58 ] , control systems [ RMD+17 , KC16 , WOB15 , KB14 , WB09 , BEGFB94 , BB91 , GPM89 ] , quantitative finance [ Pal25 , BJK+24 , BBD+17 , Nar13 , LFB07 , GK00 , Mar52 ] , and operations research [ Hal19 , BG92 , Ber91 , LAHH16 , BT04 ] . 1.1 Differentiating through convex optimization problems In many applications we are interested in the sensitivity of the solution x^{\star} with respect to the parameter \theta . We will assume there is a unique solution for parameters near \theta , and that the mapping from \theta to x^{\star} is differentiable with Jacobian \partial x^{\star}/\partial\theta\in{\mbox{\bf R}}^{n\times d} , evaluated at \theta . We then have \Delta x^{\star}\approx\frac{\partial x^{\star}}{\partial\theta}\Delta\theta, where \Delta\theta is the change in \theta , and \Delta x^{\star} is the resulting change in x^{\star} . We make a few comments on our assumptions. First, the solution x^{\star} need not be unique, and so does not define a function from \theta to x^{\star} . Even when the solution is unique for each parameter value, the mapping from \theta to x^{\star} need not be differentiable. Following universal practice in machine learning, we simply ignore these issues. When x^{\star} is not unique, or when the mapping is not differentiable, we simply use some reasonable value for the (nonexistent) derivative. It has been observed that simple gradient (or subgradient) based methods for optimizing parameters are tolerant of these approximations. Approximating the change in solution with a change in parameters can be useful by itself in some applications. As an example, consider a machine learning problem where we fit the parameters of a model to data by minimizing the sum of a convex loss function over the given training data. Considering the training data as a parameter, and the solution as the estimated model parameters, the Jacobian above gives us the sensitivity of each model parameter with respect to the training data. In particular, these sensitivities are sometimes used to compute a risk estimate for the learned model parameters [ NCB23 , NLC24 ] . As another example, suppose we model some economic variables ( e.g. , consumption, demand for products, trades) as maximizing a concave utility function that depends on parameters. The Jacobian here directly gives us an approximation of the change in demand (say) when the utility parameters change [ Wai05 ] . 1.2 Autodifferentiation framework The solution map derivative is much more useful when it is part of an autodifferentiation system such as JAX [ BFH+18 ] , PyTorch [ PGC+17 ] , or Tensorflow [ ABC+16 ] . We consider a scalar function that is described by its compution graph, which can include standard operations and functions, as well the solution of one or more convex optimization problems. We can compute the gradient of this function automatically, and this can be used for applications such as tuning or optimizing the performance of a system. We give a few simple generic examples here. In machine learning, we fit model parameters (also called weights or coefficents) using convex optimization, but we may have other hyper-parameters ( e.g. , that scale regularization terms) that we would like to tune to get good performance on an unseen, out-of-sample test data set [ Mur12 ] . The scalar function that we differentiate is the loss function computed with test data, and we differentiate with respect to the hyper-parameters of the machine learning model. A similar situation occurs in finance, where the actual trades to execute are determined by solving a parameterized problem [ BBD+17 ] , which also contains a number of hyper-parameters that set limits on the portfolio or trading, or scale objective terms, and our goal is to obtain good performance on a simulation that uses historical data, i.e. , a back-test. In this case, the scalar function that we optimize might be a metric like the realized portfolio return or the Sharpe ratio [ LW08 ] , and we differentiate with respect to the hyper-parameters of the portfolio construction model. 1.3 Related work Differentiating through convex optimization problems. There are two main classes of methods to differentiate the mapping from problem parameters to the solution. First, autodifferentiation software like PyTorch [ PGC+17 ] and Tensorflow [ ABC+16 ] , as commonly used in backpropagation for machine learning, would differentiate through all instructions of an iterative optimization algorithm. While these tools are commonly used in the deep learning domain, they neglect the structure and optimality conditions of, particularly, convex optimization problems. Second, as less of a brute-force approach, one can directly differentiate through the optimality conditions of a convex optimization problem, also referred to as argmin differentiation [ ABB+19 , AK17 ] . CVXPYlayers [ AAB+19 ] is based on prior work on differentiating through a cone program, called diffcp [ ABB+19 ] , and provides interfaces to PyTorch and Tensorflow. OptNet [ AK17 ] addresses quadratic programs. Least squares auto-tuning [ BB21 ] is a specialized hyper-parameter tuning