End-to-End Learning of Correlated Operating Reserve Requirements in Security-Constrained Economic Dispatch
Owen Shen, Hung-po Chao, Haihao Lu, Patrick Jaillet · 2026
Read End-to-End Learning of Correlated Operating Reserve Requirements in Security-Constrained Economic Dispatch by Owen Shen, Hung-po Chao, Haihao Lu, Patrick Jaillet (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05167.txt.
Introduction
Shen and three colleagues frame reserve design, choosing the shape to cut dispatch cost under a cover target. Shaping the cover constraint into a radius turns the problem into a goal, with KKT information giving task slopes. A split mixes a quantile gauge with even tuning, yielding cover. It is tested on the IEEE system with transfer constraints. The learned shape cuts dispatch cost by about 4.8 percent while keeping cover above target.
Catalog record
| CREATOR | Owen Shen, Hung-po Chao, Haihao Lu, Patrick Jaillet |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68159 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.05167
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
End-to-End Learning of Correlated Operating Reserve Requirements in Security-Constrained Economic Dispatch Owen Shen, Hung-po Chao, Haihao Lu, Patrick Jaillet Abstract Operating reserve requirements in security-constrained economic dispatch (SCED) depend strongly on the assumed correlation structure of renewable forecast errors, yet that structure is usually specified exogenously rather than learned for the dispatch task itself. This paper formulates correlated reserve-set design as an end-to-end trainable robust optimization problem: choose the ellipsoidal uncertainty-set shape to minimize robust dispatch cost subject to a target coverage requirement. By profiling the coverage constraint into a shape-dependent radius, the original bilevel problem becomes a single-stage differentiable objective, and KKT/dual information from the SCED solve provides task gradients without differentiating through the solver. For unknown distributions, a four-way train/tune/calibrate/test split combines a smoothed quantile-sensitivity estimator for training with split conformal calibration for deployment, yielding finite-sample marginal coverage under exchangeability and a consistent gradient estimator for the smoothed objective. The same task gradient can also be passed upstream to context-dependent encoders, which we report as a secondary extension. The framework is evaluated on the IEEE 118-bus system with a coupled SCED formulation that includes inter-zone transfer constraints. The learned static ellipsoid reduces dispatch cost by about 4.8% relative to the Sample Covariance baseline while maintaining empirical coverage above the target level. Keywords: Robust optimization, uncertainty sets, operating reserves, machine learning, conformal prediction, electricity markets. 1 Introduction Electricity markets rely on operating reserve procurement and pricing mechanisms to manage uncertainty arising from high penetrations of wind and solar generation. Renewable forecast errors exhibit strong spatial and temporal correlations due to common weather patterns ( 18 ; 27 ) . When such correlations are ignored, reserve capacity may be inefficiently allocated, leading to excessive costs and distorted price signals. A central modeling question is therefore how correlated reserve requirements should be parameterized for the dispatch task itself. Robust optimization provides a principled framework for incorporating uncertainty into power system operations ( 5 ; 6 ) . In security-constrained economic dispatch (SCED), uncertain parameters are assumed to lie within a prescribed uncertainty set , and the dispatch must remain feasible for all realizations within this set. The geometry of the uncertainty set fundamentally determines the conservatism and cost of the solution ( 8 ) . The resulting design problem is inherently coupled: the uncertainty-set geometry affects reserve requirements, the SCED solve determines the economic value of that geometry, and the reliability requirement determines the radius needed for coverage. The goal in this paper is to learn that uncertainty-set geometry that is economically aligned with downstream dispatch. We therefore treat reserve-set design as an end-to-end trainable problem: profile the coverage requirement into a shape-dependent radius, optimize the resulting objective with KKT/dual information from the dispatch solve, and obtain a task gradient that can also be passed upstream to contextual models. 1.1 Literature Review The most relevant prior work falls into six strands: robust power-system optimization, data-driven calibration, decision-focused learning, learned uncertainty sets, conformal prediction, and renewable forecast modeling. A) Robust Optimization for Power Systems. Robust methods have been widely adopted for unit commitment and economic dispatch. Bertsimas et al. ( 8 ) developed adaptive robust optimization for security-constrained unit commitment with polyhedral uncertainty sets. Jiang et al. ( 22 ) proposed two-stage robust unit commitment, and Zeng and Zhao ( 35 ) introduced column-and-constraint generation. Lorca et al. ( 24 ) extended these to multistage settings, while Jabr ( 21 ) developed adjustable robust OPF with ellipsoidal sets. These approaches assume a fixed uncertainty set geometry determined a priori. B) Data-Driven and Distributionally Robust Approaches. Bertsimas et al. ( 7 ) proposed data-driven robust optimization using hypothesis testing to calibrate set size. Distributionally robust optimization (DRO) optimizes over ambiguity sets: Mohajerin Esfahani and Kuhn ( 25 ) developed Wasserstein DRO with tractable reformulations, Gao and Kleywegt ( 16 ) established finite-sample guarantees, and Van Parys et al. ( 31 ) proved asymptotic optimality. In power systems, Xiong et al. ( 34 ) applied moment-based DRO to unit commitment. Roald et al. ( 28 ) survey optimization under uncertainty in power systems. These methods do not directly optimize uncertainty set geometry for downstream costs. C) Decision-Focused Learning and Differentiable Optimization. Decision-focused learning trains models to minimize downstream decision cost ( 15 ; 14 ) . Differentiable optimization layers ( 2 ; 1 ; 10 ) enable backpropagation through convex programs. Our envelope-based approach avoids differentiating through solvers entirely. D) Learned Uncertainty Sets. Wang et al. ( 33 ) proposed learning decision-focused uncertainty sets via implicit differentiation. Chenreddy and Delage ( 12 ) developed end-to-end conditional robust optimization. Goerigk and Kurtz ( 17 ) used neural networks to predict uncertainty set parameters. What remains missing for reserve procurement is an application-driven formulation in which the uncertainty-set geometry, the SCED solve, and the coverage calibration are optimized as one pipeline. The framework in this paper closes that gap by profiling the coverage constraint and using KKT/dual sensitivities from SCED to optimize the resulting single-stage objective without solver backpropagation. E) Conformal Prediction. Conformal prediction ( 32 ) provides distribution-free uncertainty quantification under exchangeability. Romano et al. ( 29 ) developed conformalized quantile regression, and Angelopoulos and Bates ( 3 ) provide a comprehensive tutorial. Johnstone and Cox ( 23 ) connected conformal prediction to robust optimization via Mahalanobis distance. Here, conformal calibration plays a specific operational role: once the shape is learned, it calibrates the radius needed to meet the reserve-coverage target. F) Renewable Forecast Uncertainty. Wind and solar forecast errors exhibit complex spatial and temporal correlations ( 18 ; 27 ) . Hong and Fan ( 19 ) surveyed probabilistic load forecasting. These works motivate the need for correlation-aware uncertainty sets that adapt to varying forecast error patterns—precisely what the learned Cholesky parameterization is designed to capture. Taken together, these literatures motivate an end-to-end reserve-learning formulation in which uncertainty-set shape is learned for the dispatch task itself while coverage is enforced out of sample. That is the role of the framework developed below. 1.2 Contribution and Organization Main contribution. • Modeling and reformulation. We formulate correlated reserve-set design in SCED as an end-to-end trainable robust optimization problem, where the uncertainty-set shape is optimized against the downstream robust dispatch value function and the coverage constraint is profiled into a shape-dependent radius. This yields a single-stage gradient-friendly objective, and the paper establishes the corresponding profiling and envelope-gradient results that justify using SCED dual information rather than solver differentiation. • Algorithm and calibration. We develop a practical learning procedure based on a train/ tune/ calibrate/ test split, where the tuning set supports estimation of the smoothed profiled gradient and the calibration set supplies a conformal radius for the final deployed set. The paper further gives a finite-sample conformal coverage guarantee under exchangeability and a consistency result for the smoothed gradient estimator under standard regularity conditions. • Empirical study. We validate the framework on the IEEE 118-bus system and show that the learned static ellipsoid reduces dispatch cost by 4.8% relative to the Sample Covariance baseline while maintaining empirical coverage above the target level. We further study the method under coupled SCED with transfer constraints, showing that the same formulation and gradient machinery remain effective in a more operationally