Latent Space Representation of Electricity Market Curves: Maintaining Structural Integrity

Martin Výboh, Zuzana Chladná, Gabriela Grmanová, Mária Lucká · 2025

Read Latent Space Representation of Electricity Market Curves: Maintaining Structural Integrity by Martin Výboh, Zuzana Chladná, Gabriela Grmanová, Mária Lucká (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11294.txt.

Introduction

Martin Výboh and co-authors studied compact versions of power market curves in 2025. Supply and demand curves drive energy market study, but storing and modeling them at full size is unwieldy. Standard squeezing methods shrink the curves into a small latent space, yet their rebuilds often break a basic economic rule: the curves must move monotonically, never bending the wrong way.

The authors weighed four methods, two linear and two nonlinear, in two- and 3D latent spaces. After preprocessing a three-year hourly data set from the Iberian power market to unify form and tame outliers. The nonlinear method UMAP ranked first across several error measures. An optional isotonic regression step, which forces monotonicity back into the output, cut errors further and restored validity for several methods, making the squeezed curves safer for downstream modeling.

Catalog record

CREATORMartin Výboh, Zuzana Chladná, Gabriela Grmanová, Mária Lucká
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-11294

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Latent Space Representation of Electricity Market Curves: Maintaining Structural Integrity Martin Výboh, Zuzana Chladná, Gabriela Grmanová, Mária Lucká Abstract Efficiently representing supply and demand curves is vital for energy market analysis and downstream modelling; however, dimensionality reduction often produces reconstructions that violate fundamental economic principles such as monotonicity. This paper evaluates the performance of PCA, Kernel PCA, UMAP, and AutoEncoder across 2d and 3d latent spaces. During preprocessing, we transform the original data to achieve a unified structure, mitigate outlier effects, and focus on critical curve segments. To ensure theoretical validity, we integrate Isotonic Regression as an optional post-processing step to enforce monotonic constraints on reconstructed outputs. Results from a three-year hourly MIBEL dataset demonstrate that the non-linear technique UMAP consistently outperforms other methods, securing the top rank across multiple error metrics. Furthermore, Isotonic Regression serves as a crucial corrective layer, significantly reducing error and restoring physical validity for several methods. We argue that UMAP’s local structure preservation, combined with intelligent post-processing, provides a robust foundation for downstream tasks such as forecasting, classification, and clustering. 1 Introduction 1.1 Related Work Despite decades of market liberalisation, predicting electricity prices remains a complex challenge due to high volatility and limited storage options. While traditional approaches have focused on direct price regression [ 2 , 3 ] or standard machine learning algorithms [ 7 ] , recent research has shifted toward modelling entire supply and demand curves. These curves offer insights exceeding simple market-clearing price analysis, uncovering shifts in production or demand. Previous studies have primarily concentrated on the supply side, employing techniques such as multivariate linear regression [ 20 ] , radial basis function (RBF) [ 14 ] , and piecewise linear functions [ 19 ] . Notably, [ 4 ] and [ 15 ] addressed the high dimensionality of supply curves by applying dimensionality reduction methods (e.g., PCA) combined with deep learning for forecasting. More recently, [ 13 ] introduced a modular framework that uses a monotonic AutoEncoder to preserve the monotonicity of reconstructed supply and demand curves. 1.2 Our Contribution In this work, we propose a framework that builds upon and extends earlier contributions in the field. We simplify and expand the preprocessing approach presented in [ 4 ] . We introduce a modular component that ensures the monotonicity of curves reconstructed from any latent space. To the best of our knowledge, this is the first study to analyse the ability of various dimensionality reduction methods to capture the intrinsic properties of electricity market curves. It also appears to be the first time that methods such as Kernel PCA and UMAP have been explored for electricity market curve analysis. To address the high-dimensional nature of market curves, we compare four distinct reduction techniques: Principal Component Analysis (PCA) for linear transformation [ 10 ] , Kernel PCA (kPCA) for non-linear mapping in Hilbert space [ 12 ] , Uniform Manifold Approximation and Projection (UMAP) for scalable manifold learning [ 8 ] , and AutoEncoder (AE) for capturing complex non-linearities through deep learning [ 17 ] . We validate our approach using data from the Iberian Electricity Market. 2 Methodology This section describes the framework for transforming supply and demand curves into a latent space. The presented procedure consists of the following phases: data processing and transformation, dimensionality reduction, and post-processing. Each phase is applied independently to both supply and demand curves. 2.1 Data Preprocessing Let the dataset consist of K stepwise market curves, where the i -th curve is a sequence of price-volume pairs [P_{i1}^{0},Q_{i1}^{0}],[P_{i2}^{0},Q_{i2}^{0}],\dots,[P_{iN_{i}}^{0},Q_{iN_{i}}^{0}], where N_{i} is the number of price steps of curve i . Inspired by the work of [ 4 ] , we implement a four-step preprocessing pipeline but omit their sigmoid transformation, as it is redundant under standardisation and complicates inverse mapping (produces values outside [0,1] interval): 1. Price winsorization: To focus on the price-sensitive region, all prices are winsorized [ 18 ] to a 99% confidence interval [P_{\min},P_{\max}] of historical market-clearing prices. The price-winsorized price-volume pairs are now denoted [P_{ij}^{1},Q_{ij}^{1}] . The values of the volume and the number of pairs describing the i -th individual curve remain unchanged, i.e. Q_{ij}^{1}=Q_{ij}^{0} . 2. Merging in prices: Here, we solve the problem of non-uniform prices, which represent the individual market curves. We adopt the merging procedure proposed by [ 4 ] . We map all prices to a uniform grid R_{P} with a fixed integer step size \delta_{P} . We define R_{P} as the sequence of integers \{P_{\text{start}},P_{\text{start}}+\delta_{P},\dots,P_{\text{end}}\} that encompasses the winsorized interval [P_{\min},P_{\max}] . Each price from \{P_{ij}^{1}\}_{j=1,\dots,N_{i}} is transformed to its nearest neighbour in R_{p} , resulting in discretized series \{P_{ij}^{2}\}_{j=1,\dots,N_{i}} . This ensures consistent prices across the dataset, while maintaining original volumes, i.e. Q_{ij}^{2}=Q_{ij}^{1} . 3. Sampling in volume: We decrease the dimension of the market curve representation by uniform sampling of reference bidding prices. This can be done by sampling the reference prices at fixed electricity volume points. Departing from [ 4 ] , we restrict the volume range to a uniform sequence R_{Q}=\{Q_{\text{start}},Q_{\text{start}}+\delta_{Q},\dots,Q_{\text{end}}\} , that encompasses a 99% confidence interval (CI) of historical clearing volumes, rather than considering the entire volume range from Q_{\min} to Q_{\max} . This shifts the focus to the high-probability region where supply and demand curves intersect. Each curve is resampled onto R_{Q} , resulting in a uniform K\times N price matrix, where N=|R_{Q}| represents the fixed number of bidding reference prices \mathbf{P}^{3}=\left\{P_{ij}^{3}\mid i=1,\dots,K;\,j=1,\dots,N\right\}. 4. Standardisation of prices. Finally, the matrix \mathbf{P}^{3} is standardised to zero mean and unit variance per dimension (calculated on the training set), yielding the input matrix \mathbf{P}^{4} . 2.2 Dimensionality Reduction The high dimensionality of market curves poses a challenge for accurate prediction. However, since these curves often exhibit recurring patterns, dimensionality reduction (DR) techniques can effectively project them into a low-dimensional latent space while preserving their essential characteristics. To achieve this, we employ four methods: PCA, kPCA, UMAP, and AE. Each method requires hyperparameter tuning, which we perform using two data subsets: a training set for optimisation and a validation set for evaluation. The selected techniques are applied to the \mathbf{P}^{4} matrix. During evaluation, the validation set is projected into the latent space and then reconstructed back to the original space. The performance of each method is assessed using Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) of the reconstructed curves in \mathbf{P}^{3} form. 2.3 Post-Processing Most DR methods do not inherently enforce monotonicity – a critical property of market curves. To address this, we apply Isotonic Regression (IR) [ 1 ] to the reconstructed curves, ensuring monotonicity without altering the underlying architecture of the DR methods. The IR model is fitted to all samples from the training set per curve type. 3 Experiments We evaluate the proposed framework using hourly supply and demand curves from the Iberian Electricity Market (MIBEL) [ 5 ] for the period 2018–2020. The dataset ( K=26\,298 ) was divided into training, validation, and test sets using a 7:1:4 ratio (1 year and 9 months : 3 months : 1 year). Experiments were implemented in the Python programming language. For DR methods, we utilised the scikit-learn [ 11 ] , skorch [ 16 ] , PyTorch [ 9 ] , and umap [ 8

Read the complete text