Calibrated uncertainty quantification for prosumer flexibility aggregation in ancillary service markets Journal: Applied Energy

Yogesh Pipada Sunil Kumar, S. Ali Pourmousavi, Jon A.R. Liisberg, Julian Lesmos-Vinasco · 2026

Read Calibrated uncertainty quantification for prosumer flexibility aggregation in ancillary service markets Journal: Applied Energy by Yogesh Pipada Sunil Kumar, S. Ali Pourmousavi, Jon A.R. Liisberg, Julian Lesmos-Vinasco (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-14663.txt.

Introduction

Yogesh Pipada Sunil Kumar, S. Ali Pourmousavi, Jon A.R. Liisberg, and Julian Lesmos-Vinasco study energy markets in this 2026 paper. Reliable forecasting of prosumer flexibility is critical for demand response aggregators. Prosumers both use and produce power, for example with rooftop solar. Aggregators bid their flexibility into frequency controlled ancillary service markets. Those markets enforce strict reliability requirements, such as the P90 standard.

Forecasting is hard for three reasons the authors name. Historical data are limited. Forecasts depend on exogenous inputs. Prosumer behavior is heterogeneous. These factors create significant epistemic uncertainty. The stakes are cash. A bid that is too bold can fail the test and cost the firm a fee. A range you can trust beats a point you can not. That is the heart of the bid plan here. The paper builds calibrated uncertainty quantification for flexibility aggregation, so bids rest on honest confidence ranges.

Catalog record

CREATORYogesh Pipada Sunil Kumar, S. Ali Pourmousavi, Jon A.R. Liisberg, Julian Lesmos-Vinasco
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-14663

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Calibrated uncertainty quantification for prosumer flexibility aggregation in ancillary service markets Journal: Applied Energy Yogesh Pipada Sunil Kumar, S. Ali Pourmousavi, Jon A.R. Liisberg, Julian Lesmos-Vinasco Abstract Reliable forecasting of prosumer flexibility is critical for demand response aggregators participating in frequency controlled ancillary service markets, where strict reliability requirements such as the P90 standard are enforced. Limited historical data, dependence on exogenous forecasts, and heterogeneous prosumer behaviour introduce significant epistemic uncertainty, making deterministic or poorly calibrated probabilistic models unsuitable for market bidding. This paper proposes the use of a scalable uncertainty quantification framework that integrates monte carlo dropout (MCD) with conformal prediction (CP) to produce calibrated, finite-sample prediction intervals for aggregated prosumer flexibility. The proposed framework is applied to a behind-the-meter aggregator participating in the Danish manual frequency restoration reserve capacity market. A large-scale synthetic dataset is generated using a modified industry-grade home energy management system, combined with publicly available load, solar, price, activation, and device-level data. The resulting machine learning surrogate model captures aggregate prosumer price responsiveness and provides uncertainty-aware flexibility estimates suitable for market bidding. Multiple multivariate conformal prediction strategies are evaluated and benchmarked against conventional MCD-based methods. Results show that standalone MCD systematically overestimates available flexibility and violates P90 compliance, whereas the proposed MCD–CP framework achieves reliable coverage with controlled conservatism. When embedded in an aggregator bidding model, conformalised methods substantially reduce overbidding risk and achieve up to 70% of perfect-information profit while satisfying regulatory reliability constraints, providing a practical, computationally efficient, and market-compliant solution for aggregator flexibility forecasting under uncertainty. Acronyms bnn ccp cp dam fcas hems is mae mcd mcp mfrr ml mmcp mpiw mse nn pcp pi picp pv rmse sota uq 1 Introduction 1.1 Motivation Independent demand response aggregators combine distributed energy resources (DERs) such as photovoltaic systems, residential batteries, electric vehicles, and flexible loads to provide demand side flexibility to energy and balance markets. They act as intermediaries between small scale prosumers, who individually offer limited flexibility, and system operators, who increasingly depend on fast and affordable flexibility as conventional generation retires. Effective aggregator models can reduce consumer costs, stimulate innovation, and accelerate DER adoption while enhancing system efficiency and supporting grid stability [ 4 ] . Aggregators therefore play a pivotal role in enabling a cost effective and resilient energy transition. Hence, the European Union has introduced measures under the Electricity Directive (EU) 2019/944 [ 22 ] and the Electricity Regulation (EU) 2024/1747 [ 23 ] to ensure equitable participation of aggregators in electricity markets. These initiatives, complemented by ongoing consultations on a demand response network code [ 21 ] , aim to remove regulatory barriers and strengthen the operational and market integration of aggregators across Europe. One of the central challenges in aggregator business models is predicting prosumer price responsiveness, that is, the extent of flexibility offered at a given incentive level and a given time. This relationship is inherently complex, as prosumer behaviour depends on interdependent factors such as weather, load variability, solar generation, and behavioural inconsistencies [ 52 , 17 ] . Data-driven models have therefore emerged as an effective solution, enabling accurate and scalable estimation of prosumer behaviour from historical data without restrictive assumptions [ 33 ] . However, the limited availability of behavioural data and the reliance on forecasts of weather, load, solar generation, and price introduce significant epistemic uncertainty into these models. Such uncertainty directly affects the procurement and delivery of critical balancing services, where reserved capacity may not be available when required. From an economic perspective, enforcing 100% availability requirements on aggregators relying on inherently uncertain flexibility sources would impose excessive financial risk, potentially rendering participation unprofitable and discouraging market entry. To balance the risk of capacity unavailability with the need to incentivise aggregator participation, the Danish system operator has adopted a P90 compliance standard for frequency controlled ancillary services (FCAS) capacity market participants, allowing a capacity deviation with a maximum probability of 10% [ 32 ] . Non-compliance can result in financial penalties or suspension from market participation, highlighting the need for predictive models with integrated uncertainty quantification. Consequently, this study aims to develop a data-driven framework that captures epistemic uncertainty while providing formal coverage guarantees, thus improving the robustness of aggregator forecasts under data scarcity and market reliability constraints. 1.2 Related work Uncertainty quantification (UQ) has become a central research focus in energy forecasting, particularly for aggregator business models that depend on reliable flexibility estimation under limited data availability and forecast dependence. Probabilistic models that capture both aleatoric and epistemic uncertainties enable more informed and risk-aware decision-making, aligning with regulatory requirements such as the P90 compliance standard in ancillary service markets. Bayesian neural networks (BNNs) have been widely studied for demand and flexibility forecasting due to their ability to capture both epistemic and aleatoric uncertainty by learning distributions over model parameters [ 9 ] . Several studies report improved uncertainty quantification for residential demand forecasting using BNNs compared to deterministic neural networks [ 47 , 54 , 7 ] . More recently, [ 16 ] applied a BNN -based transformer architecture to electric vehicle flexibility forecasting for frequency regulation markets, demonstrating strong predictive performance. However, the high computational cost of BNNs limits their scalability, particularly for complex architectures and longer forecasting horizons. Approximate Bayesian methods such as Monte Carlo dropout (MCD) provide a computationally efficient alternative [ 6 , 25 ] . By retaining dropout layers during inference, MCD produces a predictive distribution that can be interpreted as an approximation of a deep Gaussian process [ 25 ] . This approach has been applied successfully in medical imaging [ 31 ] , object detection [ 56 ] , and energy demand forecasting [ 8 ] . Nonetheless, MCD does not yield a true Bayesian posterior and can result in miscalibrated uncertainty estimates, particularly in multi-output or long-horizon settings [ 24 , 45 ] . To address calibration issues, a promising research direction is the integration of MCD with conformal prediction (CP) as proposed by authors in [ 27 ] . CP provides a lightweight and distribution-free framework to obtain prediction intervals with finite-sample coverage guarantees under data exchangeability [ 44 , 3 ] . Applications in electricity price [ 36 ] and photovoltaic generation forecasting [ 40 ] commonly employ conformalised quantile regression, where quantile regression is combined with CP to obtain calibrated prediction quantiles. However, such approaches primarily address aleatoric uncertainty and are less effective in settings dominated by epistemic uncertainty [ 42 ] . This limitation is particularly pronounced in prosumer flexibility forecasting, where epistemic uncertainty arises from limited training data and dependence on exogenous forecasts such as weather conditions and market prices. While quantile regression struggles to represent this uncertainty, MCD is well suited to capturing epistemic effects but often produces miscalibrated intervals. The hybrid integration of MCD and CP therefore offers a complementary solution, combining epistemic uncertainty representation with calibrated prediction intervals and finite-sample guarantees [ 27 ] . Despite its potential, this hybrid approach has not yet been investigated for prosumer flexibility forecasting or aggregator bidding problems. Moreover, existing applications of MCD and CP have largely been restricted to single-output settings, whereas aggregator decision-making requires multivariate, multi-time-step predictions. Although multivariate CP is an emerging area of research

Read the complete text