A unified multi-agent optimization framework for intelligent PV-integrated smart energy systems Authors: Muhammad Muneeb Khan, Sadiq Ahmad, Muhammad Naeem, Muhammad Umair Ali, Seong Han Kim Published: 2026 ABSTRACT AbstractThe increasing penetration of photovoltaic (PV) generation and distributed energy resources in residential communities requires intelligent, scalable energy management strategies that go beyond conventional cost-based approaches. This paper proposes a unified multi-agent optimization framework for PV-integrated smart energy communities that incorporates design-aware PV modeling, degradation-aware energy storage control, indoor comfort preservation, and peer-to-peer (P2P) energy exchange. The community energy management problem is formulated as a multi-agent Markov game, in which each household operates as an autonomous agent that coordinates local energy resources and demand. To address the resulting nonlinear and multi-objective optimization problem, a multi-agent reinforcement learning (MARL) approach is developed under a centralized training and decentralized execution paradigm. Simulation results demonstrate that the proposed CTDE-based MARL framework significantly improves operational efficiency and scalability. Compared to baseline strategies, it reduces unified operational cost by up to 69% relative to greedy control and by more than 37% compared to single-agent reinforcement learning, while consistently outperforming Independent Q-Learning (IQL). The framework also minimizes asset degradation, preserves indoor comfort, and achieves performance close to a centralized optimal solution, highlighting its potential for practical deployment in intelligent residential energy systems.Keywords: Energy community/microgrid, Demand response, Peer-to-peer (P2P) energy trading, Multi-agent reinforcement learning (MARL), Battery degradationSubject terms: Energy science and technology, Engineering, Mathematics and computing Introduction The increasing complexity of residential energy systems, driven by high penetration of photovoltaic (PV) generation, energy storage systems (ESS), and flexible loads, necessitates advanced energy management strategies that go beyond conventional cost-based optimization. The growing adoption of renewable energy sources, such as PV systems, in residential communities has both positive and negative effects on smart energy management1. Despite significant advances in smart home energy management, most existing frameworks treat PV generation as a passive and static input, neglecting critical design parameters, operational control strategies, and long-term degradation effects. This limitation leads to suboptimal decision-making, particularly in systems where PV performance and asset health evolve over time2. In addition, the complexity of the integration of energy storage systems (ESS) and variable appliances (i.e., controllable) is aggravated by the need for operational decision-making trade-offs among cost, the ecological impact, and the comfort of the occupants3. Smart homes can integrate solar systems, ESS, and smart controllers to manage energy at the micro level4. If multiple smart homes in a neighborhood are coordinated, peer-to-peer (P2P) energy trading becomes possible, thereby alleviating reliance on the main grid and enhancing the use of renewables5. However, energy management approaches still require refinement and should not treat PV production as a static input. Despite the rise of residential solar systems, most existing energy management frameworks treat PV systems as constant energy sources and ignore the designs and operational controls associated with PV generation, treating them as a given, a simple input. PV generation modeling is often described as a ‘black box’ in existing energy management frameworks because these frameworks fail to capture PV generation as a function of operating climate, power extraction strategies, and operational control strategies. In practice, PV performance is strongly influenced by panel orientation, thermal conditions, and real-time power extraction strategies, all of which directly affect both instantaneous energy yield and long-term degradation6,7. Control policies that ignore these aspects will result in optimally controlled systems that underperform over time and lead to unintended consequences, including increased system and asset values. The same applies to energy storage systems that use suboptimal cycling strategies to mitigate system deterioration and improve overall efficiency8. Empowered by P2P mechanisms, neighboring households in a community can further expand their use of renewables and reduce their reliance on the grid5,9. However, the more systems a community has, the more complicated it becomes to synchronize PV generation, storage operation, and P2P local energy exchange. Community-sized systems are often overlooked and ultimately fail to account for the diverse range of household assets and preferences, and they rely on centralized optimization techniques because their scope is too limited. On the other hand, basic decentralized heuristics fall short of coordinating operations sufficient to realize the full value of the community systems. The challenges are multi-objective and time-coupled. For operational decisions to succeed, an energy management framework must account for electricity costs, environmental impacts, battery and PV degradation, consumer comfort, and the long-term alignment between current control actions and future system performance. Most notably, the feedback loops of PV degradation and battery aging, where aggressive operations today diminish tomorrow’s renewable resources and operational flexibility. To address these challenges, it is essential to develop an energy management system that integrates design-aware PV modeling, operational control degradation, and scalable coordination within a single decision framework. However, existing work often focuses on these challenges in isolation, and current approaches lack a unified framework that jointly captures the dynamics of the physical system, long-term asset degradation, and scalable coordination among multiple residential agents. This paper proposes a unified multi-agent energy management system that accounts for PV position, thermal dynamics, operational control, and long-term degradation within a sustainability-focused P2P energy transaction in which neighbors obtain and supply energy to one another. Overall, the framework describes the boundaries for managing thermal and PV-integrated residential energy systems, within which sustainability, efficiency, and consumer preferences can play a role. Given the limitations identified in the literature, we identify an opportunity to bridge the gap between grounded, real-world modeling and the scaling of intelligent control in residential energy communities. To showcase the benefits of dedicated coordination learning, our framework is benchmarked against Independent Q-Learning (IQL) and compares centralized-critic and independent learners. Contributions To overcome these challenges observed in the state-of-the-art energy management techniques, in this paper, we propose a unified multi-agent optimization framework for PV-integrated residential energy systems. The key intuition of our contribution lies in combining physically grounded modeling and scalable learning-based control within a unified framework. Specifically, we utilize experiential design-aware PV modeling (including orientation, thermal dynamics, operational control, and degradation), along with degradation-aware energy storage modeling (asset behavior over a long time scale). Next, the community energy management problem is formulated as a multi-agent Markov game, which naturally aligns with the coordination of residential agents’ decisions through P2P energy exchange and flexible demand management. As such, we derive a scalable multi-agent reinforcement learning (MARL) approach for the nonlinear, temporally coupled problem in a centralized training and decentralized execution setting, thereby bridging the gap between physical system modeling and decentralized intelligent control. Literature review Coordinated operation at the household and lower levels of the community is complicated by residential energy systems with high PV penetration, battery storage, and flexible loads. In these systems, energy management must balance competing economic trade-offs with asset ageing and occupant comfort, while interconnected systems span multiple layers of decision making. These challenges are central and are characterized by strong temporal coupling, in which current operational decisions affect battery aging and PV degradation. In light of this, intelligent control, coordination, and energy sharing in residential environments have been supported by a wide variety of modeling, optimization, and learning-based methods. Smart home energy management and PV integration Smart home energy management systems (HEMS) aim to optimally control distributed energy resources (DERs), such as PV systems and battery ESS, at the lowest operational cost while maximizing indoor comfort10. Optimization-based methods typically adopt centralized approaches, which tend to struggle as systems and communities grow in size11. In such cases, reinforcement learning (RL) has proven useful as a flexible approach in nonlinear system dynamics, long-term operational dependencies, and multiobjective residential energy management12. Q-learning-based RL applications in smart homes, such as those in13, which aimed to control energy storage and schedule appliances in PV-integrated systems, demonstrated lower energy costs and higher user satisfaction than mixed-integer linear programming (MILP)-based approaches. More recent examples have integrated deep learning with RL to optimize PV and energy storage operations and to improve system-level performance across a range of operating conditions through coordinated reactive and active power flow control11. RL is also reported for real-time energy control and demand response in residential systems, where flexible loads are optimally coordinated to operate adaptively while maintaining comfort14. Multi-agent and reinforcement learning approaches Decentralized energy management systems are increasingly being designed using multi-agent reinforcement learning (MARL), in which each household or prosumer operates as an individual agent15,16. Compared with single-agent RL, MARL frameworks enable decentralized decision-making, enhancing the scalability and adaptability of the system17,18. Independent reinforcement learning approaches, such as Independent Q-Learning (IQL), provide decentralized solutions but often suffer from instability due to environmental non-stationarity, motivating the adoption of coordinated MARL frameworks. In the literature exploring the MARL framework for P2P energy markets, most studies commend MARL for enabling flexible pricing and increasing community self-sufficiency19,20. In the case of heterogeneous home agents, scalable MARL techniques such as multi-agent proximal policy optimization (MAPPO) have been shown to enhance PV system self-consumption and overall energy efficiency21–23. Recent studies suggest that multi-agent deep RL, when applied to decentralized strategies for community batteries, enhances coordination of community storage and minimizes costs24. Federated reinforcement learning (FRL) has been applied to combine the learning of distributed agent(s) towards better policy generalization in the control of smart microgrids12,21. Peer-to-peer energy trading and community energy markets Prosumers can trade surplus renewable energy locally through P2P energy trading, thereby supporting community resilience and reducing reliance on the utility grid25. Automation of bidding and participation in P2P markets, where, through RL and considering network constraints, participants adapt to supply-demand shifts, is possible20,26,27. In P2P distributed trading frameworks, MARL has been shown to be superior at minimizing violations of the distribution network’s voltage and line limits compared to traditional centralized approaches26. Economic equilibrium and bargaining in DER communities have been studied using game-theoretic and multi-agent systems in28,29. Cost and resilience in hybrid trading and energy sharing systems with a shared storage unit have also been analyzed in30,31. PV and storage degradation and design-aware optimization Numerous studies treat PV generation as an external factor, though only a few consider PV design parameters, such as orientation, tilt, and operational control-loop adjustments for degradation. These elements, when combined with multi-objective optimization, improve control and planning strategies tremendously12. Similarly, battery degradation modeling, in terms of cycling costs and state of health, has been used in strategic decision frameworks to improve the longevity of ESS and reduce lifecycle costs21. The transition from classical heuristic and centralized optimization techniques to advanced methods incorporating intelligent, decentralized, and learning-based approaches for smart home and community energy management is evident. MARL and P2P trading frameworks perform well in the complexities of system dynamics, dynamically changing operational time horizons, and multi-criteria such as cost, comfort, and asset lifecycle management. However, within multi-agent frameworks, integrating design-aware PV models with degradation-aware storage optimization remains a challenge and is the primary motivation for this research. Organization of the paper The rest of the paper is structured as follows: section “Literature review” presents smart home energy management, P2P energy exchange, multi-agent control strategies, and PV/battery modeling. In section “Proposed multi-agent energy management framework”, we detail the proposed unified multi-agent energy management framework, system architecture, PV orientation and thermal modeling, decision variables, asset degradation dynamics, operational constraints, and the objective function. Section “Reinforcement learning-based solution methodology” explained and focused on the multi-agent framework, reward design, and control training. The configuration of the case studies and simulation is described, along with the performance evaluation metrics, strategies, and benchmarks, in section “Simulation setup and case study description”. Section “Results and discussion” is devoted to the analysis of the simulation results, the system’s performance, the scalability of coordinated control, the learned behavior, and the coordinated control. Section “Conclusion” contains the conclusions and future work. Proposed multi-agent energy management framework This section presents the proposed multi-agent energy management framework for intelligent PV-integrated smart energy systems. The framework is organized in two layers. First, the application scenario and system-level architecture are described to contextualize the physical and cyber components of the residential energy community. Second, a unified mathematical formulation is developed in which all equations are interconnected through shared state and decision variables, forming a coherent, physically grounded, and degradation-aware theoretical framework. Application scenario and system model The smart energy community, depicted in Fig. 1, consisting of N homes, indexed by , operating over a discrete-time horizon with time steps of duration , each home will function as an autonomous agent, with rooftop PV systems, ESS, controllable and shiftable appliances, and indoor comfort monitoring devices. Prosumers can interact directly with one another, trading energy between households through a P2P mechanism, and may also draw from the utility system at time-varying electricity prices and emission intensities. Layered architecture of the proposed PV-integrated multi-agent energy management framework for residential energy communities. The overall structure of the community, shown in Fig. 1, is defined into three connected layers. The top layer consists of exogenous environmental/economic inputs that give rise to solar irradiance/power and ambient temperature , and to time-varying electricity prices , which result in generation and operational decisions. The middle community coordination/P2P market layer is responsible for cohort market decisions, information exchange, and coordination among residential agents. The bottom layer consists of the home agents , each comprising a fitted PV, an ESS, and flexible household appliances. The members of agents from run a local MARL controller for determining the best action based on their observed state in terms of battery operation/usage, PV power utilization, and appliance scheduling, therefore giving the actions . The energy flows within and between agents in each home residence are again among PV generation, batteries, and loads, with surplus or deficit energy either exchanged P2P with neighbors or interacted with the grid. The PV system would be exogenously defined by isolation and controllable design and operating parameters such as tilt angle, azimuth, and maximum power point tracking (MPPT) efficiency. While the ESS allows for temporal changes in energy use, it suffers from long-term degradation. Demand-side flexibility is provided by appliances and heating, ventilation, and air conditioning (HVAC) systems, while variables in the indoor environment define comfort performance standards. Taken as a whole, an intelligent household-level controller, acting as an RL agent, controls these systems, while the community coordination layer would then smooth ‘distributed energy supply’ through P2P exchange. There are additional considerations important for closing the gap between the real physical system and its numerical model: namely, how the defined variables are realized and how decisions are made in practice. Each agent in the proposed framework is a household with a rooftop PV system, a battery ESS, and smart appliances. The agent’s control decisions are encoded in a home energy management system (HEMS) to be implemented on inverters, battery controllers, and smart devices. Precisely, PV-related control variables (e.g., MPPT control actions) quantify the power extraction strategies of the inverters, battery charging and discharging actions are obtained through ESS controllers (subject to operational limits), and appliance scheduling decisions are yielded via smart plugs/home automation gateways application. Peer-to-peer energy exchange decisions are executed through a community-level coordination or market-clearing mechanism, inverting the transaction of energy among neighboring households possessing local surplus or demand. Unified mathematical model For the unified mathematical formulation, care has been taken to explicitly reflect the operation of residential energy systems and their interaction with one another within a community. Each equation describes an operational procedure (energy balancing, PV generation, storage dynamics, demand management, inter-agent energy exchange, etc.), maintaining consistency between the mathematical model and the actual system. For clarity and improved readability, all variables and parameters used in the formulation are defined at their first occurrence in the text. Energy balance equation In each prosumer (integrated energy node), as shown in Fig. 1, the node consumes, produces, stores, and transfers electrical energy with peers within the residential community. In a time slot t, prosumer i has an energy consumption profile , which comprises a household baseline load, shiftable and controllable appliances like HVAC systems. In relation to the demand, energy can be sourced from the locally produced PV , energy from battery storage/discharged to the battery , utility grid energy , and energy from P2P prosumer trading. Whenever the instantaneous demand is exceeded by local PV generation, the excess energy is represented by (), which can either be stored in the ESS or exported to neighboring prosumers. On the other hand, if local generation is not enough, the prosumer can import energy from neighboring prosumers, represented by (), or can take energy from the grid. All these energy flows must meet the power balance requirement. Accordingly, the energy balance constraint for each prosumer is given by where and . Equation (1) serves as the fundamental coupling constraint linking PV generation, storage operation, appliance scheduling, grid interaction, and peer-to-peer energy exchange. Design- and degradation-aware PV modeling In the proposed framework, each prosumer has a controllable PV system, modeled as an active energy asset. In addition to solar irradiance, PV system output is affected by design parameters, operating conditions, control actions, and degradation over time. Performance is affected by geometric design parameters such as tilt and azimuth, ambient temperature, and MPPT control decisions. Excessive operating temperatures combined with aggressive power extraction can accelerate PV aging, potentially leading to long-term issues such as reduced conversion efficiency and energy yield. The instantaneous PV power output of prosumer i at time t is modeled as where is the PV panel area, denotes solar irradiance, is the reference efficiency, and is the temperature coefficient. The function captures the effect of PV tilt and azimuth on incident irradiance. The control variable represents the maximum power point tracking (MPPT) control action and is the PV degradation factor reflecting the long-term reduction in PV efficiency. The PV cell temperature is approximated as linking thermal stress to ambient temperature and irradiance along with the thermal coefficient . PV degradation evolves according to where and are degradation coefficients associated with thermal stress and MPPT control intensity, respectively, and denotes a projection operator that bounds the degradation factor within the physically meaningful interval [0, 1]. Energy storage system dynamics The ESS provides temporal flexibility by enabling energy shifting over time while accounting for physical constraints and degradation effects. The net ESS power of prosumer i at time t is defined as where and denote charging and discharging power. The state-of-charge (SoC) evolves according to where denotes the state-of-charge of the battery, and are the charging and discharging efficiencies, respectively, represents the time interval, and represents the maximum battery energy capacity. Battery degradation is modeled via an instantaneous cost discouraging aggressive cycling and deep discharging and promoting sustainable ESS operation. where and are weighting coefficients associated with cycling-induced degradation and depth-of-discharge effects, respectively. Appliance scheduling and demand modeling Household demand consists of non-shiftable baseline load, shiftable appliances, and controllable devices. Shiftable appliances satisfy where denotes the permissible operating time window of the shiftable appliance sa, and is a binary decision variable indicating whether the appliance is switched ON or OFF at time t. The term represents the power consumption of the shiftable appliance, denotes the duration of a scheduling interval, and is the total energy requirement that must be satisfied within the allowed time horizon. While controllable appliances obey For controllable appliances, denotes the power consumption of an appliance ca at time t, bounded by its minimum and maximum operating limits and , respectively. The total household energy consumed comprises the baseline demand , the energy consumed by shiftable appliances, and the energy drawn by controllable appliances. Indoor comfort modeling Indoor comfort deviations are penalized using a soft-cost formulation: allowing flexible trade-offs between comfort and energy efficiency without enforcing hard constraints. In (11), denotes the indoor comfort deviation penalty imposed on the residential agent i at time t. The variable denotes the specific indoor environmental parameter, be it temperature (T), humidity (H), or air quality, (AQ); denotes the measured indoor condition of agent i; represents the desired comfort setpoint; and simply defines the maximum acceptable comfort deviation from the setting. The standard syntax in the function implies that the penalty is only applied to the violation once a comfort threshold has been crossed. Such a soft-cost approach allows for trade-offs between occupant comfort and energy efficiency without requiring strict constraints. Peer-to-peer energy exchange Energy exchange between prosumers is bounded as ensuring feasible and coordinated community-level energy sharing. refers to the energy exchanged from prosumer i to prosumer j at time t, while defines the overall maximum exchange during a scheduling interval between any two of the rising prosumers. Also, here, i and j are different residential agents from and . A positive value of means that prosumer i is exporting to prosumer j from his battery system. Community coordination and P2P market clearing mechanism Through the community coordination layer and P2P learning layer, P2P residential agents can trade electricity with one another. The P2P layer will function as a virtual marketplace operator by aggregating bids and offers for different amounts of electricity from residences participating in the market. These aggregated bids and offers will be used to determine a market-clearing price () that all participants will pay for electricity traded in the marketplace, thereby providing a fair and efficient means of trading electricity within the community. Furthermore, we define the power transferred from resident i to resident j at time t as . The following constraints govern how the market-clearing mechanism will function: where and are the amounts of energy that Agent i sells and purchases. Equation (13) ensures that there is a total energy balance among agents. As a result, P2P prices are set between the utility Grid’s buy and sell rates to ensure viability. Hence, We can calculate the total costs for all agents as follows: Thus, this formulation ensures that energy trading is an open and equitable process and that the architectural aspects are well aligned with the mathematical optimization approach of the overall framework. Unified objective function The unified objective integrates electricity cost, emissions, asset degradation, comfort preservation, and community cooperation: In the unified objective equation, i.e., (18), the variable represents the electricity imported by the agent i from the utility grid at time t, and denotes the corresponding time-varying electricity tariff. The term represents the grid emission factor associated with greenhouse gas emissions. The battery degradation cost is denoted by , while represents the PV degradation factor. The indoor comfort deviation cost is given by , and denotes the energy exchanged between prosumers i and j. The coefficients , , , , , and are weighting factors that balance economic cost, environmental impact, asset degradation, occupant comfort, and P2P energy cooperation. MARL problem formulation The system is formulated as a Markov game, where each agent observes a local state which selects control actions, and receives a reward defined as the negative of its contribution to (18). In (19), represents the local observation of agent i, including system states such as battery charge level, PV degradation, environmental conditions, electricity prices, indoor comfort variables, and baseline demand. The resulting formulation is a multi-agent, multi-objective, and temporally coupled optimization problem with nonlinear dynamics, motivating a learning-based decentralized solution. This formulation naturally motivates adopting a Markov game framework, which captures dynamic interactions and decentralized decision-making among multiple residential agents. Reinforcement learning-based solution methodology The unified energy management framework developed in the previous section represents a multi-agent, nonlinear, temporally coupled optimization problem that is long-term dependent due to energy storage behavior and photovoltaic degradation. In other words, traditional centralized optimization methods will not work for large residential areas due to the problem’s inherent unpredictability and computational expense. To address the nonlinearity, trade-offs among multiple objectives, and long-horizon decision-making inherent in this proposed optimization framework, we reformulate the problem as a Markov game and apply a MARL approach to solve it. The remainder of this section discusses how we formulate the problem as a Markov game, how we create an environment in which the agents can act, how the agents will learn, and the relative computational complexity of the algorithms implemented in this section. Markov game formulation We model the proposed multi-agent energy management problem as a Markov game defined by the tuple , where, denotes the set of residential agents (homes), describes the global system state, represents the action space of agent i, defines the state transition dynamics, is the reward function of agent i, and the discount factor. The considered community energy management systems feature multiple autonomous agents operating in a space with temporal coupling dynamics between successive states. Each agent makes sequential decisions regarding photovoltaic generation, energy storage scheduling, flexible load control, and P2P energy trading within time-varying environments and economic conditions. These settings are naturally suited to a Markov game formulation, which generalizes Markov decision processes (MDPs) to multi-agent settings and enables the capture of both cooperative and competitive interactions between agents. Other game theoretic models, such as Stackelberg games, for example, assume leader-follower relationships that will be hierarchical in nature and would be more applicable to centralized pricing or utility-driven market structures. Our proposed residential energy community would operate in a more decentralized, peer-based way without any particular leader in mind. As such, the Markov game formalisms are more applicable to distributed coordination and sequential decision-making. It also fit within the centralized training distributed execution (CTDE) paradigm used in our study. At each time step t, each agent observes a local state and selects an action . The joint action drives the system transition to the next state according to . The Markov property holds since the future evolution of the system depends only on the current state and the joint action. This formulation naturally captures the coupled dynamics introduced by shared energy markets, P2P exchanges, and time-varying renewable generation. Although the framework is formulated as a Markov game, the operating profiles of interest in this study are deterministic, and we adopt the formulation to capture multi-agent temporal coupling and long-horizon decision-making rather than stochastic uncertainty. Building on this Markov game formulation, we will now specify how each agent observes the environment, what decisions it can take, and how its performance is evaluated through reward signals. Agent state, action, and reward design The local observation of agent i at time t is defined as The state vector consists of the following components: (1) the ESS state of charge (SOC); (2) the PV degradation factor; (3) time-varying operating profiles; (4) solar irradiance; (5) ambient temperature; (6) time-varying electricity prices; (7) indoor comfort indicators; and (8) baseline household demand; enabling agent representation of short- and long-term operational conditions of assets. The control action vector of the agent i is expressed as which jointly governs energy storage charging and discharging, PV power extraction via MPPT, appliance scheduling decisions, and peer-to-peer energy exchanges with neighboring agents. The reward function is simply the negative of the agent’s overall contribution to the common goal: This reward captures cost-efficiency, environmental friendliness, degradation-consciousness, awareness of the consumer’s condition, and cooperative behavior with neighbors. By maximizing these rewards cumulatively, agents can learn joint, long-term optimal behavior. Having defined how agents perceive and act in the environment, the next subsection discusses the learning architecture they use to achieve stable training and coordinated control at a greater scale. Multi-agent reinforcement learning architecture Our game-theoretic framework is also suitable for the CTDE paradigm. In the centralized phase of training, each agent relies on a global/aggregated state to reduce the non-stationarity of the multi-agent environment and stabilize training, enabling better coordination and convergence, with the centralized critic evaluating joint actions/global performance. In the execution phase, agents act according to their policy using only locally available information. This entails reduced communication overhead, privacy preservation, and improved scalability, thereby making it suitable for real-world deployment. Other multi-agent reinforcement learning strategies were explored. Single-agent reinforcement learning (Simple RL) treats each agent individually, leading to instability due to the non-stationary nature of learning agents. Fully distributed reinforcement learning only utilizes local observations and peers’ information, leading to resource-heavy communication and a lack of coordination. On the other hand, CTDE strikes a trade-off among coordination, scalability, and efficiency, making the CTDE paradigm suitable for the proposed multi-agent energy management system, which requires both coordinated training and decentralized operation. To provide more insight into why the CTDE paradigm was selected in this multi-agent framework, we offer a comparison of multi-agent reinforcement learning strategies below in Table 1. Table 1Comparison of multi-agent reinforcement learning paradigms.ApproachAdvantagesLimitationsSimple Reinforcement Learning (Simple RL)Simple implementation, fully decentralized.Suffers from non-stationarity and unstableconvergenceDistributed Reinforcement LearningScalable and privacy-preservingLimited coordination; increasedcommunication overheadStackelberg-Based LearningSuitable for hierarchical systemsRequires predefined leader–followerstructuresCentralized Training with DecentralizedExecution (CTDE)Stable learning, improved coordination,scalable and privacy-preservingRequires centralized data duringtraining Comparison of multi-agent reinforcement learning paradigms. Simple implementation, fully decentralized. Suffers from non-stationarity and unstable convergence Limited coordination; increased communication overhead Requires predefined leader–follower structures Centralized Training with Decentralized Execution (CTDE) Stable learning, improved coordination, scalable and privacy-preserving Requires centralized data during training As shown in Table 1, the CTDE paradigm provides an effective balance between coordination, scalability, and privacy, making it particularly suitable for distributed cyber-physical residential energy systems. As illustrated in Table 1, the CTDE paradigm achieves a good trade-off between coordination, scalability, and privacy, making it suitable for distributed cyber-physical residential energy systems. The proposed framework is algorithm-agnostic and can be implemented using many existing state-of-the-art CTDE-like multi-agent reinforcement learning algorithms. Among the most promising implementations are policy-gradient multi-agent actor–critic algorithms (Multi-Agent Deep Deterministic Policy Gradient (MADDPG), Multi-Agent Proximal Policy Optimization (MAPPO)) and value-decomposition algorithms (QMIX). There is no unique “best” algorithm or neural network architecture; the best trade-off depends on the application’s requirements, the computational resources available for optimization, and the size (scale) of the energy control problem. Since the present work aims to propose a unifying and scalable optimization framework, specific implementation details are deferred to future work. The proposed MARL framework involves critics and actors that operate separately and scales to modern policy-gradient multi-agent actor-critic methods. While sharing, agents can reduce non-stationarity during training with a critic from the center. When training is complete, agents rely solely on local observations to improve scalability, privacy, and real-world deployment. Cooperative behavior is enabled through shared rewards and environmental feedback, while decentralized control is retained at the household level. The CTDE design best fits residential energy communities, as coordination is necessary and centralized real-time control is impractical. The subsequent sections describe the training process and information-exchange methods within this learning architecture. To explore further the effectiveness of the resulting CTDE-based architecture, the approach is compared with established multi-agent RL paradigms. In addition to classical optimization approaches such as GA, PSO, and greedy methods, these are also conceptually compared with simple RL paradigms and fully distributed MARL methods. Independent learners do not coordinate with one another and may suffer from non-stationarity, which can cause instability. Fully distributed strategies will improve scalability but may lead to suboptimal performance of the whole system due to a lack of global awareness of the system state. The centralized-critic CTDE paradigm uses global information during training to achieve more stable learning and to encourage coordinated decision-making while keeping execution decentralized. This is a very attractive combination for complex cyber-physical energy systems with interacting agents. Implementation workflow To illustrate how our MARL framework works, we briefly summarize the control and learning processes in a step-by-step sequence. The environment starts each episode with different instances of time-varying solar irradiance, electricity prices, ambient temperature, and household demand. Since each agent has a view on local states, i.e., battery SOC, PV degradation, environmental, and load demand descriptions, at each time step, they choose actions according to their current policies, which include battery charging/discharging, PV power extraction control, appliance scheduling, P2P energy exchanges, and so on. Actions have been taken; the system updates the state and emits an appropriate reward signal. Then, the next state is calculated based on the system dynamics, which include energy balance, storage updates, and degradation methods. Next, the tuple () is recorded and used to update the policy networks. The central critic, which uses global or altered information, stabilizes training, while the actors are updated using local observations. Policy updates are frequent and made incrementally using various gradient-based methods to maximize future reward. An episode at a time can be continued till it converges, and policies learned can finally be deployed on decentralized execution of agents, which happen independently and use local information only. Overall it follows a perception–action–learning loop. Training strategy and information sharing Training proceeds over multiple episodes, with each episode corresponding to a fixed operating horizon (e.g., 1 day or 1 week). At the beginning of each episode, the agents are presented with time-varying profiles of solar irradiance, electricity prices, temperature, and household energy demand. At every time step, the agents can now perform actions that put them in contact with the environment and obtain transition experiences of the form () to improve policies through iterations. In many such applications, agents can disclose only the minimum set of required quantities needed for optimal P2P energy trading, e.g., surplus sharing or local pricing, without exposing sensitive consumption information. In this case, the required information sharing is minimal, since agents only need to acknowledge whether they intend to trade or whether surplus is available, providing an acceptable trade-off among the coherence of coordination, respect for data privacy, and a reasonable, limited level of inter-agent communication. Algorithm summary and computational complexity The primary method entails alternating between the two processes: engaging with the environment, then updating the policy. At every time step, the agents in the system observe the local states, perform actions, receive rewards, and log experiences to update their learning. The policy update process will continue until a steady state is achieved. The implementation of the proposed framework will always yield a positive linear correlation between the number of agents and the time intervals. In specific terms, it will result in the mathematical complexity of (), where the number of agents and the number of time steps for every episode. The main benefit over centralized optimization techniques is that the fully distributed approach maintains the desired level of coordination while decoupling communication and computational loads. This is one of the key reasons why, in high-RES penetration scenarios, medium- to large-scale residential energy communities prefer MARL-based solutions. To evaluate the effectiveness of the proposed multi-agent reinforcement learning framework, a comprehensive simulation-based analysis is conducted. The developed methodology is validated under realistic operating conditions of residential energy systems, including time-dependent solar generation, electricity demand, price signals, and environmental conditions. The following section presents an overview of the simulation setup, parameters, system configuration, and the benchmark strategies used for comparison. Then the proposed approach is analyzed in terms of operational cost, asset degradation, indoor comfort, and system scalability, enabling thorough validation. Simulation setup and case study description We verify the effectiveness, scalability, and coordination capabilities of the proposed framework in this section through simulations conducted on a smart energy community consisting of residential houses operating under various conditions. We describe our simulation setup, community parameters, benchmarking strategies, and performance metrics used in our framework for comparison. The proposed framework can be reproduced using generic deep neural architectures based on fully connected layers with ReLU activation, trained with a gradient-based optimizer such as Adam. We summarize representative hyperparameters as recommendations for deployment in CTDE-based MARL works. Residential community configuration The simulated system considers a residential energy community with prosumer households. These autonomous agents have rooftop residential solar PV generation, an ESS, controllable electrical loads, and the ability to trade energy with other prosumer households via P2P. The operational time frame is for 24 hours and is modeled at hourly intervals. Community-level system integration is effected through decentralized decision-making rather than centralized control and market-based mechanisms. Heterogeneity is captured by assigning each agent its own set of weighting parameters that respond to varying electricity prices, asset wear, and indoor temperature. Results are presented averaged over multiple independent simulation runs. PV, battery, and appliance parameters Every household is modeled with an operational PV system whose output depends on available solar irradiance, solar panel orientation, and thermal conditions. Operational control considers PV degradation and thermal conditions. Operational control causes PV degradation through thermal and MPPT power-absorption penalties, which have ramifications for long-term energy yield. The ESS is modeled with charging and discharging actions, and the state-of-charge (SoC) is divided into 10 levels to meet RL requirements. The cost of battery degradation is applied as a function of energy throughput and changes in depth of discharge. Residential electricity demand is modeled as the sum of household load profiles, and flexible energy management is modeled using net-power variables that affect grid imports, storage, and P2P energy transfer. Coefficients in the unified objective function are selected to balance competing objectives, such as electricity cost, asset degradation, indoor temperature, and energy sharing. Model parameters are chosen based on references from the literature and relevant experience in residential systems. Battery charging/discharging efficiencies are taken from widely reported values for lithium-ion storage systems, while coefficients governing degradation are chosen to reflect typical aging behavior under moderate cycling. Parameters affecting PV performance, such as temperature coefficients, are selected to align with standard conventions for PV performance modeling. The weights in the unified objective function are selected to ensure a relatively balanced trade-off between the economic cost of energy, carbon impact, asset degradation, and indoor comfort. The parameters are normalized so that no single factor yields a dominant score, while allowing the relative weights to vary across simulations enables exploration of trade-offs between competing objectives. Parameters are chosen to be consistent (where possible) with those used in recent literature on smart home energy management and PV-integrated homes. Operating profiles for solar, load, prices, and temperature For each time step in the simulation, time-dependent profiles are constructed for solar irradiance, household electricity demand, electricity prices, and ambient temperature. Solar generation profiles are constructed based on a realistic daily generation pattern. The residential demand profiles are based on a model of typical household consumption. Throughout the simulations, ambient temperature affects PV thermal behavior and degradation, per the PV performance model, while PV generation, electricity demand, and ambient temperature profiles influence hourly electricity prices. To test the flexibility of various control strategies, the simulations are designed to run across a range of operating conditions defined by varying demand levels, solar supply, and priceability. For each operating condition, scenarios are simulated separately, and the results are consolidated to enable a comprehensive analysis of the system’s performance across a broad range of conditions. Baseline and benchmark methods The proposed MARL framework is evaluated against a set of benchmark methods representing different classes of energy management strategies: Proposed MARL: Multi-agent reinforcement learning with decentralized execution and degradation-aware reward formulation. Proposed MARL: Multi-agent reinforcement learning with decentralized execution and degradation-aware reward formulation. Independent Q-Learning (IQL): A decentralized MARL baseline in which each agent learns independently using only local observations and rewards. IQL serves as a decentralized multi-agent learning baseline without centralized coordination, enabling a direct evaluation of the advantages of the CTDE paradigm. Independent Q-Learning (IQL): A decentralized MARL baseline in which each agent learns independently using only local observations and rewards. IQL serves as a decentralized multi-agent learning baseline without centralized coordination, enabling a direct evaluation of the advantages of the CTDE paradigm. Simple RL: A single-agent reinforcement learning baseline without peer-to-peer coordination. Simple RL: A single-agent reinforcement learning baseline without peer-to-peer coordination. Genetic Algorithm (GA): A population-based heuristic optimization approach. Genetic Algorithm (GA): A population-based heuristic optimization approach. Particle Swarm Optimization (PSO): A swarm-based metaheuristic optimization method. Particle Swarm Optimization (PSO): A swarm-based metaheuristic optimization method. Greedy Strategy: A myopic baseline prioritizing immediate energy usage without long-term planning. Greedy Strategy: A myopic baseline prioritizing immediate energy usage without long-term planning. Optimal Proxy: A centralized lower-bound approximation used to contextualize achievable performance. Optimal Proxy: A centralized lower-bound approximation used to contextualize achievable performance. These benchmarks enable systematic comparison between learning-based, heuristic, and rule-based strategies under identical system configurations and objective functions. Our chosen set of baselines covers a wide variety of control paradigms from independent learning for multi-agent settings (IQL) to single-agent learning (Simple RL) through rule-based control (Greedy), heuristic optimization (GA and PSO), to a centralized lower-bound reference (Optimal Proxy). These baselines are commonplace in the literature for evaluating residential energy management systems. In addition, recent advances in multi-agent reinforcement learning, such as policy gradient-based and centralized-critic methods, have demonstrated strong performance in coordinated energy systems. While direct implementation of all such methods is beyond the scope of this study, the proposed framework is conceptually aligned with these approaches and extends them by incorporating design-aware PV modeling, degradation-aware optimization, and integrated multi-objective control. Optimal proxy baseline An optimal proxy is used to set lower bounds for evaluating the performance of decentralized learning, and the proxy is constructed under the assumptions of decentralization and learning in a centralized optimization problem, scoped to the full time horizon and fully foresighted operating profiles. As such, the optimal proxy serves as a theoretical bound on the performance of deployable control strategies, since learning must occur in decentralized systems. The proxy optimizes offline, fully leveraging and acting on knowledge of the profiles of solar generation and the demand of the involved households, as well as electricity pricing, and integrates all of that into a single objective. For all practical purposes, an optimal proxy-based solution is not feasible for real-time control design; however, it serves as a useful reference point for evaluating the performance of the proposed solutions relative to optimally coordinated operation under near-ideal conditions. The performance of the proposed MARL is best evaluated against this proxy to obtain meaningful measures of the level of coordination developed and how close the proposed solution is to the best. Performance evaluation metrics System performance assessment employs a variety of metrics aligned with the system’s unified operational objectives and long-term goals. Unified operational cost: Total operational cost comprised of grid energy cost, asset degradation penalties, comfort deviations, and P2P exchange incentives. Unified operational cost: Total operational cost comprised of grid energy cost, asset degradation penalties, comfort deviations, and P2P exchange incentives. Indoor comfort deviation: Total number of comfort deviations throughout the residential community. Indoor comfort deviation: Total number of comfort deviations throughout the residential community. Asset degradation cost: Total associated penalties due to the degradation of the PV systems and the ESS. Asset degradation cost: Total associated penalties due to the degradation of the PV systems and the ESS. Energy management dynamics: Changes over time of the ESS state-of-charge and net power exchange. Energy management dynamics: Changes over time of the ESS state-of-charge and net power exchange. Scalability: Change in performance with an increase in the size of the community. Scalability: Change in performance with an increase in the size of the community. Computational efficiency: Different control strategies in terms of average execution time and speed of convergence. Computational efficiency: Different control strategies in terms of average execution time and speed of convergence. Averaging was performed across the simulation runs for all metrics, and confidence intervals are provided to highlight statistical significance and support a fair comparison. Results and discussion This section outlines the evaluation of the proposed MARL framework under fully specified, diverse, but deterministic conditions. It covers the integration of economy- and system-level trade-offs, comfort-maintaining, degradation-deterring behavioral coupling, coordination, statistical shielding, and computational scalability. The proposed method is compared with IQL, Simple RL, GA, PSO, a greedy strategy, and an optimal proxy to assess the performance of coordinated CTDE-based learning against independent, heuristic, and rule-based baselines. The comparative performance against baseline optimization techniques, combined with the analytical evaluation of alternative MARL paradigms presented in Section 4.3, demonstrates the effectiveness and scalability of the proposed CTDE-based framework. System-level cost performance and comfort trade-offs Figures 2, 3, 4 and 5 illustrate the unified operational cost as a function of community size for different comfort-weight settings. Across all scenarios, the proposed MARL framework consistently achieves the lowest operational cost among decentralized strategies while maintaining a stable and narrow gap to the optimal proxy. Unified operational cost as a function of the number of prosumers. The proposed MARL approach demonstrates consistent cost reduction and scalability compared to baseline methods for comfort weight . Unified operational cost as a function of the number of prosumers for comfort weight . Unified operational cost as a function of the number of prosumers for comfort weight . Unified operational cost as a function of the number of prosumers for comfort weight . As the community size increases, MARL exhibits smooth and near-linear cost scaling, indicating effective decentralized coordination and mitigation of adverse scaling effects. The IQL algorithm is generally more costly than MARL across all community sizes, although it can be competitive (and sometimes better) than some heuristics, underscoring the value of centralized training to avoid non-stationarity and enable cooperative decision-making. IQL and Simple RL tend to perform poorly, with the former reflecting the limitations of independent multi-agent learning and the latter exhibiting volatile, non-monotonic behavior in medium- and large-sized communities. The GA and PSO are moderate performers but never outperform the MARL, and the greedy strategy incurs the highest cost because it is myopic. Aggregate cost comparisons under the base comfort scenario are shown in Fig. 6. MARL outperforms all decentralized baselines for all community sizes and remains close to the optimal proxy, demonstrating that cooperative learning can yield centralized-like performance. Aggregate unified operational cost comparison across algorithms for the base comfort scenario. The impact of comfort prioritization is visualized in Fig. 7, which shows unified operational cost across comfort-weight settings at maximum community size. MARL performs near-optimally across the entire spectrum, whereas Simple RL is significantly affected when comfort is prioritized. Heatmap of unified operational cost across algorithms for different comfort weight settings at maximum community size. This trend is quantified in Fig. 8, where MARL is consistently within about 20–25% of the optimal proxy, whereas heuristic and greedy methods rapidly degrade as comfort constraints tighten. Although we have chosen to compare to the core baselines, there are additional emergent MARL approaches that also show promise in similar domains. Our framework follows the common mantra of centralized training and decentralized execution, which is consistent with MARL approaches. Cost efficiency ratio of each algorithm relative to the optimal proxy across different comfort weight settings. Asset degradation, comfort preservation, and learned control behavior Apart from the economic performance, MARL offers advantages in asset preservation as well. Figure 9 shows the cumulated degradation costs of the assets under the base comfort setting. MARL incurs lower degradation costs than all benchmarks, which proves the advantage of degradation-aware operational control. IQL achieves greater comfort preservation than the single-agent and heuristic baselines, as shown in Fig. 10, but this still does not reach the level of the MARL algorithm, indicating that CTDE-based learning enables comfort-aware control coordination that independent local learning fails to achieve. The conclusions are twofold: not only are the MARL costs lower, but they also fall well within occupant comfort levels. Comparison of cumulative asset degradation cost across algorithms for the base comfort scenario. System-wide indoor discomfort comparison across algorithms as a function of the number of prosumers. The learned control behavior of MARL is shown in Fig. 11. Unlike simple RL methods, MARL uses smooth battery charging and discharging trajectories, avoiding deep cycles and saturation that lead to degradation. The IQL trajectories are also smoother and less coordinated than those of MARL, showing the stabilizing effect of centralized training. Complementarily, Fig. 12 shows the net power exchange behavior, where MARL dynamically finds its sweet spot among charging, discharging, and P2P energy sharing, whereas the greedy strategy tends to replicate extreme actions. Battery state-of-charge (SoC) trajectories under MARL, IQL, and Simple RL for the base comfort scenario. Net power exchange and action behavior under MARL and Greedy strategies for the base comfort scenario. Statistical reliability and computational scalability Figure 13 shows the expected unified operational cost with 95% confidence intervals. MARL consistently has tighter confidence bands, indicating stable convergence and reproducibility, whereas Simple RL exhibits much greater variance. We additionally present a comparison of episode-wise convergence behavior for both our proposed MARL framework and its IQL baseline in Fig. 14 under the base comfort scenario. Both MARL and IQL show that, while being stochastic throughout training, the smoothed episode cost trends ultimately will converge to a lower episode cost, with MARL consistently lower and more stable than IQL throughout the training horizon. This behavior emerges from the CTDE paradigm, in which the non-stationarity problem of independently learning agents is alleviated. Expected unified operational cost with 95% confidence intervals for the base comfort scenario. Training convergence of MARL and IQL under the base comfort scenario. The scalability of each method with respect to the number of prosumers is illustrated in Fig. 15. MARL scales approximately linearly with the number of prosumers and remains computationally feasible for medium- and large-scale residential communities. Overall, the results suggest that the MARL framework makes trade-offs among economic efficiency, battery lifetime, and occupant comfort while maintaining robustness and scalability. Compared with heuristic, greedy, and independent learning methods, MARL achieves near-optimal results through decentralized coordination and degradation-aware methods, demonstrating its suitability for intelligent PV-integrated residential energy systems. The results confirm the strong convergence stability and coordination capability of our CTDE-based MARL framework Average execution time of different algorithms as a function of the number of prosumers. . Practical implications The additional outcome of our framework has important real-world implications for residential energy for the following reasons. The lower operational cost indicates that household electricity bills can be reduced, and this effect may be more pronounced in a community of households under a high-PV-penetration scenario with dynamic pricing. Second, the inclusion of degradation-aware control strategies adds to the life expectancy of PV arrays and battery storage. Such economic implications are highly relevant for maintenance costs and replacements, as well as life-cycle energy planning in general. Third, the unified, synchronized actions enabled by the multi-agent infrastructure pave the way for efficient P2P energy exchange, thereby reducing reliance on the main electricity grid and enhancing local energy self-sufficiency in future decentralized energy markets and smart grid systems. Lastly, the identified scalability and robust performance as community sizes increase support the practical use of the proposed approach for medium- to large-scale residential energy communities, while decentralized execution in smart homes further underscores its feasibility by reducing communication overhead and enhancing user privacy. In summary, the good results from simulations clearly indicate effectiveness in intelligent and sustainable energy systems and point to the integration of these frameworks into new smart grids and energy community management systems. Security and communication considerations We consider a distributed cyber-physical system in which several agents communicate to exchange information and support their decision-making. Although the current results are formulated under the assumption of reliable communication, the cyber-physical system may be susceptible to data loss, communication delays, and cyberattacks. To defend against these potential attacks, various system-wide mitigations might be deployed. For example, secure communication and encryption might protect information shared amongst agents. CTDE, by virtue of its centralized training component, reduces the need for agents to share sensitive information during normal system operation, thereby limiting the attack surface. “Privacy preserving” methodologies of training, e.g., federated learning using secure aggregation, might further protect the general implementation since less raw data of agents is shared with other agents. Such considerations indicate that “cybersecurity” should be seen as a primary concern in the implementation of a future instantiation of our framework, specifically for smart energy systems. Conclusion This paper presented a unified multi-agent optimization framework for intelligent PV-integrated residential energy systems that jointly accounts for design-aware photovoltaic modeling, degradation-aware energy storage control, indoor comfort preservation, and peer-to-peer energy exchange. By formulating the community energy management problem as a multi-agent Markov game and solving it using a centralized training, decentralized execution multi-agent reinforcement learning (MARL) paradigm, the proposed framework enables scalable decentralized coordination while internalizing long-term asset health and human-centric comfort considerations. Within this unified setting, system-level performance improvements are achieved without relying on centralized control or full future information. Simulation results confirm that the proposed MARL framework consistently outperforms heuristic, greedy, and independent learning baselines. Under the evaluated case-study configurations, the unified operational cost is reduced by up to 69% relative to the greedy strategy and by more than 37% compared to simple RL, while remaining within approximately 20-25% of a centralized optimal proxy across all scenarios. In addition, MARL achieves lower asset degradation costs and maintains low indoor discomfort levels, with performance scaling smoothly as community size increases. Comparative analysis with Independent Q-Learning confirms that coordinated learning under the CTDE paradigm significantly enhances convergence stability, scalability, and overall system performance. Future research will focus on incorporating network-level constraints, enabling real-time market participation, and conducting field validation with real-world deployment data to further enhance practical applicability. Acknowledgements This work was partially supported by the Ministry of Science and ICT (MSIT), Korea, under the ICT Challenge and Advanced Network of HRD (ICAN) program (IITP-2024-RS-2022-00156345) supervised by the Institute of Information Communications Technology Planning Evaluation (IITP). This work was partially supported by the National Research Foundation of Korea (NRF) (grant numbers RS-2023-00219051 and RS-2023-00209107). Author contributions M.M.K. and M.U.A. carried out the conceptualization, the formal analysis and investigation, developed the methodology, and prepared the original draft. S.A. and M.N. contributed to the formal analysis and investigation. S.H.K. acquired the funding and was in charge of supervision. All authors reviewed the manuscript. Funding This work was partially supported by the Ministry of Science and ICT (MSIT), Korea, under the ICT Challenge and Advanced Network of HRD (ICAN) program (IITP-2024-RS-2022-00156345) supervised by the Institute of Information Communications Technology Planning Evaluation (IITP). This work was partially supported by the National Research Foundation of Korea (NRF) (grant numbers RS-2023-00219051 and RS-2023-00209107). Data availability All data generated or analysed during this study are included in this article. Competing interests The authors declare no competing interests. AI-assisted editing disclosure The authors used an AI-assisted language editing tool to improve the manuscript’s clarity, grammar, and readability; all scientific content, analysis, and conclusions remain the sole responsibility of the authors. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Muhammad Muneeb Khan and Muhammad Umair Ali contributed equally to this work. References Associated Data Data Availability Statement All data generated or analysed during this study are included in this article.