Exploratory Mean-Variance Portfolio Optimization with Regime-Switching Market Dynamics

Yuling Max Chen, Bin Li, David Saunders · 2025

Read Exploratory Mean-Variance Portfolio Optimization with Regime-Switching Market Dynamics by Yuling Max Chen, Bin Li, David Saunders (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-16659.txt.

Introduction

Yuling Max Chen, Bin Li, and David Saunders wrote this finance paper in 2025. It studies how to split wealth across assets when the market flips between states, such as calm and stormy spells. The classic mean-variance aim trades gain against risk. The trio let a learning agent explore choices inside that aim, rather than solving it in one closed pass.

They state and prove a rule for when a new plan truly beats an old one. The usual temporal difference way of learning, they show, fits this task poorly. In its place they use a method built on a martingale trait of the best value path. In fake markets, the method found the true market traits, and on real data it earned the best mean yearly gains with fair calm. The work ties control theory to market learning.

Catalog record

CREATORYuling Max Chen, Bin Li, David Saunders
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-16659

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Exploratory Mean-Variance Portfolio Optimization with Regime-Switching Market Dynamics Yuling Max Chen, Bin Li, David Saunders Abstract Abstract Considering the continuous-time Mean-Variance (MV) portfolio optimization problem, we study a regime-switching market setting and apply reinforcement learning (RL) techniques to assist informed exploration within the control space. We introduce and solve the E xploratory M ean V ariance with R egime S witching (EMVRS) problem. We also present a Policy Improvement Theorem. Further, we recognize that the widely applied Temporal Difference (TD) learning is not adequate for the EMVRS context, hence we consider Orthogonality Condition (OC) learning, leveraging the martingale property of the induced optimal value function from the analytical solution to EMVRS. We design a RL algorithm that has more meaningful parameterization using the market parameters and propose an updating scheme for each parameter. Our empirical results demonstrate the superiority of OC learning over TD learning with a clear convergence of the market parameters towards their corresponding “grounding true” values in a simulated market scenario. In a real market data study, EMVRS with OC learning outperforms its counterparts with the highest mean and reasonably low volatility of the annualized portfolio returns. Keywords: Mean-Variance Portfolio Optimization, Regime Switching, Stochastic Control, Reinforcement Learning 1 Introduction Mean-Variance (MV) portfolio optimization has been widely studied since being introduced in Markowitz (1952a) . In the continuous-time setting, an investor with the classical MV objective aims to simultaneously maximize their expected portfolio value and minimize the portfolio volatility at the end of the investment horizon. This has been studied as a stochastic linear-quadratic problem in Zhou & Li (2000a) , followed by Chiu & Li (2006a) who solved for the MV efficient frontier, and Xie et al. (2008a) who derived the optimal investment policy under an incomplete market setting; see Zhang et al. (2018a) ; Kalayci et al. (2019a) for a broader overview of the the past work in the MV literature. Amongst many variants of the MV problem, this paper focuses on the regime-switching setting. Past work such as Maheu & McCurdy (2000a) ; Ang & Bekaert (2002a) ; Ang & Bekaert (2004a) have demonstrated that incorporating a regime-switching model better fits real market data and that ignoring market regimes in investment has a cost. Yin & Zhou (2004a) considered a discrete-time MV problem with regimes modelled on an aggregated process and asymptotically derived the optimal portfolio selection strategy from the optimal solution of its continuous-time counterpart. Wu & Li (2011a) ; Wu et al. (2014a) solved regime-switching MV portfolio allocation problems with an uncertain investment horizon, where the time to exit the market is modeled by a conditional distribution given the market regime. Recent research in stochastic control ( Wang & Zhou, 2020a ; Jiang et al., 2022a ; Jia & Zhou, 2022c ; Jia & Zhou, 2023a ; Dai et al., 2023a ; Denkert et al., 2024a ; Wu & Li, 2024a ) has shown the advantage of adopting RL techniques into classical stochastic control problems. The basic strategy is to employ the Stochastic RL Algorithm introduced in Gullapalli (1990a) , that replaces the optimal solution to the classical stochastic control problem with a probability distribution. By introducing stochasticity to the optimal control solution, we enable exploration within the control space, while maintaining exploitation towards optimality. This allows the induced optimal policy distribution to be more robust against the randomness of the dynamic environment. Wang et al. (2020a) introduced the “exploratory extension” of the portfolio value dynamic and solved for the corresponding exploratory optimal solution to the linear-quadratic problem, where they proved the asymptotic equivalence between the classical solution and the exploratory solution. Later, Wang & Zhou (2020a) introduced the Exploratory Mean-Variance (EMV) problem and solved for the optimal investment strategy, which is a probability distribution over the control space rather than a deterministic control function. In this paper, we study the RL-facilitated EMV problem by Wang & Zhou (2020a) in the context of a regime-switching process, named as the E xploratory M ean V ariance with R egime S witching (EMVRS) problem. The continuous-time MV portfolio optimization problem with regime-switching but without reinforcement learning has previously been studied by Zhou & Yin (2003a) . On top of that, we design a value-based RL algorithm that reparameterizes the analytical solution to the EMVRS problem as the value function. In contrast to defining the value function as a randomly initialized neural network or polynomial approximator such as in ( Jiang et al., 2017a ; Wen et al., 2021a ; Wu & Li, 2024a ) , we configure a RL model whose parameters all have practical meanings. Finally, we applied the martingale property of the deduced optimal EMVRS value function, specifically the Orthogonality Condition studied by Jia & Zhou (2022b) , to the updating scheme of the parameters in the RL algorithm. In contrast to TD learning, this achieves convergence to the true parameters in our simulation study. Wu & Li (2024a) also studied the RL-facilitated MV portfolio selection problem with a regime switching setting. Wu & Li (2024a) formulated the problem as a Partially Observable Markov Decision Process with two unobservable market regimes, estimating the regimes using the the Wonham filter (Eq. (10) of Wu & Li (2024a) ). There three major differences between our work and Wu & Li (2024a) . Firstly, Wu & Li (2024a) assume that the volatility is regime-independent, while we allow for regime-dependent volatility. Secondly, the RL algorithm in Wu & Li (2024a) requires the market parameters 1 1 1 The parameters of the market dynamics and regime dynamics correspond to \tilde{\sigma},\tilde{mu}_{i},\tilde{\lambda}_{i} for i=1,2 in Algorithm 1 of Wu & Li (2024a) . to be either given or estimated from the data, and selects the Martingale Loss (first introduced by Jia & Zhou (2022b) ) to update the RL model parameters. We notice from Jia & Zhou (2022b) that the Martingale Loss requires the knowledge of the true market parameters, which is unavailable in the real market. According to the Mean-Blur problem arguments in Luenberger (2013a) , it is possible that replacing the market parameters with their estimates can cause the RL model to update towards an erroneous target. Therefore, we choose the Orthogonality Condition Loss (also introduced by Jia & Zhou (2022b) ), that does not depend on the true market parameters while simultaneously applies the martingality properties of our analytical solutions. As a result, our RL algorithm does not require the knowledge of the market parameters, instead, it learns the market parameters through training. Thirdly, by parameterizing the RL model with the market parameters, we show in our simulation study that the parameters of our proposed RL model can converge to their corresponding true values, with randomly chosen starting point. The remainder of the paper is structured as follows. In Section 2 , we present background material and the problem setup. Then, we introduce the EMVRS problem, provide an analytical solution, and establish a Policy Improvement Theorem in Section 3 . In Section 4 , we present a reparameterization of the EMVRS problem, and various algorithms for updating the market parameters. We present numerical results on simulated and real market data in Section 5 . 2 2 2 The EMVRS model is developed in Python and the source

Read the complete text