Continuous-Time Reinforcement Learning for Asset–Liability Management

Yilie Huang · 2025

Read Continuous-Time Reinforcement Learning for Asset–Liability Management by Yilie Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23280.txt.

Introduction

Balancing assets against liabilities over time is a control problem with money at stake. Yilie Huang casts asset-liability management as continuous-time reinforcement learning. The setup uses a linear-quadratic form with interim and terminal goals. A model-free soft actor-critic learns the strategy by policy gradient. The actor explores in an adaptive way, and the critic explores on a set schedule, which limits hand tuning.

Two hundred randomized market scenarios frame the tests. Rivals include enhanced classic strategies, a model-based RL method, and three modern RL algorithms. Huang's method earned higher average rewards than all of them. Gains came fast and held. The edge came from learning the strategy directly, not from bigger networks.

Catalog record

CREATORYilie Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-23280

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Continuous-Time Reinforcement Learning for Asset–Liability Management Yilie Huang Abstract . This paper proposes a novel approach for Asset–Liability Management (ALM) by employing continuous-time Reinforcement Learning (RL) with a linear-quadratic (LQ) formulation that incorporates both interim and terminal objectives. We develop a model-free, policy gradient-based soft actor-critic algorithm tailored to ALM for dynamically synchronizing assets and liabilities. To ensure an effective balance between exploration and exploitation with minimal tuning, we introduce adaptive exploration for the actor and scheduled exploration for the critic. Our empirical study evaluates this approach against two enhanced traditional financial strategies, a model-based continuous-time RL method, and three state-of-the-art RL algorithms. Evaluated across 200 randomized market scenarios, our method achieves higher average rewards than all alternative strategies, with rapid initial gains and sustained superior performance. The outperformance stems not from complex neural networks or improved parameter estimation, but from directly learning the optimal ALM strategy without learning the environment. 1. Introduction Asset-Liability Management (ALM) ( Van der Meer and Smink 1993 ) is a critical component of financial strategy, involving the careful coordination of assets and liabilities to ensure the financial health of institutions. It plays a crucial role for banks, insurance companies, and pension funds, where the alignment of assets against liabilities significantly affects financial stability and regulatory compliance. Traditionally, ALM has utilized a range of methods to effectively synchronize assets and liabilities: static approaches like cash flow matching ( Wise 1984 ) ensure liabilities are met with corresponding asset inflows, and passive value-driven strategies such as key rate duration matching ( Ho 1992 ) mitigate interest rate fluctuations by aligning the durations of assets and liabilities. On the other hand, more dynamic techniques, such as contingent immunization ( Leibowitz and Weinberger 1982 ; Leibowitz and Weinberger 1983 ) and Constant Proportion Portfolio Insurance (CPPI) ( Hakanoglu et al. 1989 ; Black and Perold 1992 ) , actively adjust asset allocation to maintain a target surplus or minimize deviation from a predefined target. Maintaining a target surplus is essential for balancing solvency and efficient capital use, helping to mitigate both the risk of insolvency from insufficient surplus and the inefficiency of excess capital. However, these traditional methods often assume a stable environment with complete information, which limits their adaptability in fast-changing market conditions. Reinforcement learning (RL) offers notable advantages in ALM by dynamically adjusting policies based on real-time feedback, making it a powerful tool for decision-making in dynamic and uncertain environments. Despite its potential, most RL methods have been developed around discrete-time Markov decision processes (MDPs) with discrete state and action spaces. While effective in many domains, this discrete-time framework faces inherent limitations when applied to systems that naturally evolve in continuous time, such as financial markets. Bridging this gap requires discretizing continuous-time problems into discrete-time models, enabling the use of standard RL algorithms. However, this discretization introduces critical challenges. Selecting an appropriate time step size is particularly challenging: large time steps can oversimplify the problem, reducing resolution and yielding suboptimal policies, while small time steps, despite their precision, increase the computational burden and may result in instability ( Munos 2006 ; Tallec et al. 2019 ; Park et al. 2021 ) . Additionally, discretization often struggles to accurately capture complex, fine-grained dynamics in continuous environments, especially in high-frequency decision-making contexts. This mismatch between discrete-time models and continuous-time dynamics limits the effectiveness of traditional RL methods in such settings, underscoring the need for continuous-time RL frameworks that can better handle the inherent complexities of financial markets. Recent advancements in continuous-time RL have marked a pivotal shift from earlier isolated studies ( Baird 1994 ; Doya 2000 ; Vamvoudakis and Lewis 2010 ) to a more cohesive and systematic framework. The introduction of an entropy-regularized control approach by ( Wang et al. 2020 ) laid a rigorous mathematical foundation for this field. Building on this, subsequent research ( Jia and Zhou 2022a ; Jia and Zhou 2022b ) developed comprehensive methods for policy evaluation and improvement. A key aspect of this research is its model-free, data-driven approach, which focuses on directly learning optimal control policies without requiring explicit model estimation. This progress has not only solidified the theoretical underpinnings of continuous-time RL but also inspired diverse extensions and practical applications in domains requiring robust real-time decision-making under uncertainty ( Huang et al. 2022 ; Tang and Zhou 2024 ; Huang et al. 2024 ; Wei and Yu 2023 ) . While recent developments in continuous-time RL have made significant theoretical progress, much of the existing work remains primarily focused on analytical results rather than empirical validation. To date, these methods have not been applied to ALM, a setting that naturally fits the continuous-time framework due to its dynamic and stochastic nature. Furthermore, empirical comparisons between continuous-time RL and traditional discrete-time RL approaches in financial applications remain largely unexplored, leaving open the question of their relative practical effectiveness. Building upon these theoretical advancements, our work mainly contributes in the following ways: • A linear-quadratic (LQ) formulation of the ALM problem is introduced, incorporating both interim and terminal objectives, and addressing the time-inconsistency and terminal constraint issues in the traditional mean-variance (MV) approach. • We design a model-free, policy gradient-based (soft) actor–critic algorithm tailored for the ALM problem within a continuous-time RL framework. To the best of our knowledge, this is the first work that applies continuous-time RL to ALM. • We introduce an adaptive exploration mechanism for the actor and a scheduled exploration strategy for the critic, enabling a robust exploration–exploitation trade-off throughout learning. • Empirical results demonstrate that our algorithm achieves superior performance over traditional ALM strategies, model-based continuous-time RL, and advanced RL baselines. The remainder of the paper is organized as follows. Section 2 formulates the ALM problem and discusses preliminary results essential for subsequent developments. Section 3 outlines our RL algorithm’s development and principles. Section 4 proves the convergence of the proposed algorithm. Section 5 evaluates our algorithm against six ALM strategies and presents experimental results. Finally, Section 6 concludes. 2. Description of the Asset-Liability Management Problem 2.1. Classical Stochastic LQ Framework for ALM One widely used approach to address the ALM problem is through stochastic control, particularly using the MV formulation, as demonstrated in several studies ( Shen et al. 2020 ; Hu et al. 2022 ; Pan et al. 2018 ) . In this framework, the state variable is typically defined as the surplus, representing the difference between assets and liabilities. The primary objective is to minimize the expected squared deviation of the terminal surplus from a predefined target surplus, subject to a terminal constraint. While the MV formulation provides a well-structured framework for ALM, it also presents several key limitations. First, MV-based formulations suffer from an inherent time-inconsistency issue, meaning that strategies deemed optimal at the outset may become suboptimal as time progresses ( Zhou and Li 2000 ) . Second, addressing the terminal constraint in MV formulations often requires additional techniques, such as introducing Lagrange multipliers, which can complicate the solution. Finally, by focusing exclusively on minimizing the terminal surplus deviation, the MV approach overlooks the importance of managing the surplus throughout the entire horizon, which is crucial for ensuring financial stability and meeting ongoing obligations. To address these limitations, we formulate

Read the complete text