A machine learning approach to risk based asset allocation in portfolio optimization Authors: Sanjay Agal, Krishna Raulji, Niyati Dhirubhai Odedra Published: 2025 ABSTRACT This paper introduces a novel machine learning framework for dynamic risk-based asset allocation that addresses fundamental limitations in traditional portfolio optimization methods. The proposed architecture integrates Long Short-Term Memory networks for volatility forecasting with differentiable risk budgeting layers and regime-switching mechanisms, enabling end-to-end training of portfolio weights under adaptive risk constraints. Unlike conventional approaches that rely on static risk budgets and historical covariance estimates, our methodology dynamically adjusts risk targets based on real-time market indicators, including volatility expectations, credit spreads, and yield curve dynamics. The framework achieves three primary research objectives: first, it demonstrates superior risk-adjusted performance with a Sharpe ratio of 1.38 during the out-of-sample period (2017-2022), representing a 55% improvement over traditional risk parity strategies and a 23% enhancement over contemporary deep learning approaches. Second, the architecture maintains computational efficiency through sparse attention mechanisms, scaling linearly with asset count while processing 50-asset portfolios in under 25 milliseconds. Third, the model preserves interpretability via SHAP-based risk attribution, providing transparent insights into allocation decisions across different market regimes. Empirical results reveal particularly strong performance during volatile market conditions, with maximum drawdowns reduced by 41% during stress periods compared to conventional methods. The framework’s proactive risk management capabilities were evidenced during the COVID-19 crisis, where it began reducing equity exposure two weeks before the market trough, demonstrating genuine predictive ability rather than reactive adjustment. Robustness checks confirm performance persistence under varying transaction costs, rebalancing frequencies, and alternative risk measures. These findings establish a new paradigm for portfolio optimization that successfully bridges theoretical finance with practical implementation. The framework’s ability to navigate complex market environments while maintaining computational efficiency and interpretability suggests readiness for widespread institutional adoption. This research contributes to the evolving literature on differentiable finance while providing portfolio managers with a robust tool for constructing adaptive, risk-aware investment strategies. Introduction Portfolio optimization represents a cornerstone of modern financial theory and practice, with its foundations established by the seminal mean-variance framework1. This paradigm introduced the quantitative trade-off between expected return and risk, formalized through quadratic optimization. Despite its theoretical elegance, the Markowitz approach has faced persistent challenges in practical implementation due to its sensitivity to estimation errors, assumption of stable market conditions, and inability to capture nonlinear dependencies in financial markets2. The evolution of risk-based allocation strategies, particularly risk parity approaches, attempted to address these limitations by focusing on risk contributions rather than return forecasts3. However, these methods still rely critically on historical covariance estimates and stable correlation structures, assumptions that frequently break down during market crises. The global financial crisis of 2008 and the COVID-19 pandemic of 2020 demonstrated the severe limitations of traditional approaches, with risk parity strategies experiencing unexpected drawdowns as correlations converged toward unity during stress periods4. The emergence of machine learning (ML) in financial economics has opened new avenues for addressing these fundamental challenges. Deep learning architectures, particularly those capable of capturing temporal dependencies and nonlinear relationships, have demonstrated remarkable success in various financial applications including volatility forecasting5, asset pricing6, and risk management7. Recent work by8 and9 has shown that neural networks can significantly improve portfolio performance by better capturing the complex dynamics of financial markets. Despite these advances, the integration of machine learning into risk-based portfolio construction remains underdeveloped. Current approaches often treat risk management as a secondary consideration or rely on static risk budgets that fail to adapt to changing market regimes10. This limitation becomes particularly critical in volatile environments where risk characteristics change rapidly and nonlinearities dominate. Furthermore, the black-box nature of many ML models creates challenges for interpretability and regulatory compliance, hindering widespread adoption in institutional settings11. Problem statement The central problem addressed in this research is the inability of conventional portfolio optimization methods to dynamically adapt risk allocations in response to changing market regimes. This problem manifests through three interconnected challenges: First, traditional risk-based strategies assume stationary covariance structures and stable risk dependencies, which contradicts empirical evidence of time-varying correlations and volatility clustering12. During stress periods, such as the 2008 financial crisis and the 2020 market crash, these assumptions break down catastrophically, leading to substantial losses precisely when risk management is most critical. Second, existing machine learning approaches to portfolio optimization often prioritize return prediction over risk management or treat risk constraints as secondary objectives13. This results in models that may generate attractive returns in backtests but fail to provide adequate risk control in live implementation. The lack of integrated risk budgeting within ML frameworks represents a significant gap in the current literature. Third, there exists a fundamental tension between model complexity and interpretability in ML-based financial applications. While sophisticated neural networks can capture complex market dynamics, their black-box nature creates challenges for regulatory approval and institutional adoption14. Current approaches either sacrifice performance for transparency or vice versa, without achieving an optimal balance. These challenges are compounded by the curse of dimensionality in high-dimensional portfolios, where estimation errors accumulate rapidly and traditional optimization methods become computationally prohibitive15. The problem is further exacerbated by the need for real-time implementation in institutional settings, where computational efficiency is as important as theoretical soundness. Research objectives This study aims to develop a comprehensive machine learning framework that addresses the aforementioned challenges through three primary research objectives: Objective 1: to design and implement an integrated neural architecture that jointly optimizes return forecasting and dynamic risk budgeting. The framework will incorporate LSTM networks for volatility forecasting, attention mechanisms for computational efficiency, and differentiable optimization layers for end-to-end training. This objective addresses the limitation of treating risk management as an afterthought in current ML approaches. Objective 2: to develop a regime-switching mechanism that dynamically adjusts risk targets based on market conditions. By incorporating leading indicators such as the VIX, TED spread, and yield curve dynamics, the model will adapt allocation strategies to prevailing market regimes, moving beyond the static risk budgets of conventional approaches16. Objective 3: to establish a methodology for maintaining model interpretability without sacrificing performance. Through the integration of SHAP value analysis and sparse attention mechanisms, the framework will provide transparent attribution of portfolio decisions while preserving the expressive power of deep learning architectures11. Additionally, the research seeks to achieve several secondary objectives:To validate the framework’s performance through comprehensive backtesting across multiple market cycles (2005-2022), including stress periods such as the 2008 financial crisis and COVID-19 pandemicTo demonstrate computational scalability to portfolios of 50+ assets, addressing the dimensionality challenges of traditional methodsTo assess robustness through sensitivity analysis incorporating transaction costs, alternative risk measures, and varying rebalancing frequenciesTo compare performance against established benchmarks including risk parity, deep portfolio approaches, and robust mean-variance optimizationThe successful achievement of these objectives will contribute to both academic literature and practical portfolio management by providing a theoretically sound, empirically validated, and computationally efficient framework for dynamic risk-based asset allocation. The research builds upon recent advances in differentiable finance17 and neural portfolio optimization while addressing critical gaps in regime adaptation, interpretability, and integrated risk management. To validate the framework’s performance through comprehensive backtesting across multiple market cycles (2005-2022), including stress periods such as the 2008 financial crisis and COVID-19 pandemic To validate the framework’s performance through comprehensive backtesting across multiple market cycles (2005-2022), including stress periods such as the 2008 financial crisis and COVID-19 pandemic To demonstrate computational scalability to portfolios of 50+ assets, addressing the dimensionality challenges of traditional methods To demonstrate computational scalability to portfolios of 50+ assets, addressing the dimensionality challenges of traditional methods To assess robustness through sensitivity analysis incorporating transaction costs, alternative risk measures, and varying rebalancing frequencies To assess robustness through sensitivity analysis incorporating transaction costs, alternative risk measures, and varying rebalancing frequencies To compare performance against established benchmarks including risk parity, deep portfolio approaches, and robust mean-variance optimization To compare performance against established benchmarks including risk parity, deep portfolio approaches, and robust mean-variance optimization Literature review The academic literature on portfolio optimization has evolved substantially since the foundational work of1, progressing from classical mean-variance frameworks to sophisticated machine learning approaches. This section provides a comprehensive review of this evolution, examining theoretical foundations, methodological advancements, and emerging trends in risk-based asset allocation. Foundations of risk-based asset allocation The theoretical underpinnings of modern portfolio optimization trace back to1’s seminal mean-variance framework, which established the quantitative relationship between expected returns and portfolio risk. The classical optimization problem can be formulated as:1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \max _w \left( w^\top \mu - \frac{\gamma }{2} w^\top \Sigma w \right) \end{aligned}$$\end{document}where w represents asset weights, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu$$\end{document} expected returns, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma$$\end{document} the covariance matrix, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma$$\end{document} the risk aversion parameter. While mathematically elegant, this framework suffers from well-documented limitations in practical implementation, particularly its sensitivity to estimation errors that grow quadratically with the number of assets2. The recognition of these limitations spurred the development of risk-based allocation strategies that minimize dependence on expected return estimates. Risk parity approaches, comprehensively surveyed by3, emerged as a prominent alternative by focusing on equal risk contributions across assets. The risk contribution of asset i is defined as:2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {RC}_i = w_i \frac{(\Sigma w)_i}{\sqrt{w^\top \Sigma w}} \end{aligned}$$\end{document}This paradigm shift from return-based to risk-based allocation represented significant theoretical progress, yet practical challenges remained. Traditional risk parity strategies still rely on historical covariance estimates and assume stable correlation structures, assumptions that frequently break down during market stress periods4. Conventional approaches to mitigate limitations A substantial body of work has developed sophisticated techniques to address the limitations of classical frameworks. These approaches can be broadly categorized into regime identification methods and robust estimation techniques. Regime identification and modeling: the recognition of structural breaks in financial markets led to the development of regime-switching models.18 introduced Hidden Markov Models (HMMs) for identifying latent market states, while12 developed Markov-switching GARCH models to capture time-varying volatility dynamics.16 extended these approaches to multivariate settings, demonstrating improved portfolio performance through regime-aware allocation strategies. However, these methods often struggle with high-dimensional portfolios and rely on parametric assumptions that may not capture the full complexity of market dynamics. Robust covariance estimation: to address estimation error in covariance matrices, several shrinkage techniques have been proposed.19 introduced a well-conditioned estimator that shrinks the sample covariance matrix toward a structured target, significantly improving out-of-sample performance. Rolling-window estimation methods provide adaptability to changing market conditions but introduce a latency trade-off that can be detrimental during rapid regime shifts. Despite these advancements, conventional approaches remain limited by their linearity assumptions and inability to capture complex nonlinear dependencies. Machine learning in portfolio optimization The application of machine learning to portfolio optimization represents a paradigm shift from traditional econometric methods. Early work by20 demonstrated that deep learning architectures could capture complex patterns in financial data that elude linear models. Guet al.8 established theoretical foundations for neural portfolio optimization, proving that deep networks can approximate the optimal portfolio function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f^*: \mathscr {X} \rightarrow \mathbb {R}^p$$\end{document} with bounded error:3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Vert f_\theta - f^*\Vert _{L_2} \le C n^{-\alpha /d} \end{aligned}$$\end{document}where n is sample size, d input dimension, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} the smoothness parameter. Recent advances have focused on specific aspects of the portfolio optimization problem: Volatility forecasting: Chong et al.5 demonstrated that LSTM architectures reduce volatility forecasting errors by 32% compared to traditional GARCH models, providing more accurate risk estimates for portfolio construction. Their work highlighted the importance of capturing long-term dependencies in financial time series, a capability where recurrent neural networks excel. High-dimensional settings: Bianchi et al.15 addressed the curse of dimensionality by proving convergence of random forest-based portfolio weights when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p/n \rightarrow c \in (0,1)$$\end{document}, providing theoretical justification for ML methods in high-dimensional settings. This work opened new possibilities for handling portfolios with hundreds of assets, a scenario where traditional methods typically fail. End-to-end optimization: most contemporaneously, Uysal et al.17 proposed an end-to-end neural architecture with differentiable risk budgeting layers, demonstrating significant performance improvements over traditional methods. Their approach integrates risk management directly into the optimization process, representing a substantial advance over two-stage approaches that separate return prediction from risk estimation. Neural networks and differentiable programming The emergence of differentiable programming has revolutionized portfolio optimization by enabling end-to-end training of complex systems. Chen et al.9 introduced deep risk parity portfolios that learn risk contributions directly from data, while21 developed neural networks for dynamic risk budgeting that adapt to changing market conditions. These approaches leverage the expressive power of deep learning while maintaining the theoretical foundations of risk-based allocation. Attention mechanisms, originally developed for natural language processing, have found increasing application in financial modeling. Lin et al.22 demonstrated that self-attention networks can capture complex dependencies in asset returns, providing superior performance compared to traditional time series models. The sparse attention variant employed in our work builds upon these foundations while addressing computational constraints through selective information processing. Explainability and interpretability The black-box nature of neural networks has been a significant barrier to adoption in institutional settings. Hansen et al.11 pioneered the application of SHAP (SHapley Additive exPlanations) values to portfolio optimization, providing post-hoc interpretability for complex models. Chaudhuri et al.14 extended this work to risk attribution, enabling portfolio managers to understand the drivers of model decisions. These advances in explainable AI (XAI) are crucial for regulatory compliance and practical implementation. Research gaps and theoretical limitations Despite substantial progress, several critical gaps remain in the literature: Dynamic risk targeting: existing ML approaches often employ static risk budgets that fail to adapt to changing market conditions, violating the adaptive markets hypothesis23. While10 demonstrated the importance of regime awareness, their approach relies on pre-specified regime definitions rather than learning regimes directly from data. Computational scalability: methods proposed by8,9 often require \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O(p^3)$$\end{document} operations, becoming computationally prohibitive for portfolios with more than 50 assets. This limitation restricts practical application in institutional settings where portfolios typically contain hundreds of securities. Integrated risk-return optimization: current literature tends to prioritize either return prediction or risk management, with few approaches successfully integrating both objectives. The separation of these tasks ignores their inherent interdependence and may lead to suboptimal solutions. Real-world implementation constraints: many proposed methods neglect practical considerations such as transaction costs, liquidity constraints, and regulatory requirements. The gap between theoretical performance and implementable strategies remains substantial. Theoretical foundations: while empirical results are promising, the theoretical properties of neural portfolio optimizers remain underdeveloped. Convergence guarantees, generalization bounds, and robustness properties require further investigation to build confidence in these methods. Our work addresses these gaps through several key innovations: a differentiable risk budgeting layer that enables end-to-end optimization; sparse attention mechanisms that reduce computational complexity to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O(p \log p)$$\end{document}; dynamic risk targets that adapt to market regimes; and comprehensive treatment of implementation constraints including transaction costs and regulatory requirements. By bridging these gaps, we aim to advance both the theoretical understanding and practical application of machine learning in portfolio optimization. The subsequent sections detail our methodological approach, which builds upon these literature foundations while addressing the identified limitations through novel architectural choices and rigorous empirical validation. Methodology This section presents the comprehensive methodological framework developed for risk-based asset allocation using machine learning techniques. The proposed approach integrates advanced neural architectures with financial theory to create a dynamic portfolio optimization system that adapts to changing market conditions while maintaining computational efficiency and interpretability. Theoretical framework and model overview The foundation of our methodology rests on the integration of modern portfolio theory with deep learning capabilities. We extend the classical risk parity framework by introducing dynamic risk budgeting that responds to market regime signals. The core optimization problem is formulated as a multi-objective function balancing expected returns, portfolio variance, and tail risk measures:4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \underset{w_t}{\text {minimize}} \ \mathbb {E}[R_p] + \lambda _1 \sigma ^2_p + \lambda _2 \text {MDD}_p + \lambda _3 \Vert {\bf w}_t - {\bf w}_{t-1}\Vert _1 \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_t$$\end{document} represents time-varying portfolio weights, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma ^2_p$$\end{document} denotes portfolio variance, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {MDD}_p$$\end{document} captures maximum drawdown, and the final term penalizes excessive turnover to control transaction costs. The novelty of our approach lies in the integration of three complementary components: (1) a volatility forecasting module using LSTM networks, (2) a regime detection mechanism based on market state variables, and (3) a differentiable risk budgeting layer that enables end-to-end training. This integrated architecture allows the model to simultaneously learn from historical patterns while adapting to current market conditions. The proposed machine learning framework is illustrated in Fig. 1.Fig. 1Comprehensive architecture of the proposed machine learning framework for dynamic risk-based asset allocation. The system processes multiple data sources through specialized modules for feature extraction, regime detection, and volatility forecasting, culminating in an optimized portfolio construction layer with integrated risk constraints. Comprehensive architecture of the proposed machine learning framework for dynamic risk-based asset allocation. The system processes multiple data sources through specialized modules for feature extraction, regime detection, and volatility forecasting, culminating in an optimized portfolio construction layer with integrated risk constraints. Data collection and preprocessing framework Our data framework employs a multi-source approach combining asset returns with macroeconomic and market-based state variables. The dataset spans January 2005 to December 2022, covering multiple market cycles including the global financial crisis, European debt crisis, COVID-19 pandemic, and the 2022 inflationary period. Asset universe composition The investment universe comprises 45 instruments selected to represent major asset classes while maintaining practical relevance for institutional portfolios:Equities (30 assets): global stocks covering 95% of market capitalization across developed and emerging markets, including sector diversificationFixed income (10 assets): government and corporate bonds representing 82% of the Bloomberg Global Aggregate Index across duration and credit quality spectraCommodities (5 assets): major commodities accounting for 89% of futures trading volume, including energy, metals, and agricultural productsThis composition ensures sufficient diversification while maintaining computational tractability. The selection criteria follow institutional standards as documented in10. Equities (30 assets): global stocks covering 95% of market capitalization across developed and emerging markets, including sector diversification Equities (30 assets): global stocks covering 95% of market capitalization across developed and emerging markets, including sector diversification Fixed income (10 assets): government and corporate bonds representing 82% of the Bloomberg Global Aggregate Index across duration and credit quality spectra Fixed income (10 assets): government and corporate bonds representing 82% of the Bloomberg Global Aggregate Index across duration and credit quality spectra Commodities (5 assets): major commodities accounting for 89% of futures trading volume, including energy, metals, and agricultural products Commodities (5 assets): major commodities accounting for 89% of futures trading volume, including energy, metals, and agricultural products State variables and feature engineering Beyond asset returns, we incorporate state variables that serve as leading indicators for market regimes and risk conditions:5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\bf x}_t = [\text {VIX}_t, \text {TED}_t, \Delta \text {Yield}_{t}^{(10Y-3M)}, \text {MktCap}_t^{1/2}, \text {Momentum}_t, \text {Liquidity}_t] \end{aligned}$$\end{document}These variables capture volatility expectations (VIX), credit risk (TED spread), yield curve dynamics, market size effects, price trends, and liquidity conditions. Each variable undergoes rigorous preprocessing including winsorization at the 1st and 99th percentiles, standardization, and missing value imputation using the expectation-maximization algorithm4. Return calculation and adjustment Excess returns are computed relative to the risk-free rate following standard practice in portfolio optimization literature2:6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} r_{i,t} = \frac{P_{i,t} - P_{i,t-1}}{P_{i,t-1}} - r_{f,t} \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P_{i,t}$$\end{document} denotes the price of asset i at time t, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r_{f,t}$$\end{document} represents the contemporaneous risk-free rate. Returns are adjusted for corporate actions, dividends, and currency effects where applicable. Machine learning architecture specifications The neural architecture combines temporal modeling, attention mechanisms, and financial constraints in an integrated framework. Each component is designed to address specific limitations of traditional portfolio optimization methods. Volatility forecasting module We employ a multi-layer LSTM network for conditional covariance matrix estimation, building upon the architecture proposed by5 but with enhancements for financial time series:7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} h_t^{\text {LSTM}}&= \sigma (W_h [h_{t-1},{ \bf x}_t] + b_h) \end{aligned}$$\end{document}8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{\Sigma }_t&= \text {Softplus}(W_\Sigma h_t^{\text {LSTM}} + b_\Sigma ) \end{aligned}$$\end{document}The LSTM configuration includes 256 hidden units per layer with layer normalization and dropout regularization (rate=0.3) to prevent overfitting. The Softplus activation ensures positive definiteness of the covariance matrix estimate, a crucial requirement for portfolio optimization. Regime detection and adaptation mechanism Market regime identification follows a hybrid approach combining unsupervised learning with economic intuition. We employ Gaussian Mixture Models (GMM) with Bayesian information criterion for regime count selection:9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p({\bf x}_t) = \sum _{k=1}^K \pi _k \mathscr {N}({\bf x}_t | \mu _k, \Sigma _k) \end{aligned}$$\end{document}The detected regimes dynamically adjust risk targets through a gating mechanism:10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \beta _t = \sum _{k=1}^K \gamma _k \cdot {\bf b}_k, \quad \gamma _k = \text {Softmax}(W_g [h_t^{\text {LSTM}}, {\bf x}_t]) \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\bf b}_k$$\end{document} represents regime-specific risk budgets and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma _k$$\end{document} denotes the gating weights. Differentiable risk budgeting layer The core innovation of our methodology is the differentiable optimization layer that solves the risk budgeting problem with gradient-based methods. Following recent advances in differentiable programming17, we implement the risk parity conditions as a constrained optimization problem:11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \min _{{\bf w}_t} \left\| \frac{{\bf w}_t \circ (\hat{\Sigma }_t {\bf w}_t)}{{\bf w}_t^\top \hat{\Sigma }_t {\bf w}_t} - {\bf b} \right\| _2^2 + \lambda _1 \Vert {\bf w}_t\Vert _1 + \lambda _2 \Vert {\bf w}_t - {\bf w}_{t-1}\Vert _2^2 \end{aligned}$$\end{document}This formulation ensures adherence to target risk contributions while controlling for turnover and sparsity. The differentiability of the layer enables end-to-end training through backpropagation. Sparse attention mechanism To address computational complexity in high-dimensional settings, we implement a sparse attention mechanism inspired by22:12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \alpha _{ij} = \frac{\exp (q_i^\top k_j / \sqrt{d})}{\sum _{l \in \mathscr {N}(i)} \exp (q_i^\top k_l / \sqrt{d})}, \quad \text {with } |\mathscr {N}(i)| \le k \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathscr {N}(i)$$\end{document} denotes the neighborhood of asset i limited to the k most relevant assets. This reduces computational complexity from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O(p^2)$$\end{document} to O(pk) while maintaining modeling flexibility. Training protocol and regularization strategy The training process employs a rigorous walk-forward validation scheme to ensure temporal robustness and prevent data leakage. This approach mirrors real-world investment processes where models are periodically retrained on expanding windows. Loss function design The composite loss function balances multiple objectives relevant to practical portfolio management:13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathscr {L}(\theta ) = -\frac{\mathbb {E}[R_p]}{\sqrt{\text {Var}(R_p)}} + \gamma _1 \text {MDD}(R_p) + \gamma _2 \text {Turnover} + \gamma _3 \Vert \theta \Vert _2^2 \end{aligned}$$\end{document}The Sharpe ratio component maximizes risk-adjusted returns, while the maximum drawdown term penalizes extreme losses. The turnover regularization controls trading costs, and the L2 penalty prevents overfitting. Walk-forward validation scheme Following best practices in financial forecasting24, we implement a rolling window approach:Training window: 5 years of monthly data (60 observations)Validation window: 1 year (12 observations) for hyperparameter tuningTesting window: 1 year (12 observations) for out-of-sample evaluationThe window rolls forward monthly, ensuring that the model is always tested on unseen data while incorporating the most recent market information. Training window: 5 years of monthly data (60 observations) Training window: 5 years of monthly data (60 observations) Validation window: 1 year (12 observations) for hyperparameter tuning Validation window: 1 year (12 observations) for hyperparameter tuning Testing window: 1 year (12 observations) for out-of-sample evaluation Testing window: 1 year (12 observations) for out-of-sample evaluation Regularization and overfitting mitigation Given the high dimensionality and noise characteristics of financial data, we employ multiple regularization techniques:Dropout: rate of 0.3 applied to LSTM layers to prevent co-adaptationEarly stopping: training terminates when validation loss fails to improve for 10 consecutive epochsLabel smoothing: applied to regime classification to prevent overconfidenceGradient clipping: norm-based clipping at 1.0 to stabilize trainingThese techniques collectively ensure that the model learns robust patterns rather than memorizing noise. Dropout: rate of 0.3 applied to LSTM layers to prevent co-adaptation Dropout: rate of 0.3 applied to LSTM layers to prevent co-adaptation Early stopping: training terminates when validation loss fails to improve for 10 consecutive epochs Early stopping: training terminates when validation loss fails to improve for 10 consecutive epochs Label smoothing: applied to regime classification to prevent overconfidence Label smoothing: applied to regime classification to prevent overconfidence Gradient clipping: norm-based clipping at 1.0 to stabilize training Gradient clipping: norm-based clipping at 1.0 to stabilize training Hyperparameter optimization We employ Bayesian optimization with Tree-structured Parzen Estimator (TPE) for efficient hyperparameter search. The optimization space includes:14\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Theta = \{\lambda _1 \in [0,1], \lambda _2 \in [0.5,2], k \in \{10,15,\dots ,30\}, \text {dropout} \in [0.1,0.5], \text {learning rate} \in [10^{-5},10^{-3}]\} \end{aligned}$$\end{document}The optimization process runs for 100 trials, with the best configuration selected based on validation set performance. Table 1 summarizes the final hyperparameter values. Benchmark models and evaluation framework To ensure comprehensive evaluation, we compare against three established benchmarks representing different approaches to portfolio optimization. Traditional Risk Parity Implemented following3’s equal risk contribution approach, solved via sequential quadratic programming with convergence tolerance \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon =10^{-6}$$\end{document}. Deep portfolio network Based on8’s architecture, featuring a 3-layer feedforward network with ReLU activations and batch normalization. Robust mean-variance optimization Incorporating Ledoit-Wolf shrinkage with optimal shrinkage intensity determined via cross-validation19. Performance metrics Evaluation employs multiple metrics to capture different aspects of portfolio performance:15\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {Sharpe Ratio}&= \frac{\sqrt{252}\mu _{\text {daily}}}{\sigma _{\text {daily}}} \end{aligned}$$\end{document}16\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {Maximum Drawdown}&= \max _{0\le \tau \le t}\left( 1-\frac{P_t}{P_\tau }\right) \end{aligned}$$\end{document}17\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {Turnover}&= \frac{1}{T}\sum _{t=1}^T \Vert {\bf w}_t - {\bf w}_{t-1}\Vert _1 \end{aligned}$$\end{document}18\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {Calmar Ratio}&= \frac{\text {Annualized Return}}{\text {Maximum Drawdown}} \end{aligned}$$\end{document}Statistical significance is assessed using Diebold–Mariano tests with Newey-West standard errors to account for autocorrelation. Implementation details The model is implemented in PyTorch 2.0 with custom CUDA kernels for efficient sparse attention computation. Training utilizes NVIDIA A100 GPUs with mixed precision to accelerate computation while maintaining numerical stability. The codebase follows modular design principles to facilitate reproducibility and extension. This comprehensive methodological framework provides a robust foundation for dynamic risk-based asset allocation, balancing theoretical rigor with practical implementation considerations. The subsequent sections present empirical results validating the approach across various market conditions. Experimental setup This section delineates the comprehensive experimental framework designed to validate the proposed methodology against the research objectives outlined in Section 1. The experimental design rigorously addresses the core research questions through systematic data partitioning, robust benchmarking, and multi-faceted evaluation metrics that collectively ensure the validity and reliability of our findings. Experimental design philosophy The experimental framework is constructed around three fundamental principles that align directly with our research objectives. First, we prioritize temporal robustness through walk-forward validation to assess the model’s adaptability across different market regimes. Second, we emphasize comparative rigor through multiple benchmark strategies representing state-of-the-art approaches. Third, we ensure practical relevance by incorporating real-world constraints including transaction costs, liquidity considerations, and computational efficiency requirements. The experimental design specifically targets the validation of our three primary research objectives: Dynamic risk targeting capability during regime shiftsComputational scalability for high-dimensional portfoliosMaintained interpretability without performance sacrificeEach component of the experimental setup is meticulously crafted to provide evidence for these objectives through quantitative metrics and statistical testing (Fig. 2).Fig. 2Comprehensive experimental design framework illustrating the sequential workflow from data partitioning through model evaluation. The design incorporates multiple validation pathways to ensure robust assessment of each research objective. Dynamic risk targeting capability during regime shifts Dynamic risk targeting capability during regime shifts Computational scalability for high-dimensional portfolios Computational scalability for high-dimensional portfolios Maintained interpretability without performance sacrifice Maintained interpretability without performance sacrifice Comprehensive experimental design framework illustrating the sequential workflow from data partitioning through model evaluation. The design incorporates multiple validation pathways to ensure robust assessment of each research objective. Data partitioning strategy The temporal division of data follows a rigorous walk-forward protocol that mirrors real-world investment processes while maintaining statistical integrity. The partitioning strategy is explicitly designed to test the model’s ability to adapt to evolving market conditions, directly addressing Research Objective 1 (dynamic risk targeting). Chronological segmentation The dataset spanning January 2005 to December 2022 is partitioned into three distinct periods with non-overlapping objectives:Initial training period (2005-2014): this 10-year window serves as the foundational training set, encompassing the global financial crisis and subsequent recovery. The extended duration ensures exposure to diverse market conditions while providing sufficient data for model convergence.Validation and hyperparameter tuning period (2015-2016): a 2-year window dedicated exclusively to hyperparameter optimization and model selection. This period represents a relatively stable market environment, allowing for clean evaluation of architectural choices without extreme regime shifts.Out-of-sample testing period (2017-2022): a 6-year comprehensive testing window that includes multiple distinct market regimes: the late-cycle bull market (2017-2019), COVID-19 crisis (2020), stimulus-driven recovery (2021), and high-inflation environment (2022). This diverse period provides the ultimate test of model robustness.The walk-forward mechanism operates with monthly rebalancing, where the training window expands incrementally while maintaining a fixed 10-year lookback period. This approach ensures that the model continuously incorporates recent market information while preserving long-term patterns. Initial training period (2005-2014): this 10-year window serves as the foundational training set, encompassing the global financial crisis and subsequent recovery. The extended duration ensures exposure to diverse market conditions while providing sufficient data for model convergence. Initial training period (2005-2014): this 10-year window serves as the foundational training set, encompassing the global financial crisis and subsequent recovery. The extended duration ensures exposure to diverse market conditions while providing sufficient data for model convergence. Validation and hyperparameter tuning period (2015-2016): a 2-year window dedicated exclusively to hyperparameter optimization and model selection. This period represents a relatively stable market environment, allowing for clean evaluation of architectural choices without extreme regime shifts. Validation and hyperparameter tuning period (2015-2016): a 2-year window dedicated exclusively to hyperparameter optimization and model selection. This period represents a relatively stable market environment, allowing for clean evaluation of architectural choices without extreme regime shifts. Out-of-sample testing period (2017-2022): a 6-year comprehensive testing window that includes multiple distinct market regimes: the late-cycle bull market (2017-2019), COVID-19 crisis (2020), stimulus-driven recovery (2021), and high-inflation environment (2022). This diverse period provides the ultimate test of model robustness. Out-of-sample testing period (2017-2022): a 6-year comprehensive testing window that includes multiple distinct market regimes: the late-cycle bull market (2017-2019), COVID-19 crisis (2020), stimulus-driven recovery (2021), and high-inflation environment (2022). This diverse period provides the ultimate test of model robustness. Regime-based analysis partitions Beyond chronological partitioning, we implement regime-specific analysis to directly evaluate dynamic adaptation capabilities. Market regimes are classified using a combination of volatility thresholds and economic indicators:19\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {Regime}_t = {\left\{ \begin{array}{ll} \text {Low Volatility} & \text {if VIX}_t< 15 \\ \text {Medium Volatility} & \text {if } 15 \le \text {VIX}_t < 25 \\ \text {High Volatility} & \text {if VIX}_t \ge 25 \end{array}\right. } \end{aligned}$$\end{document}This classification enables regime-specific performance analysis, particularly crucial for validating the model’s behavior during stress periods versus calm markets (Fig. 3).Fig. 3Visualization of the temporal data partitioning strategy with regime classification. The upper panel shows the chronological division, while the lower panel illustrates regime transitions throughout the test period. Visualization of the temporal data partitioning strategy with regime classification. The upper panel shows the chronological division, while the lower panel illustrates regime transitions throughout the test period. Computational infrastructure The hardware and software configuration is optimized for both research reproducibility and practical implementation feasibility, directly addressing Research Objective 2 (computational scalability). Hardware specifications Primary compute: NVIDIA A100 GPUs with 40GB HBM2 memory, enabling parallel processing of high-dimensional covariance matricesCPU configuration: AMD EPYC 7742 processors with 64 cores to support data preprocessing and feature engineering pipelinesMemory architecture: 512GB DDR4 RAM with NVMe storage for efficient handling of large financial datasetsNetwork infrastructure: 100 GbE connectivity for distributed training scenarios Primary compute: NVIDIA A100 GPUs with 40GB HBM2 memory, enabling parallel processing of high-dimensional covariance matrices Primary compute: NVIDIA A100 GPUs with 40GB HBM2 memory, enabling parallel processing of high-dimensional covariance matrices CPU configuration: AMD EPYC 7742 processors with 64 cores to support data preprocessing and feature engineering pipelines CPU configuration: AMD EPYC 7742 processors with 64 cores to support data preprocessing and feature engineering pipelines Memory architecture: 512GB DDR4 RAM with NVMe storage for efficient handling of large financial datasets Memory architecture: 512GB DDR4 RAM with NVMe storage for efficient handling of large financial datasets Network infrastructure: 100 GbE connectivity for distributed training scenarios Network infrastructure: 100 GbE connectivity for distributed training scenarios Software environment Deep learning framework: PyTorch 2.0 with CUDA 11.7 support, leveraging compiled kernels for sparse attention operationsOptimization libraries: custom differentiable programming layers built on top of PyTorch’s autograd engineFinancial computing: integrated with Bloomberg API for real-time data ingestion and validationReproducibility measures: containerization using Docker with version-controlled environment specificationsThe infrastructure design ensures that performance metrics reflect realistic implementation scenarios rather than idealized laboratory conditions. Deep learning framework: PyTorch 2.0 with CUDA 11.7 support, leveraging compiled kernels for sparse attention operations Deep learning framework: PyTorch 2.0 with CUDA 11.7 support, leveraging compiled kernels for sparse attention operations Optimization libraries: custom differentiable programming layers built on top of PyTorch’s autograd engine Optimization libraries: custom differentiable programming layers built on top of PyTorch’s autograd engine Financial computing: integrated with Bloomberg API for real-time data ingestion and validation Financial computing: integrated with Bloomberg API for real-time data ingestion and validation Reproducibility measures: containerization using Docker with version-controlled environment specifications Reproducibility measures: containerization using Docker with version-controlled environment specifications Hyperparameter optimization framework The hyperparameter search process employs a multi-stage approach that balances exploration efficiency with convergence reliability. The optimization strategy is specifically designed to prevent overfitting to specific market regimes while ensuring robust out-of-sample performance. Bayesian optimization protocol We implement a tree-structured Parzen estimator (TPE) approach with the following configuration:Search space: 100 trials with adaptive resource allocation based on intermediate performanceObjective function: weighted combination of validation Sharpe ratio and maximum drawdownEarly stopping: hyperband pruning for underperforming configurations after 20 epochsParallelization: simultaneous evaluation of 4 configurations across GPU clusterThe hyperparameter search space encompasses architectural parameters, regularization strengths, and optimization settings:Table 1Hyperparameter search space specifications.ParameterRangeSearch typeLSTM hidden units[64, 512]LogarithmicAttention heads[2, 8]IntegerSparsity factor (k)[10, 30]IntegerLearning rate[1e-5, 1e-3]LogarithmicDropout rate[0.1, 0.5]UniformL1 regularization[0.001, 0.1]LogarithmicBatch size[32, 256]Power-of-2 Search space: 100 trials with adaptive resource allocation based on intermediate performance Search space: 100 trials with adaptive resource allocation based on intermediate performance Objective function: weighted combination of validation Sharpe ratio and maximum drawdown Objective function: weighted combination of validation Sharpe ratio and maximum drawdown Early stopping: hyperband pruning for underperforming configurations after 20 epochs Early stopping: hyperband pruning for underperforming configurations after 20 epochs Parallelization: simultaneous evaluation of 4 configurations across GPU cluster Parallelization: simultaneous evaluation of 4 configurations across GPU cluster Table 1Hyperparameter search space specifications.ParameterRangeSearch typeLSTM hidden units[64, 512]LogarithmicAttention heads[2, 8]IntegerSparsity factor (k)[10, 30]IntegerLearning rate[1e-5, 1e-3]LogarithmicDropout rate[0.1, 0.5]UniformL1 regularization[0.001, 0.1]LogarithmicBatch size[32, 256]Power-of-2 Hyperparameter search space specifications. Cross-validation strategy Given the temporal nature of financial data, we employ a purged walk-forward cross-validation approach that accounts for autocorrelation and prevents data leakage:20\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} CV_{\text {score}} = \frac{1}{K} \sum _{k=1}^K \text {Performance}(\text {Test}_k | \text {Train}_k) \end{aligned}$$\end{document}where each fold maintains temporal ordering with appropriate gaps between training and testing periods to prevent lookahead bias (Fig. 4).Fig. 4Visualization of the hyperparameter optimization process showing convergence patterns and performance landscape across different parameter combinations. Visualization of the hyperparameter optimization process showing convergence patterns and performance landscape across different parameter combinations. Benchmark models implementation The comparative analysis includes three carefully selected benchmark models representing distinct approaches to portfolio optimization. Each benchmark is implemented with best-practice configurations to ensure fair comparison. Risk parity implementation The traditional risk parity strategy follows the equal risk contribution principle with the following specifications:Optimization method: Sequential quadratic programming with analytical gradientsCovariance estimation: Ledoit–Wolf shrinkage estimator with monthly recalibrationConstraints: full investment (weights sum to 1) with no leverageRebalancing: monthly with 10 basis points transaction cost assumption Optimization method: Sequential quadratic programming with analytical gradients Optimization method: Sequential quadratic programming with analytical gradients Covariance estimation: Ledoit–Wolf shrinkage estimator with monthly recalibration Covariance estimation: Ledoit–Wolf shrinkage estimator with monthly recalibration Constraints: full investment (weights sum to 1) with no leverage Constraints: full investment (weights sum to 1) with no leverage Rebalancing: monthly with 10 basis points transaction cost assumption Rebalancing: monthly with 10 basis points transaction cost assumption Deep portfolio network The neural network benchmark follows the architecture proposed in recent literature with enhancements for fair comparison:Architecture: 3-layer feedforward network with 256 units per layerActivation: ReLU with batch normalization and residual connectionsInput features: identical to our model’s feature set for direct comparabilityTraining: Adam optimizer with identical walk-forward validation scheme Architecture: 3-layer feedforward network with 256 units per layer Architecture: 3-layer feedforward network with 256 units per layer Activation: ReLU with batch normalization and residual connections Activation: ReLU with batch normalization and residual connections Input features: identical to our model’s feature set for direct comparability Input features: identical to our model’s feature set for direct comparability Training: Adam optimizer with identical walk-forward validation scheme Training: Adam optimizer with identical walk-forward validation scheme Robust mean-variance optimization The traditional optimization approach incorporates modern regularization techniques:Estimation: James–Stein shrinkage for expected returns, Ledoit-Wolf for covarianceConstraints: minimum weight 0%, maximum weight 10% for diversificationRisk aversion: calibrated to match target volatility of 10% annualizedTransaction costs: identical 10 basis points assumption across all strategies Estimation: James–Stein shrinkage for expected returns, Ledoit-Wolf for covariance Estimation: James–Stein shrinkage for expected returns, Ledoit-Wolf for covariance Constraints: minimum weight 0%, maximum weight 10% for diversification Constraints: minimum weight 0%, maximum weight 10% for diversification Risk aversion: calibrated to match target volatility of 10% annualized Risk aversion: calibrated to match target volatility of 10% annualized Transaction costs: identical 10 basis points assumption across all strategies Transaction costs: identical 10 basis points assumption across all strategies Evaluation metrics and statistical testing The performance assessment employs a multi-dimensional metrics framework that captures risk-adjusted returns, risk management effectiveness, and implementation practicality. Primary performance metrics Sharpe ratio: annualized measure of risk-adjusted returns with Newey-West standard errorsMaximum drawdown: peak-to-trough decline with duration and recovery analysisCalmar ratio: return relative to maximum drawdown, particularly relevant for risk-averse investorsTurnover ratio: annualized percentage of portfolio value traded, directly impacting implementation costsHit rate: percentage of periods where the strategy outperforms the equal-weighted benchmark Sharpe ratio: annualized measure of risk-adjusted returns with Newey-West standard errors Sharpe ratio: annualized measure of risk-adjusted returns with Newey-West standard errors Maximum drawdown: peak-to-trough decline with duration and recovery analysis Maximum drawdown: peak-to-trough decline with duration and recovery analysis Calmar ratio: return relative to maximum drawdown, particularly relevant for risk-averse investors Calmar ratio: return relative to maximum drawdown, particularly relevant for risk-averse investors Turnover ratio: annualized percentage of portfolio value traded, directly impacting implementation costs Turnover ratio: annualized percentage of portfolio value traded, directly impacting implementation costs Hit rate: percentage of periods where the strategy outperforms the equal-weighted benchmark Hit rate: percentage of periods where the strategy outperforms the equal-weighted benchmark Risk management metrics Value at risk (95%): maximum loss expected over a given time horizonConditional VaR: expected loss conditional on exceeding VaR thresholdVolatility targeting error: deviation from target volatility levelsRegime switching accuracy: model’s ability to correctly identify and adapt to market regimes Value at risk (95%): maximum loss expected over a given time horizon Value at risk (95%): maximum loss expected over a given time horizon Conditional VaR: expected loss conditional on exceeding VaR threshold Conditional VaR: expected loss conditional on exceeding VaR threshold Volatility targeting error: deviation from target volatility levels Volatility targeting error: deviation from target volatility levels Regime switching accuracy: model’s ability to correctly identify and adapt to market regimes Regime switching accuracy: model’s ability to correctly identify and adapt to market regimes Statistical significance testing We employ multiple hypothesis tests to ensure the robustness of performance differences:Diebold–Mariano test: for comparing predictive accuracy of portfolio returnsHansen’s SPA test: superior predictive ability test with multiple comparison adjustmentBootstrap confidence intervals: 10,000 samples with block bootstrap to preserve autocorrelationRegime-based T-tests: separate significance testing across different volatility regimes (Fig. 5)Fig. 5Comprehensive evaluation framework showing the interrelationship between different metric categories and their connection to research objectives. Diebold–Mariano test: for comparing predictive accuracy of portfolio returns Diebold–Mariano test: for comparing predictive accuracy of portfolio returns Hansen’s SPA test: superior predictive ability test with multiple comparison adjustment Hansen’s SPA test: superior predictive ability test with multiple comparison adjustment Bootstrap confidence intervals: 10,000 samples with block bootstrap to preserve autocorrelation Bootstrap confidence intervals: 10,000 samples with block bootstrap to preserve autocorrelation Regime-based T-tests: separate significance testing across different volatility regimes (Fig. 5) Regime-based T-tests: separate significance testing across different volatility regimes (Fig. 5) Comprehensive evaluation framework showing the interrelationship between different metric categories and their connection to research objectives. Robustness checks and sensitivity analysis To ensure the validity of our conclusions, we conduct extensive robustness checks across multiple dimensions: Parameter sensitivity Systematic variation of key hyperparameters within reasonable bounds to assess stability of results. This includes testing different risk aversion parameters, attention sparsity levels, and lookback periods. Market condition analysis Stratified performance evaluation across bull/bear markets, high/low volatility regimes, and different interest rate environments to verify consistent performance. Implementation assumptions Variation of transaction cost assumptions, rebalancing frequencies, and portfolio size to test practical applicability under different operational constraints. This comprehensive experimental setup ensures that our findings are not only statistically significant but also practically relevant and robust to varying market conditions and implementation scenarios. The subsequent results section presents the findings from this rigorous evaluation framework. Results and analysis This section presents a comprehensive analysis of the experimental results, systematically evaluating the proposed framework against the research objectives outlined in the introduction. The analysis progresses from overall performance metrics to specialized examinations of regime adaptation, computational efficiency, and economic interpretability, providing a multi-faceted validation of the methodology. Overall performance assessment The primary evaluation of the proposed framework reveals substantial improvements across all key performance metrics relative to established benchmarks. Table 1 presents the comprehensive out-of-sample results for the testing period January 2017 to December 2022, encompassing diverse market conditions from the pre-COVID bull market through the 2022 monetary tightening cycle. A comprehensive summary of out-of-sample performance metrics is provided in Table 2.Table 2Comprehensive out-of-sample performance metrics (2017-2022).ModelSharpe ratioMax DDCalmar ratioTurnoverVolatilityReturnRisk parity0.8929.7%0.410.1212.3%10.9%Deep portfolio1.1222.3%0.581.4514.8%16.6%Robust MV0.9526.4%0.450.8713.1%12.4%Proposed framework1.3816.2%0.850.5411.9%16.4%Significant values are in bold. Table 2Comprehensive out-of-sample performance metrics (2017-2022).ModelSharpe ratioMax DDCalmar ratioTurnoverVolatilityReturnRisk parity0.8929.7%0.410.1212.3%10.9%Deep portfolio1.1222.3%0.581.4514.8%16.6%Robust MV0.9526.4%0.450.8713.1%12.4%Proposed framework1.3816.2%0.850.5411.9%16.4%Significant values are in bold. Comprehensive out-of-sample performance metrics (2017-2022). Significant values are in bold. The proposed framework achieves a Sharpe ratio of 1.38, representing a 55% improvement over the traditional risk parity benchmark (0.89) and a 23% enhancement over the deep learning alternative (1.12). This superior risk-adjusted performance is particularly noteworthy given the controlled volatility level of 11.9%, which remains within the target band of 10–12% annualized volatility. The maximum drawdown of 16.2% demonstrates exceptional risk management capabilities, significantly outperforming all benchmarks during stress periods. Figure 6 illustrates the cumulative performance trajectory, highlighting the framework’s ability to participate in upward market movements while providing substantial protection during declines. The COVID-19 crisis period (Q1 2020) serves as a critical test case, where the proposed framework limited losses to 18.3% compared to 29.7% for risk parity and 26.4% for robust mean-variance optimization. The turnover ratio of 0.54 strikes an optimal balance between adaptation frequency and implementation costs. This metric is particularly significant as it indicates that the performance advantages are not achieved through excessive trading that would be eroded by transaction costs in practical implementation. The proposed framework generates 54% portfolio turnover annually, substantially lower than the deep portfolio approach (145%) while maintaining superior performance.Fig. 6Cumulative performance of $1 invested across different strategies from January 2017 to December 2022. The proposed framework (solid blue line) demonstrates consistent outperformance with superior risk management during market downturns. Cumulative performance of $1 invested across different strategies from January 2017 to December 2022. The proposed framework (solid blue line) demonstrates consistent outperformance with superior risk management during market downturns. Figure 1 illustrates the cumulative performance trajectory, highlighting the framework’s ability to participate in upward market movements while providing substantial protection during declines. The COVID-19 crisis period (Q1 2020) serves as a critical test case, where the proposed framework limited losses to 18.3% compared to 29.7% for risk parity and 26.4% for robust mean-variance optimization. Regime-specific performance analysis A central research objective concerned the framework’s ability to dynamically adapt risk targets across changing market regimes. The regime-specific analysis provides compelling evidence of this adaptive capability, with particularly strong performance during high-volatility periods. Regime-specific performance breakdowns are reported in Table 3.Table 3Regime-specific performance decomposition.RegimeSharpe ratioMax DDReturnDays in regimeLow volatility (VIX < 15)Risk parity1.258.3%12.1%985Proposed framework1.416.9%13.8%985Improvement12.8%16.9%14.0%Medium volatility (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$15 \le VIX < 25$$\end{document})Risk Parity0.6215.4%8.7%312Proposed framework0.959.8%11.2%312Improvement53.2%36.4%28.7%High volatility (VIX \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ge$$\end{document} 25)Risk parity0.3129.7%3.2%98Proposed framework0.8916.2%9.8%98Improvement187.1%45.5%206.3%Significant values are in bold. Table 3Regime-specific performance decomposition.RegimeSharpe ratioMax DDReturnDays in regimeLow volatility (VIX < 15)Risk parity1.258.3%12.1%985Proposed framework1.416.9%13.8%985Improvement12.8%16.9%14.0%Medium volatility (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$15 \le VIX < 25$$\end{document})Risk Parity0.6215.4%8.7%312Proposed framework0.959.8%11.2%312Improvement53.2%36.4%28.7%High volatility (VIX \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ge$$\end{document} 25)Risk parity0.3129.7%3.2%98Proposed framework0.8916.2%9.8%98Improvement187.1%45.5%206.3%Significant values are in bold. Regime-specific performance decomposition. Significant values are in bold. The results demonstrate a critical finding: the proposed framework’s outperformance increases monotonically with market volatility. During high-volatility regimes (VIX \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ge$$\end{document} 25), the framework achieves a 187% higher Sharpe ratio and reduces maximum drawdowns by 45.5% compared to traditional risk parity. Figure 7 provides a detailed visualization of the framework’s behavior during the February-March 2020 market crash. The framework began reducing equity exposure in late February as volatility indicators signaled regime transition, reaching minimum risk exposure precisely at the market trough. This proactive risk management contrasts sharply with the reactive nature of traditional approaches that only adjust after losses have materialized. This pattern directly validates the dynamic risk targeting capability, as the framework successfully anticipates and adapts to deteriorating market conditions. The performance improvement during medium-volatility regimes (53.2% Sharpe ratio enhancement) is particularly economically significant, as these transitional periods often precede more severe market dislocations. The framework’s ability to adjust risk exposures during these phases demonstrates sophisticated regime detection and response mechanisms.Fig. 7Framework response during regime transitions showing dynamic weight adjustments and risk target adaptations. The upper panel displays market volatility (VIX), while the lower panels show portfolio risk exposure and cumulative returns. Framework response during regime transitions showing dynamic weight adjustments and risk target adaptations. The upper panel displays market volatility (VIX), while the lower panels show portfolio risk exposure and cumulative returns. Figure 2 provides a detailed visualization of the framework’s behavior during the February-March 2020 market crash. The framework began reducing equity exposure in late February as volatility indicators signaled regime transition, reaching minimum risk exposure precisely at the market trough. This proactive risk management contrasts sharply with the reactive nature of traditional approaches that only adjust after losses have materialized. Computational efficiency and scalability The second research objective addressed computational scalability for high-dimensional portfolios. The framework demonstrates remarkable efficiency gains, particularly as portfolio dimensionality increases. Computational performance metrics across portfolio sizes are summarized in Table 4.Table 4Computational performance across portfolio sizes.Asset countTraining time (hrs)Inference time (ms)Memory usage (GB)Sharpe ratio10 Assets1.212.32.11.4225 Assets1.818.73.81.3950 Assets2.424.56.21.38100 Assets3.131.210.41.36200 Assets4.342.818.71.33 Table 4Computational performance across portfolio sizes.Asset countTraining time (hrs)Inference time (ms)Memory usage (GB)Sharpe ratio10 Assets1.212.32.11.4225 Assets1.818.73.81.3950 Assets2.424.56.21.38100 Assets3.131.210.41.36200 Assets4.342.818.71.33 Computational performance across portfolio sizes. The framework maintains nearly consistent performance across portfolio sizes, with only a 6% degradation in Sharpe ratio when scaling from 10 to 200 assets. The computational scaling advantages of the framework are shown in Fig. 8. This scalability is achieved through the sparse attention mechanism, which reduces computational complexity from cubic to near-linear growth patterns. The inference time of 24.5 milliseconds for a 50-asset portfolio enables real-time implementation even for high-frequency rebalancing scenarios. This represents an 8.7× speed improvement compared to conventional neural approaches that require full covariance matrix operations.Fig. 8Computational scaling analysis showing near-linear growth in training time versus portfolio size. The proposed framework (blue) maintains efficiency while traditional approaches (red) exhibit exponential complexity growth. Computational scaling analysis showing near-linear growth in training time versus portfolio size. The proposed framework (blue) maintains efficiency while traditional approaches (red) exhibit exponential complexity growth. Figure 3 illustrates the computational advantage through complexity analysis. The proposed framework’s scaling curve remains substantially below the theoretical O(\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hbox {n}^{3}$$\end{document}) boundary, validating the efficiency claims. This scalability is crucial for practical implementation in institutional settings where portfolios often contain hundreds of securities. Risk attribution and interpretability analysis The third research objective concerned maintaining interpretability without sacrificing performance. The risk attribution analysis provides transparent insights into the framework’s decision-making process.Table 5Feature importance analysis across market conditions.FeatureNormal conditionsStress conditionsRegime sensitivityVIX index18.3%42.7%HighCredit spreads12.4%38.9%HighYield curve15.8%25.4%MediumMomentum28.7%8.3%HighVolatility clustering14.2%22.6%MediumLiquidity measures10.6%31.5%High Table 5Feature importance analysis across market conditions.FeatureNormal conditionsStress conditionsRegime sensitivityVIX index18.3%42.7%HighCredit spreads12.4%38.9%HighYield curve15.8%25.4%MediumMomentum28.7%8.3%HighVolatility clustering14.2%22.6%MediumLiquidity measures10.6%31.5%High Feature importance analysis across market conditions. The attribution analysis reveals economically intuitive patterns that enhance trust in the framework’s decisions. During normal market conditions, the factors of the momentum and yield curve dominate allocation decisions, The importance values of the characteristics dependent on the market condition are detailed in Table 5. consistent with traditional factor investing literature. However, during stress periods, the framework automatically shifts attention to risk-focused indicators including the VIX index, credit spreads, and liquidity measures. This dynamic feature weighting demonstrates sophisticated pattern recognition that aligns with fundamental financial principles. The framework’s ability to prioritize different information sources based on market conditions represents a significant advancement over static weighting schemes. Risk contributions and their decomposition are visualized in Fig. 9.Fig. 9Waterfall chart of risk attribution showing contribution decomposition by asset class and risk factor. The analysis provides transparent insights into portfolio construction decisions. Waterfall chart of risk attribution showing contribution decomposition by asset class and risk factor. The analysis provides transparent insights into portfolio construction decisions. Figure 4 provides a detailed decomposition of risk contributions during the March 2020 crisis period. The framework allocated 42% of risk budget to defensive assets (bonds and gold) while reducing equity exposure to 38% from the typical 52%. This proactive de-risking occurred two weeks before the market trough, demonstrating anticipatory capabilities rather than reactive responses. Robustness and sensitivity analysis Comprehensive robustness checks validate the stability of results under varying assumptions and market conditions.Table 6Robustness analysis under alternative scenarios.ScenarioSharpe ratioMax DDTurnoverPerformance changeBaseline1.3816.2%0.54–+25bps transaction costs1.3516.4%0.52-2.2%+50bps transaction costs1.2917.1%0.51-6.5%Weekly rebalancing1.3116.8%0.23-5.1%Monthly rebalancing1.3616.3%0.18-1.4%Alternative risk measure (CVaR)1.3515.7%0.62-2.2%Reduced training data (3 years)1.2218.9%0.61-11.6% Table 6Robustness analysis under alternative scenarios.ScenarioSharpe ratioMax DDTurnoverPerformance changeBaseline1.3816.2%0.54–+25bps transaction costs1.3516.4%0.52-2.2%+50bps transaction costs1.2917.1%0.51-6.5%Weekly rebalancing1.3116.8%0.23-5.1%Monthly rebalancing1.3616.3%0.18-1.4%Alternative risk measure (CVaR)1.3515.7%0.62-2.2%Reduced training data (3 years)1.2218.9%0.61-11.6% Robustness analysis under alternative scenarios. The framework maintains performance superiority across all sensitivity scenarios, with maximum degradation of 11.6% under reduced training data conditions. Robustness checks across alternative model assumptions are presented in Table 6. The relatively modest impact of transaction cost increases demonstrates the framework’s careful balance between adaptation frequency and implementation efficiency. The stability under alternative risk measures (Conditional Value at Risk) confirms that the performance advantages are not specific to variance-based optimization but extend to tail risk management approaches. This robustness enhances the framework’s applicability across different institutional preferences and regulatory environments.Fig. 10Performance surface across hyperparameter variations showing broad regions of stable performance. The framework demonstrates robustness to parameter selection within reasonable bounds. Performance surface across hyperparameter variations showing broad regions of stable performance. The framework demonstrates robustness to parameter selection within reasonable bounds. Figure 5 illustrates the performance stability across hyperparameter variations. The framework exhibits a wide plateau of strong performance, indicating that results are not dependent on precise parameter tuning but represent fundamental advantages of the architectural approach. The stability of performance across hyperparameter variations is demonstrated in Fig. 10. Statistical significance testing Formal hypothesis testing confirms the statistical significance of performance differences. Diebold-Mariano tests reject the null hypothesis of equal predictive accuracy at the 1% significance level (p = 0.0032) for all benchmark comparisons. Hansen’s Superior Predictive Ability test further validates that the outperformance is not due to random chance or data snooping bias. Bootstrap analysis with 10,000 samples generates 95% confidence intervals of [1.29, 1.47] for the Sharpe ratio, entirely above the benchmark ranges. The results remain statistically significant after adjusting for multiple comparisons and controlling for autocorrelation in returns. The comprehensive results analysis demonstrates that the proposed framework successfully addresses all three research objectives: dynamic risk targeting across regimes, computational scalability to large portfolios, and maintained interpretability through transparent risk attribution. The following discussion section explores the theoretical and practical implications of these findings. Conclusion This research has presented a comprehensive machine learning framework for dynamic risk-based asset allocation that addresses fundamental limitations in traditional portfolio optimization methods. Through rigorous empirical testing and theoretical innovation, we have demonstrated that integrating advanced neural architectures with financial theory produces substantial improvements in risk-adjusted performance, particularly during volatile market conditions. The concluding section synthesizes our key findings, addresses each research objective with corresponding results, and outlines implications for both academic research and practical portfolio management. Summary of key findings The experimental results unequivocally demonstrate the superiority of the proposed framework across multiple dimensions of portfolio performance. The framework achieved a Sharpe ratio of 1.38 during the out-of-sample testing period (2017-2022), representing a 55% improvement over conventional risk parity strategies and a 23% enhancement over contemporary deep learning approaches. More significantly, the maximum drawdown of 16.2% during stress periods showcases robust risk management capabilities, with a 41% reduction compared to traditional methods. The framework’s ability to maintain consistent outperformance across diverse market regimes–from the low-volatility bull market of 2017–2019 through the COVID-19 crisis and subsequent recovery–validates its adaptive capabilities. This performance persistence under varying market conditions suggests that the methodology captures fundamental risk-return relationships rather than exploiting transient patterns or data anomalies. Addressing research objectives The study successfully addressed each of the three primary research objectives through targeted methodological innovations and empirical validation: Objective 1: dynamic risk targeting across market regimes The results comprehensively validate the framework’s capability to dynamically adjust risk allocations in response to changing market conditions. The regime-specific analysis revealed a monotonic improvement in performance as market volatility increased, with the framework achieving a 187% higher Sharpe ratio during high-volatility periods compared to traditional risk parity. This adaptive behavior stems from the integrated regime detection mechanism that processes leading indicators like the VIX and credit spreads to anticipate market transitions. The proactive risk reduction observed during the February 2020 market downturn–where the framework began decreasing equity exposure two weeks before the market trough–demonstrates genuine predictive capability rather than reactive adjustment. Objective 2: computational scalability for high-dimensional portfolios The framework successfully overcome the curse of dimensionality that plagues traditional optimization methods. The sparse attention mechanism reduced computational complexity from cubic to near-linear growth, enabling efficient processing of portfolios with up to 200 assets while maintaining inference times under 50 milliseconds. This scalability was achieved without sacrificing performance, as the Sharpe ratio degradation from 10 to 200 assets was limited to just 6%. The practical implication is significant: institutional investors can now apply sophisticated machine learning techniques to complex multi-asset portfolios without encountering computational bottlenecks that have historically constrained implementation. Objective 3: maintained interpretability without performance sacrifice The risk attribution analysis confirmed that the framework maintains transparency despite its neural architecture. The SHAP-based interpretation revealed economically intuitive decision patterns, with the model prioritizing momentum factors during calm markets and shifting attention to risk indicators during stress periods. This dynamic feature weighting aligns with fundamental financial principles while providing portfolio managers with clear explanations for allocation decisions. The 22% reduction in interpretability clarity compared to linear models represents an acceptable trade-off given the substantial performance improvements, particularly since the framework remains sufficiently transparent for regulatory compliance and institutional adoption. Theoretical implications The findings challenge several conventional assumptions in portfolio theory. First, the superior performance during high-volatility regimes contradicts the efficient market hypothesis by demonstrating that machine learning can systematically identify and exploit regime-dependent patterns. Second, the framework’s success with joint risk-return optimization suggests that the traditional separation of these objectives may be suboptimal. The neural architecture’s ability to capture complex interactions between return predictors and risk factors indicates that integrated approaches may better approximate investor utility functions. The convexity analysis of the loss landscape provides important insights for financial machine learning. The identification of saddle points corresponding to traditional portfolio strategies suggests that these methods represent local optima in a more complex optimization space. The framework’s ability to navigate this landscape and converge to superior solutions demonstrates the value of gradient-based optimization in high-dimensional financial problems. Practical applications and implementation considerations The framework offers immediate practical value for institutional investors seeking to enhance risk-adjusted returns while maintaining robust risk management. The controlled turnover ratio of 0.54 indicates that performance advantages are achievable without excessive trading costs, addressing a common criticism of machine learning approaches in finance. For a typical portfolio of $10 billion, the framework would generate approximately $27 million annual cost savings compared to deep learning alternatives while delivering superior performance. The modular architecture facilitates implementation in existing investment processes, with the regime detection and risk budgeting layers capable of integration with traditional portfolio construction systems. The framework’s sensitivity to different rebalancing frequencies and transaction cost assumptions further enhances its practical applicability, as performance remains robust across various implementation scenarios. Limitations and future research directions Although the framework demonstrates significant advantages, several limitations warrant attention in future research. The current implementation requires a minimum of three years of daily data for stable convergence, which may restrict the application to emerging asset classes or newly established markets. Additionally, the interpretability framework, while sufficient for most institutional purposes, could be enhanced through more sophisticated visualization techniques and interactive explanation interfaces. Future research should explore several promising directions. First, incorporating alternative data sources such as satellite imagery, sentiment on social media, and supply chain information could enhance regime detection capabilities. Second, extending the framework to incorporate ESG constraints and sustainability objectives would align with growing investor preferences for responsible investment. Third, investigating quantum computing applications could address the remaining computational challenges for ultra-large portfolios exceeding 500 assets. The methodology also opens possibilities for applications beyond traditional portfolio optimization, including risk management for derivative portfolios, asset-liability management for pension funds, and dynamic hedging strategies for insurance companies. The flexibility of the framework suggests broad applicability across financial domains that require adaptive risk management. Concluding remarks This research establishes a new paradigm for risk-based asset allocation that successfully bridges the gap between theoretical finance and practical implementation. The integration of machine learning with portfolio theory has produced a framework that not only outperforms traditional methods but does so through economically intuitive mechanisms that enhance rather than obscure the understanding of market dynamics. The validation of the framework over multiple market cycles, including severe stress periods, provides confidence in its robustness and practical utility. As financial markets continue to evolve in complexity and interconnectedness, approaches that combine the pattern recognition capabilities of machine learning with the theoretical foundations of finance will become increasingly essential. This work represents a significant step toward that future, offering both immediate practical benefits and a foundation for continued innovation in intelligent portfolio management. The demonstrated improvements in risk-adjusted returns, coupled with computational efficiency and maintainability, suggest that machine learning-enhanced portfolio optimization is ready for widespread adoption in institutional settings. The framework provides a template for how advanced computational techniques can be responsibly integrated into financial decision making while preserving the discipline and transparency required for prudent investment management.