Plan Before You Trade: Inference-Time Optimization for RL Trading Agents
Eun Go Rohan Deb Siebel School of Computing and Data Science Siebel School of Computing and Data Science University of Illinois Urbana-Champaign University of Illinois Urbana-Champaign [email protected] [email protected] Arindam Banerjee Siebel School of Computing and Data Science University of Illinois Urbana-Champaign [email protected] · 2026
Read Plan Before You Trade: Inference-Time Optimization for RL Trading Agents by Eun Go Rohan Deb Siebel School of Computing and Data Science Siebel School of Computing and Data Science University of Illinois Urbana-Champaign University of Illinois Urbana-Champaign [email protected] [email protected] Arindam Banerjee Siebel School of Computing and Data Science University of Illinois Urbana-Champaign [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12653.txt.
Introduction
Go, Deb and Banerjee give trading agents a way to adjust at run time using price guesses. Static policies lack any way to adapt from guesses when deployed, yet under the price-taker rule, holdings actions do not move future prices. Their FinPILOT plugin lets a multi-step price model supply dreamed market paths without step-by-step action-tied rollouts. At each step, the model's given path builds a share-based dreamed return goal, and the ready policy is tuned at run time before one trade step runs.
Catalog record
| CREATOR | Eun Go Rohan Deb Siebel School of Computing and Data Science Siebel School of Computing and Data Science University of Illinois Urbana-Champaign University of Illinois Urbana-Champaign [email protected] [email protected] Arindam Banerjee Siebel School of Computing and Data Science University of Illinois Urbana-Champaign [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59970 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12653
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Plan Before You Trade: Inference-Time Optimization for RL Trading Agents Eun Go Rohan Deb Siebel School of Computing and Data Science Siebel School of Computing and Data Science University of Illinois Urbana-Champaign University of Illinois Urbana-Champaign [email protected] [email protected] Arindam Banerjee Siebel School of Computing and Data Science University of Illinois Urbana-Champaign [email protected] Abstract Reinforcement learning agents for portfolio management are typically trained and deployed as static policies, with no mechanism for using price forecasts at inference time. We propose 𝙵𝚒𝚗𝙿𝙸𝙻𝙾𝚃 \color[rgb]{0,1,0}\mathtt{FinPILOT} ( Fin ancial P lugin I nference-time L earning for O ptimal T rading), a plugin inference-time optimization framework inspired by Model Predictive Control (MPC). Our key structural insight is that future prices mostly do not depend on one agent’s portfolio allocation, so a suitable predictive model can produce a multi-step price trajectory without iterative action-conditioned rollouts as in typical reinforcement learning. At each decision step, we use the forecaster’s predicted price trajectory to construct an allocation-based imagined return objective, and optimize the policy at inference-time before executing one step of the trade. Our framework is compatible with any pre-trained agent and adapts the policy to the forecaster’s predictions without any retraining. Evaluated across five policy learning algorithms on the TradeMaster DJ30 benchmark, 𝙵𝚒𝚗𝙿𝙸𝙻𝙾𝚃 \color[rgb]{0,1,0}\mathtt{FinPILOT} produces consistent improvements in total return and return-based risk-adjusted metrics (Sharpe, Sortino, Calmar), with stochastic policies benefiting more than deterministic ones. Further, using synthetic forecasts at calibrated quality levels, we show that gains consistently improve with forecaster quality, suggesting that our performance will improve based on advances in financial forecasting. 1 Introduction Reinforcement learning for financial trading (RLFT) has emerged as a principled framework for sequential portfolio decision-making, with recent benchmarks reporting competitive performance across a range of market conditions ( 16 ; 23 ) . The fit is natural: portfolio management requires sequential decisions under shifting market conditions, and reinforcement learning is built for exactly that. In practice, however, RLFT is one of two largely independent threads of machine learning addressing this problem, and neither thread fully closes the loop between forecasting and decision-making. The first thread treats trading as a supervised forecasting task: predict short-horizon returns or price movements with increasingly expressive models, ranging from the linear-to-tree spectrum systematized by 7 to LSTM and Transformer-based movement predictors ( 5 ; 29 ) . Even state-of-the-art forecasters achieve only modest out-of-sample R^{2} (typically below 1% at monthly horizons ( 7 ) ) yet such signals are widely deployed in practice when paired with hand-crafted position sizing rules. This thread produces predictive edges but offloads the decision problem to heuristics external to the model. The second thread treats trading as reinforcement learning. Building on the deep RL formulation of 14 , subsequent methods add risk-aware objectives ( 26 ) , augmented state representations carrying price-prediction signals ( 28 ) , and standardized benchmarking platforms ( 23 ) . The trained policy is then deployed as a fixed function. Forecasts enter only as additional state features during training, never as a planning signal that adapts behavior at inference. A separate line of model-based RL has meanwhile matured around learned world models for planning and control: Dreamer ( 9 ; 10 ) , MuZero ( 19 ) , Trajectory Transformer ( 13 ) , Decision Diffuser ( 1 ) , and TD-MPC2 ( 11 ) . These methods assume high-fidelity dynamics natural to robotics but harder to come by in financial markets, and they ship as integrated systems where the policy is co-designed with the world model. None takes an arbitrary pre-trained RL agent and improves it at deployment. The most directly relevant work ( 4 ) establishes this paradigm for offline RL with continuous-control benchmarks but does not address financial trading. How to drive inference-time policy optimization from a separately trained forecaster, without retraining the agent, remains open. The world-model paradigm itself, however, points at the answer. If an agent can simulate near-future states internally, it can optimize its policy against imagined futures at inference time — the core idea behind Model Predictive Control (MPC), in which an agent repeatedly simulates a short horizon, updates its behavior, and executes the first resulting action before re-planning ( 2 ) . Translating this paradigm to finance raises two key questions: (Q1) What should a world model look like for a financial market? (Q2) Given the rollouts from the world model, should we optimize an action sequence over the planning horizon as in standard MPC, or update the parameters of the pre-trained policy? For (Q1) , financial markets have a structural property that simplifies world modeling: the reward is closed-form. The per-step reward is a deterministic function of the agent’s allocation and realized returns, so the world model reduces to a price forecaster. No separate reward model needs to be learned. This concerns structure, not fidelity. Short-horizon return predictability is weak ( 7 ) , so the right expectation is not high-fidelity rollouts but a forecaster with positive, even if small, predictive R^{2} . For (Q2) , we adapt the pre-trained policy itself rather than optimizing an action sequence, and execute only the first resulting action before re-planning. The forecaster is trained separately on historical data and enters only at deployment, and RL agents see no forecaster signal during training. We further introduce a downside-risk-penalized variant of the imagined objective. We call this framework FinPILOT (Financial Plugin Inference-time Learning for Optimal Trading). With a practical XGBoost forecaster at R^{2}\approx 0.01 , FinPILOT produces consistent improvements in total return (20%+ returns with strong baselines) and risk-adjusted metrics (Sharpe, Sortino, Calmar) on the TradeMaster DJ30 benchmark. The framework is plugin by design: it modifies nothing about training, so we apply it to the five algorithms standard in the RLFT literature, namely PPO, SAC, A2C, TD3, and DDPG ( 17 ; 23 ) . We find consistent gains, with stochastic policies benefiting more than deterministic ones. The same hyperparameters carry over to TradeMaster’s foreign exchange (FX) dataset without retuning. To probe why such modest forecast quality is enough, we run cheating experiments that feed the policy synthetic forecasts at calibrated R^{2} levels. The resulting curve is threshold-like, with performance improving as signal strength rises. Contributions. We summarize our contributions as follows: 1. FinPILOT, a plugin inference-time MPC framework compatible with any pre-trained RL agent, with a downside-risk-penalized variant. 2. Empirical validation showing that a practical XGBoost forecaster ( R^{2}\approx 0.01 ) improves return and risk-adjusted metrics across five RL algorithms on TradeMaster DJ30, transferring to foreign exchange (FX) without retuning. 3. Cheating experiments methodology that calibrates forecast quality ( R^{2} ) against policy gains and can be applied to any forecaster–RL agent pair. 2 Problem Formulation We consider a portfolio management problem over a universe of N assets. At each discrete trading step t , e.g., a day, an agent observes market information and allocates its available capital across the asset universe, for the next trading step (t+1) . The agent’s goal is to learn an allocation policy that maximizes long-run cumulative (risk adjusted) return while remaining subject to the realistic constraints imposed by the trading environment. Figure 1: Overview of FinPILOT . A standard RL agent is pre-trained on historical market data with no access to any price forecaster. At