When Valid Signals Fail: Regime Boundaries Between LLM Features and RL Trading Policies
Zhengzhe Yang · 2026
Read When Valid Signals Fail: Regime Boundaries Between LLM Features and RL Trading Policies by Zhengzhe Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10996.txt.
Introduction
Yang asks whether language models can generate numerical features for reinforcement learning trading agents. He builds a pipeline where a frozen language model extracts features, turning daily news and filings into a vector for a PPO agent. A prompt loop tunes extraction against the information coefficient, the rank link between predicted and realized returns. The tuned prompt finds predictive features. Yet during an economy-wide shock, the language-model features add noise and the agent trails a price-only baseline. Economy-wide state variables stay the most robust driver.
Catalog record
| CREATOR | Zhengzhe Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27298 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10996
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Valid Signals Fail: Regime Boundaries Between LLM Features and RL Trading Policies Zhengzhe Yang Abstract Can large language models (LLMs) generate continuous numerical features that improve reinforcement learning (RL) trading agents? We build a modular pipeline where a frozen LLM serves as a stateless feature extractor, transforming unstructured daily news and filings into a fixed-dimensional vector consumed by a downstream PPO agent. We introduce an automated prompt-optimization loop that treats the extraction prompt as a discrete hyperparameter and tunes it directly against the Information Coefficient—the Spearman rank correlation between predicted and realized returns—rather than NLP losses. The optimized prompt discovers genuinely predictive features (IC above 0.15 on held-out data). However, these valid intermediate representations do not automatically translate into downstream task performance: during a distribution shift caused by a macroeconomic shock, LLM-derived features add noise, and the augmented agent under-performs a price-only baseline. In a calmer test regime the agent recovers, yet macroeconomic state variables remain the most robust driver of policy improvement. Our findings highlight a gap between feature-level validity and policy-level robustness that parallels known challenges in transfer learning under distribution shift. 1 Introduction Recent work increasingly applies large language models to financial decision-making, whether as end-to-end trading agents or as modular sentiment classifiers feeding downstream models. While these modular pipelines separate the language model from the trading algorithm, they frequently suffer from an objective mismatch : the LLM is optimized against standard NLP losses (e.g., classification accuracy or sentiment polarity) rather than downstream financial utility. Consequently, it remains difficult to guarantee that the extracted narratives form a robust state representation for a continuous trading policy. We address this gap by maintaining strict architectural separation while directly aligning the feature extraction process with a financial objective. The frozen LLM acts as a stateless feature extractor : given a bundle of news articles and SEC filings for a ticker on day d , it emits a fixed-length numerical vector (sentiment, impact, conflict flags, etc.). A separate PPO agent then consumes this vector alongside price data and macroeconomic indicators to make portfolio decisions. This design ensures that the intermediate representations are genuinely predictive, allowing us to evaluate the LLM’s true contribution in isolation. Figure 1: Cumulative portfolio value across 2025. Solid lines: 5-seed mean; shaded: \pm 1 std. H1 (red) captures a tariff-driven shock; H2 (yellow) is calmer. The regime split isolates when LLM features succeed and fail. Our contributions are: 1. Prompt-as-hyperparameter optimization. We introduce a mutation–evaluation–selection loop that treats the LLM extraction prompt as a discrete hyperparameter and optimizes it against the Information Coefficient (IC)—the rank correlation between predicted and realized returns—rather than NLP losses like BLEU or accuracy. The winning prompt improves IC from -0.024 to +0.104 (Table 2 ). 2. Feature-validity-to-policy-utility gap. We show that valid intermediate LLM representations do not automatically translate into downstream RL performance. The gap is regime-dependent: under distribution shift caused by a macroeconomic shock, news-derived features add noise rather than signal. 3. Multi-regime ablation on held-out data. A controlled four-configuration ablation (Baseline, LLM-only, Macro-only, LLM+Macro) across a volatile validation period (H1 2025) and a calmer held-out test period (H2 2025) reveals that macroeconomic state variables are the most reliable driver of policy robustness (Figure 1 ). 2 Related Work Reinforcement Learning for Trading. Deep RL has been applied to portfolio management and order execution with increasing sophistication ( 2 ) . FinRL ( 5 ) provides a standardized library for training PPO, A2C, and DDPG agents on market environments. A persistent challenge is non-stationarity: the data distribution shifts between training and deployment, a problem well-studied in the broader RL literature as distributional shift ( 4 ) . In finance, these shifts are driven by macroeconomic regime changes ( 1 ) , and recent work has explored conditioning RL policies on detected regimes ( 8 ) . LLMs in Finance. BloombergGPT ( 9 ) demonstrated that domain-specific pre-training improves financial NLP benchmarks. FinGPT ( 10 ) pursued the same goal with open-source fine-tuning. 6 demonstrated that frontier models can forecast subsequent-day stock returns using raw headline text, while later approaches instruction-tune open source variants directly against financial tasks ( 11 ) . However, these lines of work overwhelmingly evaluate the LLM as a standalone classifier (sentiment polarity, NER, QA). By contrast, we treat the LLM as a frozen, zero-shot numerical feature extractor—analogous to using a pre-trained vision model as a fixed encoder for downstream tasks—and evaluate its representations against a continuous downstream RL objective rather than NLP classification metrics. Prompt & Pipeline Optimization. Frameworks like DSPy ( 3 ) formally compile language model calls by optimizing prompts against programmatic validation metrics. While these frameworks routinely optimize for exact-match accuracy or retrieval scores, our optimization loop extends this paradigm to a domain-specific continuous metric (rank correlation of predicted returns) prior to RL integration, treating the prompt as a discrete hyperparameter in the same spirit as architecture search. Information Asymmetry and News Latency. Insider-trading research ( 7 ) established that information edges decay rapidly as they become public. For free-tier news feeds, institutional desks have already acted on the headline by the time a retail pipeline ingests it. This latency shapes the horizon at which LLM-derived features can carry signal—a constraint we quantify in Section 5.1 . 3 System Architecture News & Flows RSS, EDGAR, Benzinga Go Bundler (date \times ticker) LLM Feature Extractor (Qwen3-235B API) Feature Vector (9 Dims) SB3 VecNormalize (clip_obs=10) PPO Policy Agent (MLP Network) Trading Actions \{-1,0,+1\} Figure 2: System overview. A Go-based ingestion pipeline collects news, filings, and macro data into a relational store. The LLM produces per-ticker feature vectors, which the PPO agent consumes alongside OHLCV and technical indicators. 3.1 Data Ingestion Pipeline Reproducible feature extraction requires a deterministic historical record. We built a concurrent Go pipeline that ingests: (1) news from Alpaca’s Benzinga feed, (2) RSS aggregations from financial outlets, and (3) SEC EDGAR filings (Form 4 insider trades and 8-K disclosures). Raw text is bundled per ticker per trading day and persisted to a SQLite database (the “backfill layer”). This backfill-first design prevents look-ahead bias: the LLM always reads from a frozen snapshot whose information boundary is strictly \leq day d . It also allows the prompt-optimization loop (Section 4 ) to re-extract features from identical text without re-scraping. Because our pipeline enforces strict daily information boundaries (aggregating all feeds at the close of day d ), it is intentionally blind to intra-day volatility and high-frequency market microstructure. Consequently, the RL agent operates on smoothed, day-over-day narrative shifts rather than instantaneous headline shocks. We measure the effect of this daily resolution in Section 5.1 . 3.2 Feature Schema A frontier LLM ( Qwen3 235B A22B Instruct 2507 ) processes each ticker’s daily bundle and outputs a structured JSON mapping into continuous RL observation bounds: 1. Stock-level LLM features (4 dims): • sentiment \in[-1,1] : Directional conviction from the daily news flow ( -1 : very bearish, +1 : very bullish). • impact \in[0,1] : Financial materiality of the bundle (e.g., CEO resignation vs. routine marketing). • conflicting_signals \in[0,1] : Evidentiary contradiction across competing sources within the same bundle. • news_novelty \in[0,1] : Divergence of the current day’s narrative from historical baselines. 2. Macroeconomic features (5