When Reasoning Fails: Evaluating “Thinking” LLMs for Stock Prediction
Rakeshkumar H Sodha · 2025
Read When Reasoning Fails: Evaluating “Thinking” LLMs for Stock Prediction by Rakeshkumar H Sodha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08608.txt.
Introduction
Rakeshkumar Sodha asks whether reasoning-style language models, the kind that show their thinking, actually beat plain models at picking stocks. The testbed is Indian equities from the NIFTY index, checked with a rolling walk-forward design that trains on four years and predicts one day ahead.
As the stock universe grows from five names to thirty-six under a fixed reasoning budget, the thinking model's ranking quality fades while a direct model stays flat. Its answers also swing more from day to day, needing calming fixes such as winsorization and blending. Once trading costs enter the backtest, no net edge remains. Sodha argues that fixed token budgets and next-token training fit noisy, heavy-tailed returns poorly.
Catalog record
| CREATOR | Rakeshkumar H Sodha |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-08608
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Reasoning Fails: Evaluating “Thinking” LLMs for Stock Prediction Rakeshkumar H Sodha Abstract Problem. “Thinking” LLMs (TLLMs) expose explicit or hidden reasoning traces and are widely believed to generalize better on complex tasks than direct LLMs. Whether this promise carries to noisy, heavy–tailed and regime–switching financial data remains unclear. Approach. Using Indian equities (NIFTY constituents), we run a rolling 48m/1m walk–forward evaluation at horizon k = 1 k=1 day and dial cross–sectional complexity via the universe size U ∈ { 5 , 11 , 21 , 36 } U\in\{5,11,21,36\} while keeping the reasoning budget fixed ( B = 512 B=512 tokens) for the TLLM. We compare a direct LLM ( gpt-4o-mini ), a TLLM ( gpt-5 ), and classical learners (ridge, random forest) on cross–sectional ranking loss 1 − IC 1-\mathrm{IC} , MSE, and long/short backtests with realistic costs. Statistical confidence is measured with Diebold–Mariano, Pesaran–Timmermann, and SPA tests. Main findings. (i) As U U grows under a fixed budget B B , the TLLM’s ranking quality deteriorates, whereas the direct LLM remains comparatively flat and classical baselines are stable. (ii) TLLM variance is higher, requiring ex–post calibration (winsorization and blending) for stability. (iii) Portfolio results under transaction costs do not support a net advantage for the TLLM. Hypotheses. Our results are consistent with the following testable hypotheses: H1 (Capacity–Complexity Mismatch): for fixed B B , TLLM accuracy degrades superlinearly in cross–sectional complexity. H2 (Reasoning Variance): TLLM outputs exhibit higher dispersion date–by–date than direct LLMs, increasing error bars and turnover. H3 (Domain Misfit): next–token prediction objectives and token–budgeted inference are poorly aligned with heavy–tailed, weakly predictable stock returns. Implication. In our setting, “thinking” LLMs are not yet ready to replace classical or direct methods for short–horizon stock ranking; scaling the reasoning budget and/or re–aligning objectives appears necessary. Positioning and Claims Our conclusions are conditional on: (i) short-horizon stock ranking at k{=}1 day on NIFTY constituents, (ii) cross-sectional complexity dialed via U\in\{5,11,21,36\} , (iii) fixed thinking budget B{=}512 tokens for the TLLM, (iv) the feature blocks and calibration described herein, and (v) the 2019–present sample under our walk-forward protocol. We do not claim universal failure of TLLMs; rather, we document that under these conditions TLLMs do not outperform direct LLMs or classical baselines. 1 Data and Stock Selection We evaluate the liquid Indian equity universe (NIFTY constituents). 1 1 1 The exact list and rolling windows are defined in the configuration manifest produced by our code. At each training/testing step: • Compute average daily volume per ticker within the training window. • Select the top- U tickers by liquidity and hold that set fixed for the following test window. • Roll forward using a 48-month train / 1-month test schedule (as configured). Complexity is varied primarily through U\in\{5,11,21,36\} at horizon k{=}1 trading day. Horizon variants k\in\{5,20\} can also be evaluated at fixed U . Preprocessing: Standardization and Outlier Control For each date d , model scores \{s_{i}(d)\}_{i=1}^{N} are standardized cross-sectionally: z_{i}(d)\;=\;\frac{s_{i}(d)-\mu(d)}{\sigma(d)}\,,\quad\mu(d)=\tfrac{1}{N}\sum_{j=1}^{N}s_{j}(d),\quad\sigma(d)=\sqrt{\tfrac{1}{N-1}\sum_{j=1}^{N}\big(s_{j}(d)-\mu(d)\big)^{2}}. If \sigma(d)<10^{-8} we set z_{i}(d)=0 for numerical stability. For LLM predictions we then calibrate to the training target scale (return mean \hat{\mu}_{\text{train}} , std. \hat{\sigma}_{\text{train}} ): \tilde{s}_{i}(d)\;=\;z_{i}(d)\cdot\hat{\sigma}_{\text{train}}+\hat{\mu}_{\text{train}}. To limit outlier influence we winsorize \tilde{s}_{i}(d) at the 5th and 95th percentiles within each date . This sequence (z-score \to calibrate \to winsorize) is exactly what we implement in code. 2 2 2 See the per-date calibration and winsorization immediately after model inference; the logic is applied to each *_cal column. 2 Features From daily prices and volumes we derive a per-ticker panel and forward returns r_{t\to t+k} , then construct: • Momentum: mom_2, mom_5, mom_10, mom_20 . • Volatility: realized stdev of daily returns ( vol_5, vol_20 ). • Reversal & Volume: rev_5 = -mom_5 , volchg_5 (5-day volume change). • Drawdown: drawdown_20 vs trailing 20-day high. • RSI & MACD block: rsi_14 (Wilder), macd , macd_signal , macd_hist . • Cross-sectional ranks (per date): *_xrank in [0,1] . • Market context (per date): mkt_mean_mom_5 , mkt_mean_mom_20 , mkt_mean_vol_20 . Scalar features are z-scored per ticker over time ; *_xrank & mkt_mean_* are per-date cross-sectionals. As U increases, we optionally gate a richer feature block (e.g., inclusion of RSI/MACD/ranks/context) to raise cross-sectional complexity. 3 Models and Prompting Model variants and budgets. We evaluate two LLM paths: a direct LLM (no explicit chain) using gpt-4o-mini and a Thinking LLM using gpt-5 . Token budget B controls internal reasoning: B={{0}} for the direct model and B={{512}} for the Thinking model. Universe sizes are U\in\{5,11,21,36\} , with walk-forward windows of 48 months train and 1 month test. Implementation notes. Inference is batched per date/universe; outputs are cached per (model, horizon, universe, date) and a features snapshot is written for reproducibility. Direct LLM (no explicit chain) and Thinking LLM (TLLM; hidden chain-of-thought) share a single system prompt, differing only by an internal effort flag for TLLM. 3 3 3 We do not log or reveal model reasoning. Only strict TSV outputs are accepted. Token budget for TLLM is fixed at B{=}512 ; direct LLM corresponds to B{=}0 . Classical baselines are ridge regression and random forest trained on the same features. System prompt (verbatim). ⬇ INPUTS : standardized technical features per ticker and horizon k ( trading days ). \bullet Most core features are per - ticker z - scores ( mean \approx 0, std \approx 1) over time . \bullet * _xrank features are cross - sectional percentile ranks in [0,1] within a date ( higher = stronger relative that day ). \bullet Market context scalars ( mkt_mean_ *) are per - date cross - sectional means . FEATURE PRESENCE VARIES BY UNIVERSE : \bullet Not all features will be present for every run . Treat missing features as unavailable ( do NOT substitute zeros or invent ). \bullet When richer blocks are present , you may also see : mom_60 , mom_120 , vol_60 , volchg_20 , ret1_skew_20 , ret1_kurt_20 , beta_mkt_60 , obv_like , and interactions such as int_mom_5__rsi_14 , int_macd_hist__vol_20 , int_drawdown_20__mom_20 . Use them only if provided . TASK : For EACH input ticker , predict the EXPECTED k - day forward return as a decimal ( e . g ., 0.012 = +1.2%). Higher score = more bullish expected return . MODE : \bullet If an ’ effort ’ flag ( low | medium | high ) is present in INPUT , reason internally ( do not reveal reasoning ) and output only scores . \bullet Otherwise , produce calibrated scores directly . Always obey the caller ’ s output format strictly . SCALE & MAGNITUDE GUIDANCE ( typical NIFTY cross - sectional ranges ): \bullet k =1 day : most in [-0.03, +0.03]; \sigma _xs \approx 0.008 - 0.015 \bullet k =5 days : most in [-0.08, +0.08]; \sigma _xs \approx 0.02 - 0.04 \bullet k =20 days : most in [-0.20, +0.20]; \sigma _xs \approx 0.05 - 0.10 Keep the per - date cross - sectional MEAN near 0 ( market - neutral prior ). Use realistic dispersion for the given k ; avoid both collapse to ~0 and unjustified extremes . FEATURE SEMANTICS ( z - scores unless noted ): \bullet mom_ {2,5,10,20,(60,120 if present )}: recent / sustained momentum ( price pct change ). Higher