Heads, Not Backbones: Output Heads Dominate Architectures on Fat-Tailed Returns Conference: ACM International Conference on AI in Finance; 2026; CCS: Applied computing Economics CCS: Computing methodologies Machine learning

Sichao He, Yansong Zhang · 2026

Read Heads, Not Backbones: Output Heads Dominate Architectures on Fat-Tailed Returns Conference: ACM International Conference on AI in Finance; 2026; CCS: Applied computing Economics CCS: Computing methodologies Machine learning by Sichao He, Yansong Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30037.txt.

Introduction

Sichao He and Yansong Zhang ask a practical question about forecasting financial returns. In a deep learning pipeline, what matters more: the backbone or the output head? Backbones are the big sequence models that digest price history. Output heads turn that digest into a point forecast or a distribution. Returns are fat-tailed, so extreme moves matter most.

The team tests four modern backbones against three output heads. The data are S&P 500 monthly log-returns from 1871 to 2023. Validation follows an anchored walk-forward scheme. Moving from a point head to a Gaussian head improves distributional scores modestly. Adding a Gaussian mixture head improves them further still. Switching backbones changes scores by less, on average. The mixture head helps most in high-volatility regimes, like nineteen seventies stagflation. At long horizons, though, the backbone retakes the lead.

Catalog record

CREATORSichao He, Yansong Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30037

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Heads, Not Backbones: Output Heads Dominate Architectures on Fat-Tailed Returns Conference: ACM International Conference on AI in Finance; 2026; CCS: Applied computing Economics CCS: Computing methodologies Machine learning Sichao He, Yansong Zhang Abstract . In a deep forecasting pipeline for fat-tailed financial returns at short horizons, which matters more — the backbone architecture or the output head? We compare four modern backbones (TimesNet, DLinear, N-BEATS, iTransformer) under three output heads: a point head (linear + Huber), a single-Gaussian density head (linear + Gaussian NLL), and a Gaussian mixture density head (linear + GMM NLL, K = 4 K{=}4 ). On S&P 500 monthly log-returns (1871–2023) under anchored walk-forward validation, the three heads form a strict gradient: switching from point to Gaussian improves CRPS by ∼ 1.3 % \sim\!1.3\% ; switching from Gaussian to mixture adds a further ∼ 2.4 % \sim\!2.4\% . Switching between backbones, in contrast, changes CRPS by less than 1.5 % 1.5\% on the point-head row and on the backbone-mean axis; density-head backbone spread is larger (up to 5.1 % 5.1\% on the h = 1 h{=}1 Gaussian row, driven by N-BEATS) but the head gradient ( 3.7 3.7 percentage points) still dominates the backbone effect in the mean. The Model Confidence Set on squared errors ( 12 ) does not exclude any of the 12 variants at the 5 % 5\% level — neither backbone architecture nor output head distinguishes models on point-prediction accuracy; the head separates them only on the distributional metrics (CRPS, pinball, coverage), not on squared error. The mixture head’s incremental value over a single Gaussian is largest in the highest-volatility regimes (e.g. + 13.9 % +13.9\% in 1970s stagflation at h = 12 h{=}12 ; the full per-regime breakdown is in §4.2), confirming that the mixture specifically captures tail risk beyond what a unimodal Gaussian can express. The picture is horizon-dependent: the head dominates at short horizons, but at long horizons ( h ≥ 6 h\geq 6 ) the backbone re-takes the lead — an h-split we document against classical baselines in §5.1. We conclude that on fat-tailed returns at short horizons, the head dominates the backbone, and the mixture distribution adds genuine value over a single Gaussian during crisis periods when risk-management decisions actually matter. 1. Introduction Risk management, capital allocation, and regulatory stress-testing all consume distributional forecasts — Value-at-Risk, expected shortfall, and Basel capital charges are functionals of the predictive distribution, not point estimates. Yet the dominant deep forecasting architectures (TimesNet, DLinear, N-BEATS) ship with point heads trained by Huber or MSE loss, which are not strictly proper scoring rules for fat-tailed returns ( 11 ; 5 ) . The standard fix is to swap the point head for a Gaussian mixture density head trained by NLL — a 2-line change that yields a strictly proper density forecaster. We study this change, together with an intermediate single-Gaussian head, and ask a more pointed question: On fat-tailed financial returns, which matters more — the backbone architecture or the output head? We answer with a clean three-layer experiment. On S&P 500 monthly log-returns (1871-2023) under five anchored walk-forward folds and three seeds, we train 4 backbones \times 3 heads = 12 variants on the same protocol, then compare their CRPS, MAE, coverage, and Pinball loss. The result is a strict gradient on the mean-across-backbones axis: point \to Gaussian \to mixture improves CRPS on the backbone-mean at every horizon, with \sim\!1.3\% from the first step and \sim\!2.4\% from the second. The point \to Gaussian step inverts on a small minority of cells; the per-cell breakdown is in Table 2 . Switching backbones, in contrast, changes CRPS by less than 1.5\% on the point-head row and on the backbone-mean axis; density-head backbone spread is larger (up to 5.1\% on the h{=}1 Gaussian row, driven by N-BEATS) but the head gradient ( 3.7 percentage points) still dominates the backbone effect in the mean. The Hansen–Lunde–Nason Model Confidence Set ( 12 ) on squared errors does not exclude any of the 12 variants at the 5\% level: neither backbone architecture nor output head distinguishes models on point-prediction accuracy; the head separates them only on the distributional metrics (CRPS, pinball, coverage), not on squared error (Section 4.3 ). We replicate the same 12-variant protocol on four additional panels — daily S&P 500, daily VIX, daily 10 -year Treasury yields, and daily EUR/USD — spanning a change of asset class, frequency, and distribution type; the head gradient survives on return-like processes (and on the VIX level itself, whose extreme spikes are best captured by a mixture) but inverts on non-return-like panels such as Treasury yields and EUR/USD (Section 5.3 ). The mixture’s incremental value over a single Gaussian is largest in the highest-volatility regimes, where fat-tailed risk matters most: the mixture specifically captures tail risk beyond what a unimodal Gaussian can express, and the gain is monotonic in the target distribution’s excess kurtosis. Per-regime magnitudes are in Appendix A ; the small minority of cells where the gradient inverts is enumerated in Table 2 . What this paper is not. This paper does not propose a trading strategy: a naive mean-reversion strategy that consumes our density forecasts loses money on every variant (Section 5 ). We do not propose a new backbone: the dominant gain comes from the head, not from any of the four architectures tested. (The exclusion of PatchTST ( 20 ) from the main grid is documented in Section 3.1 ; the patching- \sigma incompatibility diagnostic is in Appendix I .) Generalisation and economic value. The head gradient survives a change of data frequency and asset on return-like processes (including the VIX level, whose extreme spikes make a Gaussian mixture strictly necessary), but does not extend to non-return-like ones (Treasury yields and EUR/USD). We quantify cross-asset generalisation in Section 5.3 and regulator-grade risk-management value via Basel FRTB capital in Section 5.5 ; we also document a tail-pinning limitation of the bounded K{=}4 GMM at the extreme 1\% -VaR threshold in Appendix H . Reproducibility. A single CLI call per experiment reproduces every number in this paper. All data, code, and results are at https://github.com/Routhleck/heads-not-backbones . 2. Background We use the S&P 500 monthly log-return series (1,832 monthly observations from 1871–2023) from the Shiller dataset ( 2 ) . Empirical kurtosis is \approx 6 — well above the Gaussian value of 3 . This motivates the search for strictly proper scoring rules. Strictly proper scoring rules. A scoring rule S(P,y) is strictly proper if its expectation under the true distribution P_{\text{true}} is uniquely minimised at P=P_{\text{true}} ( 11 ) . Huber ( 13 ) and MSE losses incentivise the conditional median / mean, not the conditional distribution . CRPS and the quantile (Pinball) loss are strictly proper. We use CRPS as our primary metric, with Pinball loss (originally formalised for quantile regression by ( 15 ) ) and predictive-interval coverage as secondary checks. Period-aware backbones. TimesNet ( 26 ) uses FFT-based top- k period discovery to reshape the input into 2D tensors for convolution. DLinear ( 27 ) decomposes the input into trend and seasonal components. N-BEATS ( 22 ) is a residual stack of generic basis-expansion blocks. iTransformer ( 17 ) inverts the canonical Transformer:

Read the complete text