Forecasting With LLMs: Improved Generalization Through Feature Steering

Humzah Merchant, Bradford Levy · 2026

Read Forecasting With LLMs: Improved Generalization Through Feature Steering by Humzah Merchant, Bradford Levy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27199.txt.

Introduction

Merchant and Levy probe whether language models truly forecast or merely recall answers from training. A model asked about a past event can cheat by remembering the outcome rather than reasoning from what was knowable then. Sparse autoencoders, tools that split internal activity into readable features, inspect the model's inner states. Features tied to time-aware reasoning and to look-ahead bias both appear. The team then steers those features in a different domain during this 2026 interpretability study.

Boosting time-awareness features sharply cuts look-ahead bias on forecasting prompts. General reasoning performance stays intact under that push. Steering the suspected look-ahead features, by contrast, produces no clear effect. Readable temporal features can therefore causally shift models toward historically grounded reasoning. Forecasting evaluations gain a concrete guard against silent memorisation. The method also shows internal features can be tested as causes, not just correlations.

Catalog record

CREATORHumzah Merchant, Bradford Levy
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.27199

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Forecasting With LLMs: Improved Generalization Through Feature Steering Humzah Merchant, Bradford Levy Abstract Successful forecasting involves identifying patterns between historical and future states of the world which generalize to future observations. We apply LLMs to a variety of forecasting tasks and inspect their internal states using sparse autoencoders to understand whether they appear to rely on time-specific pieces of knowledge versus generalizable patterns. Our analyses identify features associated with both time-aware reasoning and look-ahead-biased reasoning. We then apply the LLMs to an entirely different domain and intervene on these features. We find that amplifying time-awareness features substantially reduces look-ahead bias on forecasting prompts while preserving general reasoning performance. In contrast, steering the candidate look-ahead-bias features does not produce an effect. These results suggest that interpretable temporal features can be used to causally shift LLMs toward more historically grounded reasoning. 1 Introduction The dominance of LLMs and transformer-based models across a variety of domains has led to a rise in the desire to apply them to forecasting. Forecasting is a difficult task because it requires learning generalizable insights, i.e., a mapping from the current state to future states, often times in highly stochastic and noisy environments. When an LLM is applied to a forecasting problem involving an outcome from its training data, the easiest way to arrive at an answer may simply be to recall the outcome from knowledge stored in the model’s parameters. While this answer will be correct, this is not a generalizable solution likely to yield comparable out-of-sample performance. Instead, one might prefer that an LLM reason over the information set which was valid at the time the forecast would have been made, i.e., the knowledge cutoff relative to the forecast period, to arrive at the best forecast a decision maker could have arrived in the moment. In this paper, we explore the extent to which reliance on memorization versus reasoning over knowledge is detectable. Specifically, we apply sparse autoencoders ( Huben et al. 2024 ) to identify features in LLMs associated with these concepts. We find that amplifying time-awareness in the model can reduce its reliance on memorized content “from the future” and induce it to focus on reasoning over knowledge available at the time the forecast would have been made (see, e.g., Appendix A ). Figure 1 : Amplifying time-awareness in models reduces reliance on knowledge from after a inference-time specified knowledge cutoff while maintaining utility. Error bars denote \pm 1 SE. There is precedent for the idea that temporal behavior can be induced in models. The Llama 2 technical report describes a small supervised fine-tuning intervention intended to improve time awareness ( Touvron et al. 2023 ) . In addition, among the models evaluated in Figure 3 , the Llama 3 series—which vastly outperform the other models on the simple evaluation—are also the only that include information about time and knowledge cutoff in the system prompt. However, such evidence does not reveal what internal representations support this behavior, nor whether time awareness and look-ahead bias correspond to distinct model features. We investigate this question directly. First, we use prediction-market data to identify Sparse Autoencoder features associated with two contrasting behaviors: reasoning in line with the historical expectation at the time of the market, and answering with information that appears to reflect post-hoc knowledge. We then test whether these features are causally relevant by amplifying them during generation. Importantly, we evaluate steering on free-form reasoning tasks rather than short structured outputs, using M&A and pharmaceutical forecasting settings where out of sample forecasting ability is \approx 0 and look-ahead bias can be identified in natural text. Our results show that features associated with time awareness generalize across tasks and can substantially reduce look-ahead bias, while candidate look-ahead-bias features do not yield the same effect. 2 Method Figure 2 : Overview of Method Sparse autoencoders (SAEs) decompose dense transformer activations into a much larger set of sparse, learned features. Given a hidden activation \mathbf{x} from a language model layer, an SAE produces feature activations \mathbf{z} and reconstructs the original activation: \mathbf{z}=\mathrm{ReLU}(\mathbf{W}_{\mathrm{enc}}\mathbf{x}+\mathbf{b}_{\mathrm{enc}}),\qquad\hat{\mathbf{x}}=\mathbf{W}_{\mathrm{dec}}\mathbf{z}+\mathbf{b}_{\mathrm{dec}}. The model is trained to preserve the original activation while only a small number of features to activate at once. Empirically, individual SAE features often correspond to human-interpretable concepts or patterns, making them useful for studying what information is represented inside a model. As illustrated in Figure 2 , we use SAE features to identify and intervene on temporal representations. If feature j is associated with time-aware reasoning or look-ahead-biased reasoning, we can test its causal role by modifying its activation during generation. In particular, we amplify a selected feature by adding a steering magnitude \alpha before decoding: \hat{\mathbf{x}}_{\mathrm{steered}}=\mathbf{W}_{\mathrm{dec}}\left(\mathbf{z}+\alpha\mathbf{e}_{j}\right)+\mathbf{b}_{\mathrm{dec}}, and insert the modified activation back into the language model. We then measure whether the model’s downstream behavior changes. We use the released Gemma Scope 2 dictionaries for Gemma 3 models ( Lieberum et al. 2024 ) which provides SAEs for layers 16, 31, 40 and 53, along with associated Neuronpedia annotations to help vet candidate temporal features surfaced by our activation-contrast procedure. In addition, we use the Qwen Scope ( Deng et al. 2026 ) for Qwen 3.5 27B, which provides SAEs for all layers though only includes Neuronpedia labels for layer 31. 3 Results Figure 3 : Knowledge of M&A activity and look-ahead bias vary independently across model families and do not have positive relationships even within all families. Dot size \propto\log_{10} number of parameters; error bars denote \pm 1 SE. 3.1 Identifying Temporal Features We use prediction-market data to identify features associated with temporal reasoning. Prediction markets record historical expectations for an unresolved event, which we can use as a ground truth. Under the assumption that markets aggregate all available information towards forecasting, and LLMs currently have no out of sample edge ( Yang et al. 2025 ) a perfectly calibrated forecaster should pick the market favorite. More generally, under the assumption that the model knows the outcome of all events, if the model picks the market favorite at the time that suggests time-aware reasoning while if the model picks the eventual outcome but not the favorite at the time that suggests look-ahead bias. An additional advantage is that prediction market questions are diverse, covering fields such as the economy, politics, and pop-culture, allowing the isolation of general features. To surface features associated with the two informative cases, we capture activations on the question tokens in prompts of the form: Today is {Month} {Day}, {Year}. {Question} where the date is the market opening date. We append answer choices, instruct the model to reason before answering, and require a final multiple-choice response. Each question is sampled four times at temperature 1.0 , and we retain only questions for which all four samples agree. We then rank SAE features by their difference in activation rate between unanimous time-aware and look-ahead-biased cases. For the Gemma 3 model, we use Neuronpedia annotations as a first pass on the highest-ranked candidates and discard features that appear tied to dataset artifacts rather than temporal reasoning. For example, \mathrm{L31{:}3831} initially surfaces as a candidate look-ahead-bias feature, but its label, “dates like 2023,” suggests specificity to our 2022–2023 question pool. After filtering, we retain five features: two associated with time-aware cases and three

Read the complete text