Reasoning Stabilization Point: A Training-Time Signal for Stable Evidence and Shortcut Reliance

Sahil Rajesh Dhayalkar · 2026

Read Reasoning Stabilization Point: A Training-Time Signal for Stable Evidence and Shortcut Reliance by Sahil Rajesh Dhayalkar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11625.txt.

Introduction

Sahil Rajesh Dhayalkar tracks when a model's reasoning settles in training. The 2026 preprint defines the Reasoning Stabilization Point, or RSP. Fine-tuning a pretrained model can lift task scores quietly. It can also shift the evidence the model relies on.

The view here is a training-time interpretability lens. Token-level attributions are tracked across fine-tuning epochs. Explanation drift is the epoch-to-epoch change in those attributions. It is measured on a fixed probe set of inputs. RSP is the earliest epoch after which drift stays low. It needs no tuning on out-of-distribution data at all. A stable point can flag shortcut reliance early. Trainers gain a signal for when to stop or intervene.

Catalog record

CREATORSahil Rajesh Dhayalkar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-11625

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reasoning Stabilization Point: A Training-Time Signal for Stable Evidence and Shortcut Reliance Sahil Rajesh Dhayalkar Abstract Fine-tuning pretrained language models can improve task performance while subtly altering the evidence a model relies on. We propose a training-time interpretability view that tracks token-level attributions across fine-tuning epochs. We define explanation drift as the epoch-to-epoch change in normalized token attributions on a fixed probe set, and introduce the Reasoning Stabilization Point (RSP), the earliest epoch after which drift remains consistently low. RSP is computed from within-run drift dynamics and requires no tuning on out-of-distribution data. Across multiple lightweight transformer classifiers and benchmark classification tasks, drift typically collapses into a low, stable regime early in training, while validation accuracy continues to change only marginally. In a controlled shortcut setting with label-correlated trigger tokens, attribution dynamics expose increasing reliance on the shortcut even when validation accuracy remains competitive. Overall, explanation drift provides a simple, low-cost diagnostic for monitoring how decision evidence evolves during fine-tuning and for selecting checkpoints in a stable-evidence regime. 1 Introduction † † footnotetext: This paper is under review at ACL Rolling Review Fine-tuning pretrained language models delivers strong performance with modest compute, but it can also change how a model arrives at decisions in ways that are not visible from accuracy alone. In NLP, models often exploit dataset artifacts and shallow heuristics that perform well in-domain yet fail under distribution shift 8 ; 13 ; 6 ; 16 ; 7 ; 14 . This motivates interpretability methods that expose decision evidence, such as feature-attribution and saliency 18 ; 23 ; 12 ; 21 . However, explanations are typically reported as static snapshots (e.g., after selecting the best checkpoint), leaving open a basic question: how do explanations evolve during fine-tuning time? We propose a lightweight, training-time interpretability view: treat token-level explanations as a time series over epochs. Intuitively, if fine-tuning is progressively “locking in” decision evidence, attributions should become stable; conversely, continued changes in evidence with small accuracy gains may reflect shifting reliance toward brittle cues. Concretely, we define explanation drift as the change in normalized token attributions between consecutive epochs on a fixed probe set, and we summarize the drift curve by a single interpretable event called the Reasoning Stabilization Point (RSP), the earliest epoch after which drift remains low for a short window. RSP is fully determined by the drift dynamics of a run and does not require tuning on OOD (Out-Of-Distribution) data. Our focused contributions are: • Explanation Drift: a model-agnostic scalar curve measuring epoch-to-epoch change in token attributions on a frozen probe set. • Reasoning Stabilization Point (RSP): a compact training-time signal that marks when token-importance patterns become stable. • Empirical evidence: drift stabilizes by (or before) accuracy saturation and reveals shortcut reliance hidden by validation accuracy. Because the method is post-hoc over saved checkpoints and uses off-the-shelf gradient-based attributions, it adds minimal overhead beyond standard fine-tuning. We view RSP as a practical diagnostic for deciding when a model’s decision evidence stabilizes, and for surfacing shortcut adoption that may be hidden by aggregate accuracy. 2 Related Work Post-hoc explanations in NLP. A large body of work studies explanations via feature attribution 3 ; 11 ; 4 and local surrogate models, including LIME 18 and Shapley-based frameworks such as SHAP 12 . Gradient-based attributions (e.g., Integrated Gradients) are widely used for neural models because they are simple and require no auxiliary explainer 23 . Parallel to attribution, rationale-based approaches aim to produce human-readable evidence, with benchmarks such as ERASER supporting systematic evaluation of rationales and faithfulness 5 . Our work is complementary: we do not propose a new explainer or rationale extractor, but rather a temporal lens over any token-level explanation function. Faithfulness, sanity checks, and stability of explanations. The reliability of explanations has been scrutinized via sanity checks and robustness criteria. 1 showed that some saliency maps can be insensitive to model parameters or data, motivating careful evaluation of explanation methods. Robustness of interpretability methods, whether similar inputs yield similar explanations, has also been formalized and empirically tested 2 . Debate around using internal signals such as attention as explanations further underscores the need for principled comparison and diagnostics 9 ; 26 ; 20 . Our approach differs from input-perturbation robustness: we measure stability across training time on a fixed probe set, asking when the model’s token-importance profile becomes consistent across epochs. Shortcut learning, dataset artifacts, and training dynamics. Models can achieve high benchmark performance by exploiting spurious correlations and annotation artifacts 8 , and NLI systems in particular often adopt shallow heuristics that fail on challenge sets such as HANS 13 . More broadly, shortcut learning has been argued to underlie many failures of generalization 6 . Prior tools for understanding training behavior include data-centric analyses (e.g., influence functions linking predictions to training points) 10 . Our work contributes a complementary, low-cost training-time diagnostic: explanation drift and RSP summarize how decision evidence shifts (or stabilizes) as fine-tuning proceeds, and can help identify when continued fine-tuning may be reallocating importance toward brittle cues rather than learning robust features. 3 Explanation Drift Across Fine-Tuning Time Main claim. We study how a fine-tuned classifier arrives at its predictions over training time, not just whether it is accurate. Our central claim is that token-level explanations (attributions) form a meaningful time series during fine-tuning and often stabilize earlier than peak in-domain validation accuracy. We operationalize this phenomenon via explanation drift , the change in attribution patterns between successive epochs, and define a single interpretable training-time event, the Reasoning Stabilization Point (RSP) , as the first epoch where drift becomes small and remains small. Intuitively, RSP marks when the model “locks in” a relatively stable set of decision features. When fine-tuning continues past RSP, we often observe diminishing accuracy gains alongside non-trivial residual drift, consistent with the model reallocating importance toward brittle or shortcut features. A practical corollary is that early stopping at (or near) RSP can preserve or improve robustness (e.g., OOD/challenge performance) while matching in-domain accuracy (TODO can perhaps remove this corollary) 3.1 Setup and token-level explanations Let f_{\theta_{t}} denote a classifier after epoch t\in\{1,\ldots,T\} of fine-tuning, where \theta_{t} are the model parameters at epoch t . We fix a probe set P=\{(x_{i},y_{i})\}_{i=1}^{n} (e.g., a small held-out subset of the development set) that remains constant across epochs. Fixing P ensures that changes in explanations reflect changes in the model, not changes in sampled inputs. For an input sequence x=(w_{1},\ldots,w_{|x|}) , we compute a token-level explanation \displaystyle E_{\theta}(x)\in\mathbb{R}^{|x|}, (1) \displaystyle E_{\theta}(x)_{j}\;\;\text{is the attribution for token }w_{j}. We instantiate E_{\theta} using standard gradient-based attribution methods over the embedding layer (e.g., integrated gradients or gradient \times input), which are lightweight and require no auxiliary models. To compare explanations across epochs, we convert raw attributions into a normalized distribution over tokens. Specifically, we take absolute values (capturing magnitude of influence regardless of sign) and normalize onto the simplex: \tilde{E}_{\theta}(x)_{j}\;=\;\frac{\left|E_{\theta}(x)_{j}\right|}{\sum_{k=1}^{|x|}\left|E_{\theta}(x)_{k}\right|+\varepsilon}, (2) where \varepsilon is a small constant for numerical stability. This makes \tilde{E}_{\theta}(x) comparable across epochs and across inputs of different lengths. 3.2 Explanation drift We quantify how a model’s reasoning evidence changes from epoch to epoch by measuring the discrepancy between successive normalized explanations on the same input. For each example x\in P ,

Read the complete text