HyperLens: Quantifying Cognitive Effort in LLMs with Fine-grained Confidence Trajectory
Chengda Lu, Xiaoyu Fan, Wei Xu · 2026
Read HyperLens: Quantifying Cognitive Effort in LLMs with Fine-grained Confidence Trajectory by Chengda Lu, Xiaoyu Fan, Wei Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05741.txt.
Introduction
How hard is a language model thinking? Chengda Lu, Xiaoyu Fan, and Wei Xu argue the answer is written in its layer-by-layer confidence. They identify a magnification effect: deeper transformer layers inherently amplify small changes in confidence, yielding a fine-grained trajectory through inference.
Their probe, HyperLens, traces that trajectory and quantifies cognitive effort. Across eight models and eight datasets, trajectories for complex and simple tasks diverged consistently. The pattern distills into a practical signal for separating tasks that strain the model from routine ones, without extra computation. Such trajectories offer a window into inference dynamics that final answers alone cannot provide.
Catalog record
| CREATOR | Chengda Lu, Xiaoyu Fan, Wei Xu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05741
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HyperLens: Quantifying Cognitive Effort in LLMs with Fine-grained Confidence Trajectory Chengda Lu, Xiaoyu Fan, Wei Xu Abstract While Large Language Models (LLMs) achieve strong performance across diverse tasks, their inference dynamics remain poorly understood because of the limited resolution of existing analysis tools. In this work, we identify an intrinsic magnification mechanism in transformer architectures: deeper layers inherently magnify the small changes of layer-wise confidence, providing a fine-grained confidence trajectory. Building on this insight, we introduce HyperLens , a high-resolution probe designed to trace confidence trajectories and quantify the cognitive effort during inference. Across 8 8 LLMs and 8 8 datasets, HyperLens reveals a consistent divergence in confidence trajectories that separates complex from simple tasks. We abstract this pattern into a quantitative cognitive effort metric. Our analysis reveals a fundamental principle: complex tasks consistently require higher cognitive effort. Finally, we provide a mechanistic diagnosis of a common side effect of standard Supervised Fine-Tuning (SFT): it can reduce cognitive effort and consequently degrade performance on in-domain tasks. † † affiliationnotice: * Corresponding authors. 1 Introduction While Large Language Models (LLMs) exhibit remarkable capabilities, understanding their inference dynamics , i.e., how internal states evolve, is crucial for real-world robustness. A prevailing paradigm interpreting the dynamics involves monitoring the evolution of confidence via layer-wise projections ( 25 ; 30 ; 38 ; 3 ) . This signal is proven effective for various tasks like hallucination and jailbreak detection ( 7 ; 23 ) , and early-exit strategies ( 41 ; 5 ) . Figure 1 : The confidence trajectory of Qwen2.5-7B with different focal depths on easy (CoNaLa ( 45 ) ) versus hard (APPS ( 17 ) ) coding tasks. The confidence of three figures is derived with (1) zero focal depth, which is equivalent to Logit Lens; (2) shallow focal depth with one single layer; and (3) sufficient focal depth with five layers. Intuitively, the trajectory of confidence should reflect the model’s cognitive effort . Specifically, for trivial tasks (e.g., knowledge retrieval) where the answer is readily apparent, the internal confidence should converge swiftly, whereas complex tasks (e.g., hard math problems) necessitate a longer low-confidence “struggle” phase to reach the solution. However, observations using current probing methods fail to validate this intuition, frequently producing indistinguishable trajectories regardless of task difficulty. We attribute this failure to the limited resolution of existing probes, which limits the utility to only the most obvious anomalies (e.g., sudden spikes / collapses or stable changes). Existing probes primarily rely on static projections ( 1 ; 16 ; 27 ) , most notably the Logit Lens ( 1 ) , which decodes hidden states directly through the last unembedding layer. These methods rely on a strong assumption that intermediate representations are linearly aligned with the output space. Unfortunately, because of the anisotropy ( 14 ; 34 ) and representation drift ( 15 ; 36 ) , these methods function as a myopic observer : without proper focal correction, they fail to resolve latent signals effectively. To address this limitation, another line of work, like Tuned Lens ( 3 ) and others ( 11 ; 28 ; 16 ) introduces an external learned adapter to correct the projection misalignment. However, they inevitably act as a biased distortion filter : the external training introduces systematic distortions to the original signal ( 19 ; 37 ) . Unlike the current myopic or biased tools, we identify an inherent self-magnifying mechanism within transformer layers (Section 3 ). We prove that instead of directly decoding the hidden state, propagating it through a sequence of future layers before projection magnifies the derived confidence. We term the number of the future layers as focal depth . Based on this foundation, we propose a high-resolution probe, HyperLens , which reveals a critical phenomenon aligning with our intuition: deepening the focal depth magnifies confidence, with the magnitude constrained by the intrinsic difficulty of the task . As Figure 1 illustrates, under zero focal depth (i.e., equivalent to Logit Lens), the confidence trajectories for easy and hard tasks are indistinguishable across all layers. Increasing the focal depth by 1 introduces a global magnification, while the trajectories remain entangled. Notably, when we increase the focal depth to a sufficient level, the confidence trajectories for easy and hard tasks are clearly separated. The easy task exhibits a more rapid increase, whereas the hard task remains at a low-confidence state for much longer. This divergence indicates that sufficient magnification makes the model’s internal cognitive effort observable. To quantify this, we introduce the refinement area ( \Omega ) , defined as the area above the magnified confidence trajectory, serving as a precise measure of cognitive effort required to generate the final output. Intuitively, harder tasks require more cognitive effort to generate the final output, therefore have a higher \Omega . For instance, the \Omega of Qwen2.5-7B increases significantly from 1.78 (Easy) to 3.05 (Hard) in code tasks (Figure 1 (3)). Comprehensive evaluations across 8 models and 8 datasets (see Section 5 ) confirm that task difficulty consistently correlates with higher \Omega . Beyond robust measurement, HyperLens provides novel insights into the side effects of Supervised Fine-Tuning (SFT) (Section 6 ). We uncover a “blind confidence” phenomenon: simple SFT can cause abnormal decreases in \Omega on in-domain tasks, thereby harming the performance. This suggests that the model learns to shortcut the low-confidence “struggle” phase, jumping to obtaining the output without sufficient consideration. In summary, the contributions include: (1) We propose HyperLens , a high-resolution probe that visualizes and quantifies the model’s cognitive effort. Utilizing HyperLens , we clearly separate the confidence trajectories of easy and hard tasks, revealing that complex tasks necessitate a longer low-confidence phase. (2) We formally characterize the transformer’s inherent self-magnifying mechanism, proving that increasing focal depth strictly magnifies confidence, thereby ensuring a high-resolution view. (3) By applying HyperLens , we uncover that simple SFT causes a “blind confidence” problem by cutting the cognitive effort when facing in-domain tasks, providing an interpretability explanation for performance degradation often observed in SFT models. 2 Related Works Inference dynamics interpretation focuses on theoretically understanding the evolution of internal states. Intuitively, the inference includes two phases: (1) transform the input into a high-dimensional latent space; and (2) map the latent state to the vocabulary space to generate the next-token ( 26 ; 35 ; 13 ) Moving beyond this high-level decomposition, a growing body of research focuses on a more thorough analysis. For instance, 31 prove that the residual stream effectively performs Bayesian updates across layers from the geometric perspective. 33 model the inference process as a kernel renormalization flow, formalizing the evolution of predictor statistics in a thermodynamic perspective. Probing methods. To decode these internal dynamics, researchers propose a set of probing methods to observe the internal state. According to the dependency on extra parameters, these probing methods can be classified into static and parametric probing. Static probing methods include the widely adopted Logit Lens ( 25 ) and its variants like ( 44 ) , which uses cross-query semantic patching and a cosine-based representational lens to interpret the implicit reasoning phenomena. In this work, we focus on the static probing methods and mainly compare our method with the widely-adopted Logit Lens. Parametric