From Actions to Understanding: Conformal Interpretability of Temporal Concepts in LLM Agents

Trilok Padhi* Ramneet Kaur*, Krishiv Agarwal, Adam D. Cobb, Daniel Elenius, Manoj Acharya, Colin Samplawski, Alexander M. Berenbeim, Nathaniel D. Bastian, Susmit Jha Anirban Roy · 2026

Read From Actions to Understanding: Conformal Interpretability of Temporal Concepts in LLM Agents by Trilok Padhi* Ramneet Kaur*, Krishiv Agarwal, Adam D. Cobb, Daniel Elenius, Manoj Acharya, Colin Samplawski, Alexander M. Berenbeim, Nathaniel D. Bastian, Susmit Jha Anirban Roy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19775.txt.

Introduction

Padhi and 11 colleagues present a framework for interpreting how concepts evolve over time inside language model agents, using a step-wise conformal lens. The method combines step-wise reward modeling with conformal prediction to label each step's internal representation as good or failing. Linear probes then learn from those representations to find directions of concept change over time, revealing the hidden workings behind sequential agent behavior.

Catalog record

CREATORTrilok Padhi* Ramneet Kaur*, Krishiv Agarwal, Adam D. Cobb, Daniel Elenius, Manoj Acharya, Colin Samplawski, Alexander M. Berenbeim, Nathaniel D. Bastian, Susmit Jha Anirban Roy
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42018
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19775

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Actions to Understanding: Conformal Interpretability of Temporal Concepts in LLM Agents Trilok Padhi* Ramneet Kaur*, Krishiv Agarwal, Adam D. Cobb, Daniel Elenius, Manoj Acharya, Colin Samplawski, Alexander M. Berenbeim, Nathaniel D. Bastian, Susmit Jha Anirban Roy Abstract Large Language Models (LLMs) are increasingly deployed as autonomous agents capable of reasoning, planning, and acting within interactive environments. Despite their growing capability to perform multi-step reasoning and decision-making tasks, internal mechanisms guiding their sequential behavior remain opaque. This paper presents a framework for interpreting the temporal evolution of concepts in LLM agents through a step-wise conformal lens. We introduce the conformal interpretability framework for temporal tasks , which combines step-wise reward modeling with conformal prediction to statistically label model’s internal representation at each step as successful or failing. Linear probes are then trained on these representations to identify directions of temporal concepts—latent directions in the model’s activation space that correspond to consistent notions of success, failure or reasoning drift. Experimental results on two simulated interactive environments, namely ScienceWorld and AlfWorld, demonstrate that these temporal concepts are linearly separable, revealing interpretable structures aligned with task success. We further show preliminary results on improving an LLM agent’s performance by leveraging the proposed framework for steering the identified successful directions inside the model. The proposed approach, thus, offers a principled method for early failure detection as well as intervention in LLM-based agents, paving the path towards trustworthy autonomous language models in complex interactive settings. 1 Introduction Figure 1 : A. We consider the problem of temporal interpretability of LLM agents trained to perform sequential tasks in complex environments. B. Proposed framework combines step-wise reward modeling with conformal labeling to distinguish success and failure at each timestep. Linear probes are trained on the model’s internal representations to test the hypothesis that step-wise notions of success and failure are linearly separable within the representation space. Large Language Models (LLMs) have rapidly evolved from static text generators ( 1 ; 33 ) to autonomous agents capable of navigating, planning, and acting in the physical world ( 37 ; 31 ; 16 ) . When deployed in embodied simulators such as Alfworld ( 30 ) , and ScienceWorld ( 38 ) , these agents exhibit impressive decision-making abilities on sequential goals for accomplishing the assigned task ( 21 ) . However, the reasoning process behind such sequential behavior remains largely opaque as these LLM agents continue to operate as black box , leaving practitioners without reliable explanations for why a model succeeds or where it fails within the task trajectory. This has led to increasing interest in the interpretability of LLMs, aimed at uncovering how models process inputs to generate outputs in a manner that is transparent and understandable to humans. Traditional interpretability frameworks such as attribute or feature visualization ( 40 ; 23 ) , mechanistic interpretability via sparse autoencoders ( 9 ) , activation space analysis via representation engineering ( 46 ) and universal steering ( 6 ) offer valuable insights into mappings between standalone input and the model’s latent space. These approaches are, however, fundamentally limited in capturing temporal dynamics within LLM-based agents as they operate on single standalone inputs such as a image, text or an image-text pair. While performing a sequential task in an interactive environment, the agent’s decision at any step depends not only on the current observation but also on the entire trajectory history with prior reasoning traces, environment responses, and accumulated context. Model’s internal representation space evolves through time, encoding both correct and incorrect reasoning directions. Understanding these evolving representations requires a temporally aware interpretability framework interpretability. A straightforward approach to adapting the existing interpretability frameworks for temporal dynamics would be treating the entire trajectory as a single input and performing analysis on those. This approach, however, does not provide any granularity on which intermediate step of the agent contributed to success or failure of the assigned task. An agent that successfully completes the task after ten steps hides the possibility that six of those might be suboptimal or irrelevant. Without a step-level notion of “success”, it becomes impossible to pinpoint when the model’s internal dynamics shift from aligned reasoning to failure modes such as hallucination or irrelevant exploration. A motivating example of the Llama-2 ( 33 ) agent trained to perform household tasks in the AlfWorld environment is as follows. For the assigned task of “placing a roll of paper next to the toilet” , we observed that the agent starts on the right track by moving towards the toilet and searching shelves to locate the roll. Yet midway through the trajectory, its internal reasoning begins to drift. Instead of continuing the search, the agent hallucinates that a non-existent drawer contains the paper roll, issuing an invalid action sequence. This behavioral deviation—despite earlier correct reasoning—leads to task failure, even though the initial steps were coherent and purposeful. Such examples (more shown in Fig. 2 ) highlight a central challenge in temporal interpretability : the final success or failure of a task often obscures where in the sequence the agent’s internal representations began to diverge. A trajectory that ultimately fails may contain multiple locally successful steps, while an apparently successful run may involve accidental recoveries after erroneous decisions. Current interpretability methods—focused on isolated prompts, static embeddings, or single-step explanations—cannot capture these evolving internal transitions, nor can they identify when the model’s reasoning begins to deviate. This work introduces a step-wise conformal interpretability framework that transforms the problem of agent’s final evaluation on a multi-step sequential task into temporal representation analysis . We posit that the directions of success and failure are geometrically separable in an LLM fine-tuned to perform sequential tasks, .i.e, these models internally have separable notions of success and failure. To test this hypothesis, we build the proposed interpretability framework on the following three key components (Fig. 1 (B)): 1. Step-Wise Reward Modeling: We follow 41 ’s approach on generating fine-grained step-wise rewards using Monte Carlo sampling over future trajectories. This transforms sparse final rewards into dense temporal feedback signals on success (or failure) of every step. 2. Statistical Labeling via Conformal Prediction: We propose using Inductive Conformal Prediction (ICP) to label model’s internal representation at each step as successful or failing with provable confidence bounds. 3. Probing Latent Directions: We train classifiers (or linear probes) on layer and time-conditioned agent’s activations to distinguish the latent space of success from failure. Key Findings. High accuracy and F1-scores of linear probes on Llama2-7B in two complex simulated environments, namely ScienceWorld ( 38 ) and AlfWorld ( 30 ) , validate our hypothesis that an LLM fine-tuned for sequential tasks develops an internal notion of linearly separable step-wise success across (a) timesteps, (b) layers, and (c) domains: both in-distribution and out-of-distribution. Steering the model towards Success. As observed in prior work ( 28 ; 39 ) and validated in our experiments, these agents lack the intrinsic ability to self-correct themselves back towards the success directions and often continue drifting along failure trajectories once the deviation occurs. The proposed framework lays the foundation for steering LLM agents towards the right direction by enabling targeted interventions when early signs of hallucination or misalignment emerge in their internal representations. We conduct preliminary experiments

Read the complete text