AttnTrace : Attention-based Context Traceback for Long-Context LLMs

Yanting Wang, Runpeng Geng, Ying Chen, Jinyuan Jia · 2025

Read AttnTrace : Attention-based Context Traceback for Long-Context LLMs by Yanting Wang, Runpeng Geng, Ying Chen, Jinyuan Jia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03793.txt.

Introduction

Yanting Wang, Runpeng Geng, Ying Chen, and Jinyuan Jia trace which passages in a long prompt made a language model answer as it did, in a 2025 paper. Such traceback supports forensic checks after attacks and builds trust in grounded answers. Existing tools can take hundreds of seconds per answer. Their method, AttnTrace, reads the model's own attention weights instead, with two tricks to make those weights speak clearly.

Tests show AttnTrace is both more accurate and much faster than the best prior methods. Bolted in front of a detector, it also sharpened the spotting of injected instructions in long contexts. As a live demo, it pinpointed planted instructions inside a paper written to game machine reviewers. The authors release the code.

Catalog record

CREATORYanting Wang, Runpeng Geng, Ying Chen, Jinyuan Jia
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-03793

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AttnTrace : Attention-based Context Traceback for Long-Context LLMs Yanting Wang, Runpeng Geng, Ying Chen, Jinyuan Jia Abstract Long-context large language models (LLMs), such as Gemini-2.5-Pro and Claude-Sonnet-4, are increasingly used to empower advanced AI systems, including retrieval-augmented generation (RAG) pipelines and autonomous agents. In these systems, an LLM receives an instruction along with a context—often consisting of texts retrieved from a knowledge database or memory—and generates a response that is contextually grounded by following the instruction. Recent studies have designed solutions to trace back to a subset of texts in the context that contributes most to the response generated by the LLM. These solutions have numerous real-world applications, including performing post-attack forensic analysis and improving the interpretability and trustworthiness of LLM outputs. While significant efforts have been made, state-of-the-art solutions such as TracLLM often lead to a high computation cost, e.g., it takes TracLLM hundreds of seconds to perform traceback for a single response-context pair. In this work, we propose AttnTrace , a new context traceback method based on the attention weights produced by an LLM for a prompt. To effectively utilize attention weights, we introduce two techniques designed to enhance the effectiveness of AttnTrace , and we provide theoretical insights for our design choice. We also perform a systematic evaluation for AttnTrace . The results demonstrate that AttnTrace is more accurate and efficient than existing state-of-the-art context traceback methods. We also show AttnTrace can improve state-of-the-art methods in detecting prompt injection under long contexts through the attribution-before-detection paradigm. As a real-world application, we demonstrate that AttnTrace can effectively pinpoint injected instructions in a paper designed to manipulate LLM-generated reviews. The code is at https://github.com/Wang-Yanting/AttnTrace . The demo is at https://huggingface.co/spaces/SecureLLMSys/AttnTrace . 1 Introduction Large language models (LLMs), such as Claude-Sonnet-4 [ 3 ] , Gemini-2.5-Pro [ 22 ] , and GPT-4.1 [ 42 ] , serve as the foundation to empower systems such as autonomous agents [ 60 , 69 , 1 ] and retrieval-augmented generation (RAG) [ 25 , 28 ] . Given an instruction and a context as input, an LLM can generate a response grounded in the provided context by following the given instruction. Thanks to the long context capabilities of LLMs, i.e., the ability to process an input prompt with tens of thousands of tokens, these LLM-empowered systems can leverage long contextual information collected from external environments to effectively perform user tasks, such as question answering. For instance, given a user question, an RAG system can retrieve relevant texts from a knowledge database and use the retrieved texts as the context to enable an LLM to provide a more accurate and up-to-date answer. Similarly, in autonomous agents, an LLM can leverage texts in the memory collected from the environment to make informed decisions. Figure 1: AttnTrace can trace back to embedded prompts in a context that manipulate LLM outputs. Section 4.6 shows a case study for pinpointing injected prompts manipulating the LLM-generated review. Research question and motivation: As an LLM can generate a response based on a context, a research question is: given a context and a response generated by an LLM based on that context, which subset of texts in the context—where each text may be a paragraph, passage, or sentence—contributes most significantly to the response? To answer this question, many context traceback methods [ 38 , 21 , 15 , 58 ] were proposed in past years, aiming to identify and attribute the most influential parts of the input context that induce the LLM’s response. Context traceback has many real-world applications in enhancing the security, safety, and trustworthiness of LLM-empowered systems. From a security perspective, context traceback can be used for post-attack forensic analysis [ 46 , 51 , 58 ] (serving as a complementary approach to prevention- and detection-based defenses) by pinpointing the adversarial texts that induce an LLM to generate malicious responses. For instance, we can trace back to malicious instructions crafted by prompt injection attacks [ 64 , 20 , 23 , 32 ] that mislead an LLM-empowered agent to perform a malicious action; we can also identify malicious texts crafted by knowledge corruption attacks [ 75 ] that induce a RAG system to generate an incorrect answer. Beyond adversarial settings, context traceback can also be used to enhance the transparency and interpretability of LLM-empowered systems deployed in high-stakes environments, e.g., in safety-critical domains such as healthcare or finance, context traceback helps verify whether an LLM’s recommendation is grounded in trustworthy information, reducing the risk of hallucination or reliance on outdated data. Existing context traceback solutions and their limitations: In the past decade, many feature attribution methods [ 53 , 52 , 54 , 35 , 47 ] such as Shapley [ 35 ] and LIME [ 47 ] have been proposed to quantify the contribution of individual input features to the output of a machine learning model. By viewing each text in the context as a feature, these methods can be applied for context traceback. For instance, Wang et al. [ 58 ] proposed TracLLM, a generic context traceback framework that can leverage existing feature attribution methods for context traceback. By comparing with existing feature attribution methods (including perturbation-based [ 35 , 47 ] , gradient-based [ 53 ] , and LLM citation-based [ 38 , 21 ] methods), Wang et al. [ 58 ] showed that their context traceback framework achieves state-of-the-art performance. However, as shown in our experiments, existing context traceback methods, including TracLLM, still achieve a sub-optimal performance and/or incur a high computation cost. For instance, while TracLLM [ 58 ] improves the efficiency of previous feature attribution methods such as Shapley for context traceback under long context, it still takes hundreds of seconds to perform the traceback for a response-context pair. The reason is that TracLLM is still based on the perturbation-based feature attribution methods, which need to collect many output behaviors from an LLM under various perturbed versions of the input context. This process is inherently computationally expensive, as it involves many forward passes through LLMs to measure the contribution of each text in the context. Additionally, these perturbation-based methods leverage the conditional probability of an LLM in generating an output under perturbed contexts. However, the conditional probability can be noisy and influenced by various other factors, such as context length. As a result, these methods also achieve a sub-optimal performance as shown in our results. We note that other context traceback methods–such as those leveraging gradient information [ 53 , 52 , 54 ] or prompting LLM [ 38 , 21 ] –also achieve a sub-optimal performance as shown in previous work [ 58 , 15 ] and validated in our results. Our work: In this work, we propose AttnTrace (visualized in Figure 1 ), a new context traceback method based on the attention weights produced by an LLM for a prompt. Existing LLMs are based on the Transformer architecture [ 56 ] that leverages attention mechanisms to capture contextual relationships between tokens. In particular, given an input prompt consisting of a sequence of tokens, each token is represented as a vector in each layer. The representation vector of each token is iteratively updated through the attention mechanism in each layer of the

Read the complete text