InTraVisTo: Inside Transformer Visualisation Tool

Nicolò Brunello * , Davide Rigamonti * , Andrea Sassella * , Vincenzo Scotti † , Mark James Carman * · 2025

Read InTraVisTo: Inside Transformer Visualisation Tool by Nicolò Brunello * , Davide Rigamonti * , Andrea Sassella * , Vincenzo Scotti † , Mark James Carman * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13858.txt.

Introduction

Nicolò Brunello and co-authors built InTraVisTo, a tool for peering inside large language models, in a 2025 paper. Models have grown strong but stay hard to predict. A user sees the final words, not the steps that made them. The tool traces how each word forms as it moves through the model layers.

Two views drive the design. One view decodes word states at each layer, so a user can watch a half-formed word sharpen step by step. The other view maps how facts flow between parts of the model with a Sankey diagram, a chart of thick and thin streams. The authors aim the tool at teams who test and ship these models. Clear views of the inner steps may help them spot odd acts and fix flaws before release.

Catalog record

CREATORNicolò Brunello * , Davide Rigamonti * , Andrea Sassella * , Vincenzo Scotti † , Mark James Carman *
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-13858

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

InTraVisTo: Inside Transformer Visualisation Tool Nicolò Brunello * , Davide Rigamonti * , Andrea Sassella * , Vincenzo Scotti † , Mark James Carman * Abstract The reasoning capabilities of Large Language Models (LLMs) have increased greatly over the last few years, as have their size and complexity. Nonetheless, the use of LLMs in production remains challenging due to their unpredictable nature and discrepancies that can exist between their desired behavior and their actual model output. In this paper, we introduce a new tool, InTraVisTo (Inside Transformer Visualisation Tool), designed to enable researchers to investigate and trace the computational process that generates each token in a Transformer-based LLM. InTraVisTo provides a visualization of both the internal state of the Transformer model (by decoding token embeddings at each layer of the model) and the information flow between the various components across the different layers of the model (using a Sankey diagram). With InTraVisTo, we aim to help researchers and practitioners better understand the computations being performed within the Transformer model and thus to shed some light on internal patterns and reasoning processes employed by LLMs. I Introduction The widespread adoption of Large Language Models (LLMs) is having a game-changing impact on Natural Language Processing applications in areas from medicine and education, to finance and personal assistants [ 1 , 2 ] . While these large new models come with greatly improved language understanding and text generation skills [ 3 , 4 ] , the underlying Transformer Neural Network they rely on [ 5 ] , is still primarily a black-box, in the sense that it is hard to understand exactly how the low level data manipulations it performs can lead to the incredible high level reasoning outputs observed. This lack of understanding represents a major issue, especially if considered together with the occurrence of occasional faults in the reasoning process, which lead to imprecise, nonsensical or non-factual outputs – i.e., “hallucinations” [ 6 ] . The combination of the lack of interpretability and the presence of errors limits the reliability of LLMs, preventing their deployment in cases where sound reasoning and coherent output are critical. Thus, better understanding of the internal reasoning mechanisms and the ability to track down the source of errors are essential to fully exploit LLMs’ capabilities. Fig. 1 : InTraVisTo application overview: given an input prompt and the generate configurations, a LLM generates a completion. We track the hidden (i.e., internal) states produced by the LLM during this process to visualise both these internal states and the information flow using, respectively, a heatmap and a Sankey diagram. Fig. 2 : The main control panel of InTraVisTo. The Input Prompt text box on the top left corner allows the user to provide an input to the LLM, the Generate button right after starts the generation process on the selected model (the Model dropdown allows to change the LLM), and, finally, the Output Text box contains the completion to the prompt generated by the LLM. All the remaining controls allow to modify the decoding process and the visualisations. With this goal in mind, we introduce InTraVisTo 1 1 1 “intravisto” is Italian word meaning “glimpsed”. (Inside Transformer Visualisation Tool), which is an open-source visualisation tool 2 2 2 https://github.com/daviderigamonti/InTraVisTo depicting the internal computations performed within a Transformer. The tool 3 3 3 live demo: https://tinyurl.com/2wtudxhf , shown schematically in Figure 1 , provides visualisations of both the internal state of the LLM , using a heat-map of decoded embedding vectors for all layer/token positions, and the information flow between components of the LLM , using a Sankey diagram to depict paths through which information accumulates to produce next-token predictions. The primary contribution of InTraVisTo is its capability to enable researchers and practitioners to interpret and manipulate large language models in real time through a comprehensive set of customizable features, all without requiring any programming expertise. II InTraVisTo We now describe the mechanisms and components of the tool. For each main section, we first examine the underlying mathematical implementation, followed by the corresponding application in a use case to offer a practical perspective on the tool and its capabilities. Reference [ 7 ] claimed that some model reasoning errors could be tied back to attention glitches , minor errors propagated throughout the attention pattern in the model, leading to imperfect state information transferred through the layers. They tested the model on the Flip-Flop language, but a simpler yet effective use case turned out to be reversing sequence s. Thus, the input prompt for our example is: Write numbers in reverse order. Number: 13843234 Reverse: . II-A Decoding Internal States Fig. 3 : Transformer model architecture. The high-level view of the architecture is represented on the left. In the centre there are the details of a single transformer layer and on the right the details of the language modeling head . Decoding the meaning of hidden state (aka embedding ) vectors at various depths of a Transformer stack is essential for providing an intuition as to how the model is working. We focus on causal (i.e., auto-regressive) models, and assume the state-of-the-art LLM architecture shown in Figure 3 , which involves a decoration process whereby each Transformer layer adds the results of its transformations to a residual embedding vector from the layer below. The final embedding vector is then decoded by a language modeling head to produce a probability distribution for the next token given the previous tokens. InTraVisTo provides a human-interpretable representation of this internal decoration pipeline by decoding each hidden state with a specific decoder and displaying the most likely token from the model’s vocabulary. Given a hidden state vector \mathbf{x}\in\mathbb{R}^{d} , which could be the output of a self-attention or feed-forward layer or a residual connection within or outside a transformer block, and a decoder matrix \mathbf{W}_{\textit{decoder}}\in\mathbb{R}^{d\times|\mathcal{V}|} [ 8 ] consisting of token embeddings, we can decode the state by normalising the vector by its RMS value, multiplying by the decoder, and applying the softmax: \displaystyle P(\textit{token})=\mathrm{softmax}\left(\frac{\mathbf{x}}{\sqrt{\frac{1}{d}\sum_{i}x_{i}^{2}}}\cdot\mathbf{W}_{\textit{decoder}}\right) (1) to produce a probability distribution over the vocabulary, from which the most likely token is chosen. Two natural decoders to use are (i) the transpose of the input embedding matrix \mathbf{W}_{\textit{in}}^{\top} used by the model to convert tokens to vectors on input, and (ii) the output decoder \mathbf{W}_{\textit{out}} used upon output within the language modeling head. Some models, like GPT-2 [ 9 ] and Gemma [ 10 ] tie these two parameter matrices together during training ( \mathbf{W}_{\textit{in}}^{\top}=\mathbf{W}_{\textit{out}} ) greatly simplifying the decoding process. Popular models like Mistral [ 11 ] and Llama [ 12 ] , however, allow these two matrices to differ ( \mathbf{W}_{\textit{in}}^{\top}\neq\mathbf{W}_{\textit{out}} ), which means that earlier layers tend to be much more interpretable when decoded with the input embedding ( \mathbf{W}_{\textit{in}}^{\top} ) while latter layers are more meaningful if the output decoder ( \mathbf{W}_{\textit{out}} ) is used. Previous works have looked to train specialised decoders [ 13 ] for this purpose, at the cost of introducing a great deal of additional complexity and potential errors. In InTraVisTo we instead make use of a simpler, yet effective, alternative by interpolating the input and output decoders based on the depth l\in[0,L] of the model layer we wish

Read the complete text