Exemplar Partitioning for Mechanistic Interpretability
Jessica Rumbelow · 2026
Read Exemplar Partitioning for Mechanistic Interpretability by Jessica Rumbelow (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14347.txt.
Introduction
Rumbelow introduces Exemplar Partitioning, a method for building feature dictionaries from language model activations. Each EP dictionary splits activation space into Voronoi regions, formed by leader-clustering a stream of activations. A region is defined by a seen exemplar and a member average. They link regions to inputs, letting dictionaries be compared across layers, checkpoints, and models. Comparing EP dictionaries on base and tuned models shows tuning reorganises harmful activations at varied scales. Edits on these regions let both models answer harmful requests once refused.
Catalog record
| CREATOR | Jessica Rumbelow |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 99071 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14347
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exemplar Partitioning for Mechanistic Interpretability Jessica Rumbelow Abstract We introduce Exemplar Partitioning (EP), an unsupervised method for constructing interpretable feature dictionaries from Large Language Model (LLM) activations with ∼ 10 3 × \sim 10^{3}\times fewer tokens than comparable sparse autoencoders. An EP dictionary is a Voronoi partition of activation space, built by leader-clustering streamed activations within a distance threshold. Each region is anchored by an observed exemplar that serves as both its membership criterion and intervention direction; dictionary size is not prespecified, but determined by the activation geometry at that threshold. Because exemplars are observed rather than learned, dictionaries built from the same data stream are directly comparable across layers, models, and training checkpoints. This paper characterises EP as an interpretability object via targeted demonstrations of properties newly accessible through this construction, plus one head-to-head benchmark. In Gemma-2-2b, we find that EP dictionary regions are interpretable and support causal interventions: refusal in instruction-tuned Gemma concentrates in a region whose exemplar ablation can collapse held-out refusal. Cross-checkpoint matching between base and instruction-tuned dictionaries separates the directions preserved through finetuning from those introduced by it. EP regions and Gemma Scope SAE features decompose activation space differently, but agree on a shared core: ∼ 20 % {\sim}20\% of EP regions match an SAE feature at F 1 > 0.5 F_{1}>0.5 , and EP one-hot probes retain ∼ 97 % {\sim}97\% of raw-activation probe accuracy at ℓ 0 = 1 \ell_{0}=1 : the linearly-decodable identity that probing tests is largely preserved by density structure alone. Nearest-exemplar distance provides a free out-of-distribution signal at inference. On AxBench latent concept detection at Gemma-2-2B-it L20, EP at p 1 p_{1} reaches mean AUROC 0.881 0.881 , + 0.126 +0.126 over the canonical GemmaScope SAE leaderboard entry and within 0.030 0.030 of SAE-A’s 0.911 0.911 , at ∼ 10 3 × \sim 10^{3}\times less build compute. Code: https://github.com/jessicarumbelow/exemplar-partitioning . 1 Introduction Figure 1 : An Exemplar Partitioning dictionary built from Gemma-2-2B L12 activations at p_{2} ( K=5{,}129 ). Left: eight sample regions, each shown with its member count, its exemplar’s logit-lens ( 18 ) decode, and an excerpt of a member input with the activating tokens highlighted. Right: a PCA-projected 3D rendering of the Voronoi partition; each cell is one region, with a random selection also labelled with logit-lens decode. Mechanistic interpretability aims to reverse-engineer neural networks into human-legible algorithms. Three strands have emerged: circuit analysis, which identifies architectural components that implement specific behaviours ( 20 ; 28 ) ; supervised probing, which tests for human-specified concepts in activations ( 3 ) ; and unsupervised feature discovery, which identifies recurring units in activation space that downstream work names, compares across models, and intervenes on ( 7 ; 4 ; 25 ) . Each has a cost. Circuit analysis isolates per-task subgraphs rather than producing a model-wide feature inventory ( 6 ; 24 ) . Probing only surfaces concepts we know to look for. Sparse autoencoders (SAEs) – currently the dominant approach for feature discovery – produce addressable dictionaries through training that bundles reconstruction, sparsity, and a fixed dictionary size into a single objective ( 25 ; 15 ) . These commitments are natural for reconstructive decomposition, but not obviously required for interpretability. How much of the feature structure SAEs uncover is due to the training objective itself, and how much is already present in the model’s activation geometry and accessible by a more direct construction? The intuition: an SAE encoder is, in a sense, learning n soft or hard clusters of activations (depending on SAE flavour), where n is the fixed dictionary size. If we are willing to commit to a clustering directly, perhaps we can access different properties of the activation space that are not accessible under the other commitments of an SAE. Exemplar Partitioning does just this by explicitly clustering activation space instead of projecting it onto a learned basis. 2 Exemplar Partitioning Clustering activations into a dictionary of regions (interpretably labelled in some way) is a natural approach to unsupervised feature discovery, but the details of the construction matter for tractability and interpretability. Sparse autoencoders are trained on millions or billions of tokens – we cannot hold these in memory at once. Therefore, we use online clustering to build the dictionary in a single streaming pass over the activations. Online clustering requires a centroid to measure distances from, but we want to avoid averaging over streamed activations for two reasons – firstly, averaging arbitrary activations can produce centroids that are not coherent directions (i.e. they don’t correspond to any real region in the model’s learned activation space), which is not helpful for interpretability; and secondly, averaging causes centroids to shift over time as their cluster absorbs new members. This results in clustering instability, which typically presents as giant degenerate clusters that absorb most incoming activations as their centroid moves ever closer to the global mean. Thus we cluster on fixed exemplars. The exemplar is a real activation from the data stream, so it is a coherent representative of the region; and it is fixed at region creation, so the region’s identity and boundaries remain stable throughout the streaming process. It also enables direct comparison across all dictionaries built from the same data stream (regardless of layer, checkpoint, or even model), because an exemplar of one region in one dictionary will either be an exemplar in another, or will have been absorbed by an earlier region, which we can easily check via distance between it and that regions exemplar. This is a key advantage for interpretability: we can track how specific regions evolve across layers, models, and training checkpoints by following their shared and subsumed exemplars. Calibration and Centring. Leader-clustering requires a distance threshold by which to determine whether a new activation joins an existing cluster or seeds a new one. In order to select this threshold in a principled way, EP begins by sampling a fixed budget of calibration activations from the build corpus and computing pairwise distances between them. The distance threshold \theta_{p} is set to the p -th percentile of these distances, where p is a parameter chosen by the user to determine dictionary resolution. Reporting the threshold as a percentile rather than an absolute distance makes it comparable across layers and models – typical pairwise cosines differ, but p_{10} has the same operational meaning everywhere. Smaller p produces tighter, more granular regions and a larger dictionary; higher- p dictionaries are coarser and smaller. In practice, p is swept over a range to produce a family of dictionaries at different resolutions, and the downstream results are compared across that family. The same calibration activations can also be used to approximate the mean of the corpus \mu . In this construction, \mu is subtracted from each activation so that clusters reflect direction-only differences; cosine distance is the natural metric on the resulting unit sphere. Note that this approach was selected as a reasonable starting point, not because it is the only or necessarily the best one. The same construction could be done with other distance metrics (e.g. Euclidean rather than cosine), which would produce different dictionaries with different hypotheses about the geometry of activation space. We leave exploration of this design space to future