Patterning: The Dual of Interpretability

George Wang, Daniel Murfet · 2026

Read Patterning: The Dual of Interpretability by George Wang, Daniel Murfet (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13548.txt.

Introduction

George Wang and Daniel Murfet set up patterning as the twin task of interpretability. Interpretability works back from a trained net to how it acts past its train data. It picks the net's inner parts apart to read its rules.

Patterning runs the next way. Given a wished form of wide act, what train data yields it? Their tool is susceptibilities, marks of how out comes shift when the data spread is nudged a bit. Such marks can point to which data taught which act. For teams that must shape a model's acts by data picks, a twin task with its own tools is a fresh lever.

Catalog record

CREATORGeorge Wang, Daniel Murfet
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-13548

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Patterning: The Dual of Interpretability George Wang, Daniel Murfet Abstract Mechanistic interpretability aims to understand how neural networks generalize beyond their training data by reverse-engineering their internal structures. We introduce patterning as the dual problem: given a desired form of generalization, determine what training data produces it. Our approach is based on susceptibilities , which measure how posterior expectation values of observables respond to infinitesimal shifts in the data distribution. Inverting this linear response relationship yields the data intervention that steers the model toward a target internal configuration. We demonstrate patterning in a small language model, showing that re-weighting training data along principal susceptibility directions can accelerate or delay the formation of structure, such as the induction circuit. In a synthetic parentheses balancing task where multiple algorithms achieve perfect training accuracy, we show that patterning can select which algorithm the model learns by targeting the local learning coefficient of each solution. These results establish that the same mathematical framework used to read internal structure can be inverted to write it. 1 Introduction What is dual to interpretability? The goal of interpretability is to understand how neural networks generalize from their training data: for example, to reverse-engineer the internal structures (e.g. features, circuits) that determine how a trained model behaves on inputs beyond those it was trained on. If this is interpretability, then the dual problem should be: given a specified form of generalization, determine what training data leads to it. We call this patterning , and in this paper we develop foundations for patterning based on the same mathematical framework that underlies recent approaches to interpretability via susceptibilities ( 1 ; 25 ; 9 ) . The starting point is Structural Bayesianism ( 17 ) , the hypothesis that internal structure in neural networks is encoded in the local posterior distribution. The posterior p(w|D_{n}) , formed after observing training data D_{n} of size n , concentrates on parameters w that predict well on this dataset, and further its “shape” is sensitive to the computational mechanisms by which that prediction is achieved. Internal structures like circuits leave traces in the response of the loss to small perturbations: some weight changes are irrelevant to outputs, others can be compensated by coordinated changes elsewhere, and these “degeneracies” reflect the trade-offs among parts of the underlying algorithm. This structural information can be extracted by computing expectation values \mu^{n}_{i}=\int\phi_{i}(w)p(w|D_{n})dw (1) where the \phi_{i} are observables: functions on parameter space that probe aspects of model structure. We collect these into a vector \mu^{n}=(\mu^{n}_{1},\mu^{n}_{2},\ldots) of structural coordinates. Examples of such expectation values include estimators for the local learning coefficient \lambda ( 28 ; 13 ) , susceptibilities \chi ( 1 ) and influence functions ( 12 ) . To find the dual of interpretability, we study the affordances of structural coordinates. The posterior depends on three ingredients: the model architecture, the data distribution q , and the prior. The finite-sample coordinates \mu^{n} depend on the specific dataset D_{n} , but in the large-sample limit we obtain coordinates \mu^{\infty} that depend continuously on q . Formally, \mu^{\infty} is computed using the annealed posterior p(w|D_{\infty})\propto\exp\{-n\beta L(w)\}\varphi(w) where L(w)=\mathbb{E}_{q}[\ell(w)] is the population loss (see Section 2.2 for details). Let h denote a vector of hyperparameters governing the data distribution (for instance, mixture weights over a baseline distribution and a collection of probe distributions). Then d\mu^{\infty}=\chi\,dh (2) where \chi is the matrix of susceptibilities: the ik -entry \chi_{ik} measures how the expectation of observable \phi_{i} responds to an infinitesimal shift of the data distribution toward probe k . Given a desired change d\mu^{\infty}_{\mathrm{target}} in the structural coordinates, the minimum-norm intervention in the data distribution is dh_{\mathrm{opt}}=\chi^{\dagger}\,d\mu^{\infty}_{\mathrm{target}} (3) where \chi^{\dagger} denotes the Moore-Penrose pseudoinverse. We refer to ( 3 ) as the fundamental equation of patterning. In this way, we can derive principled -- one-off (applied once before training) or online (adjusted dynamically during training) -- changes in the data distribution that steer the configuration of the posterior and thus internal structure in the network. Since this structure determines generalization, we have found (in principle) a dual to interpretability: shaping training data to control generalization. 1 1 1 This reasoning assumes that SGD finds solutions favored by the Bayesian posterior. This correspondence is supported by some theoretical and empirical evidence ( 16 ) but remains a hypothesis rather than a theorem; our experiments provide further evidence by showing that posterior-guided interventions produce predicted effects under SGD training. The main claim of the present paper is that this works, in simple experiments . We demonstrate patterning in several settings, showing that susceptibility-guided interventions in training data produce predictable changes in the internal structures that form during training: • Modulating a circuit. The simplest application of patterning targets a single mode of the susceptibility matrix. In a 3M parameter language model, 1 showed that PC2 of the susceptibility matrix couples induction patterns in the data (roughly speaking, the right singular vector v_{2} ) to the induction circuit in the weights (the left singular vector u_{2} ). Setting d\mu^{\infty}_{\text{target}}=\pm u_{2} , the fundamental equation yields dh_{\text{opt}}\propto\pm v_{2} : the optimal intervention is simply to re-weight training data along the second principal data pattern. We show that up-weighting tokens with negative v_{2} loadings accelerates circuit formation, while down-weighting them delays or prevents it ( Section 3 ). • Selecting between competing algorithms. When multiple solutions achieve zero training loss, the principle of internal model selection determines which one the posterior favors: among equal-loss solutions, Bayesian inference prefers the one with lower local learning coefficient. Patterning can exploit this by raising the LLC of an undesired solution, shifting posterior weight toward the desired one. Following 15 , we study transformers trained on a bracket classification task where two algorithms – Nested and Equal-Count – both achieve perfect loss on the training data. The susceptibility matrix measures how each training sample affects each solution’s LLC. Inverting the fundamental equation, we show that the optimal re-weighting is approximately proportional to the susceptibility gap \chi^{EQ}_{x}-\chi^{N}_{x} . Samples with large gaps turn out to be interpretable (sequences that are “almost nested” or “almost equal”), and retraining on appropriately modified distributions shifts the proportion of models implementing each algorithm ( Section 4 ). We also perform sanity-check experiments to validate the basic paradigm: stunting the development of the spacing fin discovered in 25 and deliberately growing a similar fin for delimiters ( Section A.2 ). These experiments test a crux for the patterning methodology: susceptibilities must be sensitive to internal structure, and interventions in training data must produce measurable changes in those susceptibilities. Once this is established by the main experiments, the sanity checks become less central, so we relegate them to the appendix. In this paper we focus on one-off interventions in training data (made at the beginning of training), but in future work we plan to develop the full online form of patterning. Organization. Section 2 reviews the necessary background on singular learning theory and susceptibilities, and derives the patterning solution via mode decomposition. Section 3 demonstrates patterning in a small language model, showing that re-weighting training data along the second principal component of the susceptibility matrix accelerates or delays the formation of the induction circuit. Section 4 applies patterning

Read the complete text