Tapping into the Black Box: Uncovering Aligned Representations in Pretrained Neural Networks
Maciej Satkiewicz · 2025
Read Tapping into the Black Box: Uncovering Aligned Representations in Pretrained Neural Networks by Maciej Satkiewicz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22832.txt.
Introduction
Maciej Satkiewicz, Roberto Corizzo, and Marcin Pietroń offer a new way to see what a deep network has learned, in a 2025 paper titled Pulling Back the Curtain on Deep Networks. In simple linear models, a weight vector points at the input pattern the model prefers. In deep networks, gradient-based pictures tend to look brittle or adversarial instead.
Their method, Semantic Pullbacks, treats each network as an input-dependent affine operator. The adjoint action then pulls a neuron's preferred direction back into input space. Extra softening and iterative enhancement steps sharpen the picture into coherent local structure. The view unifies older ideas such as SmoothGrad and feature accentuation. Across convolutional and transformer models, the explanations scored best overall on faithfulness, stability, and target sensitivity. The authors stress the method is cheap and drops into existing pipelines.
Catalog record
| CREATOR | Maciej Satkiewicz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22832
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tapping into the Black Box: Uncovering Aligned Representations in Pretrained Neural Networks Maciej Satkiewicz Abstract In this paper we argue that ReLU networks learn an implicit linear model we can actually tap into. We describe that alleged model formally and show that we can approximately pull its decision boundary back to the input space with certain simple modification to the backward pass. The resulting gradients (called excitation pullbacks ) reveal high-resolution input- and target-specific features of remarkable perceptual alignment on a number of popular ImageNet-pretrained deep architectures. This strongly suggests that neural networks do, in fact, rely on learned interpretable patterns that can be recovered after training. Thus, our findings may have profound implications for knowledge discovery and the development of dependable artificial systems. Figure 1: A rudimentary 5-step pixel-space gradient ascent guided by excitation pullbacks for pretrained ResNet50. Each cell shows the difference between the perturbed and clean image, targeting the class in the column. Diagonal: original class; off-diagonal: counterfactuals. Last column: randomly selected extra label. See Section 5 for details. 1 Introduction This work investigates gradient-based explanations in ReLU networks, with a focus on convolutional architectures commonly used in computer vision. While our empirical results and theoretical developments concentrate on standard ReLU networks trained on image classification tasks, the core arguments are sufficiently general to apply - with the appropriate modifications - to other activation functions and modalities. We introduce a simple but effective method for interpreting neural predictions via excitation pullbacks , which isolate the robust informative signal in neural activations from the noise introduced by local data peculiarities. We motivate this idea extensively and demonstrate its empirical effectiveness on the following ImageNet-pretrained ReLU models: ResNet50 ( He et al. 2015 ) , VGG11_BN ( Simonyan and Zisserman 2015 ) , and DenseNet121 ( Huang et al. 2018 ) . These specific architectures were chosen for reasons detailed in Section 5 . We also give compelling theoretical arguments backing the claim that excitation pullbacks are indeed faithful local approximations of the network’s decision boundary. In particular, we hypothesize that they reveal a simpler linear model approximated by the trained network, encoded in the pre-activation patterns. The paper outline is as follows: • Section 2 introduces necessary notions and notational conventions; in particular, the gating function and gating-induced pullbacks are described; • Section 3 defines tensor field pullbacks and shows that neural networks are linear in their path space; • Section 4 defines excitation pullbacks and postulates that they approximate the decision boundary of an implicit kernel machine that is learned in the network’s pre-activation patterns; • Section 5 shows that excitation pullbacks of selected architectures pretrained on the ImageNet are perceptually aligned and can be used to produce faithful and meaningful local feature accentuations by just a few steps of gradient ascent along their directions; • Section 6 describes the related works, indicates a possible biological feasibility of presented methods and discusses their limitations. This paper thus bridges practical gradient visualization with theoretical insights into neural function, suggesting a path toward more transparent and robust deep models. 2 Preliminaries and notational conventions This section establishes important notions and conventions used throughout the rest of the paper. 2.1 ReLU networks Let M be a feedforward ReLU network with parameters \theta\in\mathbb{R}^{d_{net}} , expressible as a composition of affine transformations and pointwise ReLU nonlinearities. 1 1 1 we omit max-pooling layers from the analysis to improve clarity, while noting that similar arguments are likely to apply due to the structural similarity between max-pooling and ReLU; see Section 5.3 . Let L denote the number of layers, and let d_{\ell} be the width of the \ell -th layer, with d_{0} denoting the input dimension. Let x_{0} denote the identity function on \mathbb{R}^{d_{0}} , i.e. x_{0}(x):=x . The network applies the following operations: \displaystyle z_{\ell}=W_{\ell}x_{\ell-1}+b_{\ell},\quad x_{\ell}=\text{ReLU}(z_{\ell}),\qquad\text{for }\ell=1,\dotsc,L where W_{\ell}\in\mathbb{R}^{d_{\ell}\times d_{\ell-1}} , b_{\ell}\in\mathbb{R}^{d_{\ell}} , and \text{ReLU}(z)=\max(0,z) is applied element-wise. We define the overall network function M:\mathbb{R}^{d_{0}}\to\mathbb{R}^{d_{L}} by the recursive application of the layers, i.e., M(x_{0}):=x_{L} In this paper we’ll treat M as backbone for some linear model, i.e. x_{L} represents a penultimate layer in some chosen architecture. Specifically, we’ll consider f:\mathbb{R}^{d_{0}}\to\mathbb{R} defined by: f_{c}:=y_{c}^{\top}M where y_{c} is a trainable set of weights y_{c}\in\mathbb{R}^{d_{L}} (a neuron in the network’s head) and y_{c}^{\top} is it’s corresponding covector, i.e. the linear action of y_{c} on \mathbb{R}^{d_{L}} by the dot product: y_{c}^{\top}x=\langle y_{c},x\rangle . Standard classification network with C classes has C corresponding vectors y_{c} . We will skip the lower index c assuming that it’s fixed. For notational brevity, we consider augmented matrices and vectors where the bias term is incorporated by appending a 1 to the end of the input vector. Specifically, we define the augmented input and weight matrix as \tilde{x}_{\ell-1}=\begin{bmatrix}x_{\ell-1}\\ 1\end{bmatrix}\in\mathbb{R}^{d_{\ell-1}+1},\quad\tilde{W}_{\ell}=\begin{bmatrix}W_{\ell}&b_{\ell}\\ 0\quad\cdots\quad 0&1\end{bmatrix}\in\mathbb{R}^{(d_{\ell}+1)\times(d_{\ell-1}+1)} Then, the augmented pre-activation vector is given by \tilde{z}_{\ell}=\tilde{W}_{\ell}\tilde{x}_{\ell-1} Conventions Throughout the paper, we will often abuse notation by omitting the tilde and the explicit increase of dimension, writing simply z_{\ell}=W_{\ell}x_{\ell-1} with the understanding that the bias is included in the matrix W_{\ell} and the input vector x_{\ell-1} . We will occasionally write \mathbb{R}^{d^{*}_{0}}\subset\mathbb{R}^{d_{0}} to denote the true input space (without the implicitly added coordinate). Notice that, by construction, the \mathbb{R}^{d^{*}_{0}} remains the actual domain of f . 2.2 Gating representation of ReLU networks In ReLU networks, each activation can be equivalently viewed as a gating mechanism applied to the pre-activations. Specifically, for every layer \ell\in\{1,\dots,L\} , the activation function x_{\ell}=\text{ReLU}(z_{\ell}) can be rewritten as x_{\ell}=g_{\ell}\odot z_{\ell} where g_{\ell}\in\{0,1\}^{d_{\ell}} is a binary gating vector defined by g_{\ell}[i]=\begin{cases}1&\text{if }z_{\ell}[i]>0,\\ 0&\text{otherwise}.\end{cases} Since z_{\ell} is itself a function of the input x_{0} , we may define a gating function G:\mathbb{R}^{d_{0}}\to\prod_{\ell=1}^{L}\{0,1\}^{d_{\ell}},\quad G(x):=(g_{1},\dots,g_{L}) (1) Let G_{\ell}(x)=\mathrm{diag}(g_{\ell})\in\mathbb{R}^{d_{\ell}\times d_{\ell}} denote the diagonal matrix whose diagonal entries are given by g_{\ell} . Then, the network output M(x)=x_{L} can be written as a product of (augmented) weight matrices interleaved with input-dependent diagonal matrices: M(x)=G_{L}(x)W_{L}G_{L-1}(x)W_{L-1}\cdots G_{1}(x)W_{1}x where each W_{\ell} includes the bias terms via the convention introduced earlier. This representation makes the piecewise-linear structure of ReLU networks explicit: for a fixed input x , the gating pattern (g_{1},\dots,g_{L})\in\prod_{\ell=1}^{L}\mathbb{R}^{d_{\ell}} determines a purely linear computation. Similarly, let’s define the pre-activation function as: Z:\mathbb{R}^{d_{0}}\to\prod_{\ell=1}^{L}\mathbb{R}^{d_{\ell}},\quad Z(x):=(z_{1},\dots,z_{L}) (2) i.e. the concatenation of all pre-activations of M(x) . 2.3 Gating-induced networks More generally, given any gating function: \Lambda:\mathbb{R}^{d_{0}}\to\prod_{\ell=1}^{L}[0,1]^{d_{\ell}},\quad\Lambda:=(\lambda_{1},\dots,\lambda_{L}),\quad\lambda_{\ell}:\mathbb{R}^{d_{0}}\to[0,1]^{d_{\ell}} (3) we can define the \Lambda -induced network as M_{\Lambda}(x):=\Lambda_{L}(x)W_{L}\Lambda_{L-1}(x)W_{L-1}\cdots\Lambda_{1}(x)W_{1}x (4) where \Lambda_{\ell}(x)=\mathrm{diag}(\lambda_{\ell}(x))\in\mathbb{R}^{d_{\ell}\times d_{\ell}} is the diagonal matrix formed from the \ell -th component \lambda_{\ell} of \Lambda(x) . In particular, M is G -induced, i.e. M=M_{G} . Note that M_{\Lambda} implicitly depends on the network parameters \theta . We can also define the matrix field \vec{M}_{\Lambda}:\mathbb{R}^{d_{0}}\to\mathbb{R}^{d_{L}}\times\mathbb{R}^{d_{0}} by: \vec{M}_{\Lambda}(x):=\Lambda_{L}(x)W_{L}\Lambda_{L-1}(x)W_{L-1}\cdots\Lambda_{1}(x)W_{1} (5) so that M_{\Lambda}(x)=\vec{M}_{\Lambda}(x)x . We set f_{\Lambda}:=y^{\top}M_{\Lambda} and therefore f=f_{G} . 2.4 Pullbacks and vector fields Note: the symbols in this subsection 2.4 are unrelated to the rest of the section 2 . Let f:\mathbb{R}^{n}\to\mathbb{R} be a scalar-valued function, and let \phi:\mathbb{R}^{d_{0}}\to\mathbb{R}^{n} be a map. The pullback of f through \phi , denoted \phi^{*}f , is defined as the composition \phi^{*}f:=f\circ\phi which is a function \phi^{*}f:\mathbb{R}^{d_{0}}\to\mathbb{R} given explicitly by (\phi^{*}f)(x)=f(\phi(x)) This construction allows us to view f as a scalar function defined directly on the domain \mathbb{R}^{d_{0}} (through the action of \phi ). Convention Whenever a piecewise continuous vector