Steering Sparse Autoencoder Latents to Control Dynamic Head Pruning in Vision Transformers (Student Abstract)
Yousung Lee, Dongsoo Har · 2026
Read Steering Sparse Autoencoder Latents to Control Dynamic Head Pruning in Vision Transformers (Student Abstract) by Yousung Lee, Dongsoo Har (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26743.txt.
Introduction
Lee and Har steer sparse autoencoder features to control dynamic head pruning in vision transformers. Pruning redundant attention heads saves compute, but past rules are hard to read or steer. An autoencoder trained on the final layer's residual stream splits dense codes into sparse, legible parts. Amplifying chosen parts shifts which heads get pruned for each class. For the bowl class, accuracy rises from 76% to 82% while head use drops from 0.72 to 0.33, driven by two heads.
Catalog record
| CREATOR | Yousung Lee, Dongsoo Har |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 11518 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.26743
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Steering Sparse Autoencoder Latents to Control Dynamic Head Pruning in Vision Transformers (Student Abstract) Yousung Lee, Dongsoo Har Abstract Dynamic head pruning in Vision Transformers (ViTs) improves efficiency by removing redundant attention heads, but existing pruning policies are often difficult to interpret and control. In this work, we propose a novel framework by integrating Sparse Autoencoders (SAEs) with dynamic pruning, leveraging their ability to disentangle dense embeddings into interpretable and controllable sparse latents. Specifically, we train an SAE on the final-layer residual embedding of the ViT and amplify the sparse latents with different strategies to alter pruning decisions. Among them, per-class steering reveals compact, class-specific head subsets that preserve accuracy. For example, bowl improves accuracy (76% → \rightarrow 82%) while reducing head usage (0.72 → \rightarrow 0.33) via heads h 2 h_{2} and h 5 h_{5} . These results show that sparse latent features enable class-specific control of dynamic pruning, effectively bridging pruning efficiency and mechanistic interpretability in ViTs. Introduction Vision Transformers (ViTs) leverage multi-head self-attention to capture diverse token interactions. However, many attention heads are redundant, increasing computation without proportional performance gain. To address this, adaptive frameworks such as AdaViT ( 6 ) have been introduced, which use auxiliary networks to select which heads to prune. This input-dependent pruning substantially reduces computation while preserving accuracy. However, a key limitation remains: since these pruning policies rely on residual embeddings, the decision process is often opaque and difficult to control at the latent level. As a result, while dynamic head pruning improves efficiency, its mechanism lacks interpretability. If pruning decisions could be explained or controlled at the latent level, head selection would become both interpretable and controllable. Sparse Autoencoders (SAEs) offer a natural tool for this purpose, as they disentangle dense, polysemantic embeddings into sparse latent features that tend to encode more monosemantic and interpretable concepts in transformer representations ( 2 ; 4 ) . Recent studies have shown that this disentanglement enables steering specific SAE latent dimensions to control model behavior in a desired direction ( 3 ; 1 ) . In this work, we propose a novel framework that integrates k -sparse autoencoders ( 5 ) with dynamic head pruning in ViTs to make pruning decisions controllable at the latent level. This framework also enhances interpretability by revealing class-specific head subsets. As illustrated in Figure 1 , we amplify selected SAE latent dimensions, reconstruct the steered embedding, and feed it into the decision network to observe how pruning behavior changes. Our experiments show that per-class latent steering is particularly effective, reducing head usage while largely maintaining accuracy. Overall, these results suggest that sparse latents provide an effective way to interpret and control dynamic pruning, bridging the gap between efficiency and mechanistic interpretability in ViTs. Method Figure 1: Comparison of dynamic head pruning before (left) and after (right) SAE latent steering, where x denotes the CLS token from the final-layer residual stream input, and \tilde{x} denotes the steered embedding. In this work, we adopt AdaViT as the baseline dynamic pruning framework, with a particular focus on head pruning. We first train a Vision Transformer with layer-wise decision networks. In this setup, each lightweight network receives the class ( CLS ) token from the residual stream input and outputs head importance logits a_{\ell}\in\mathbb{R}^{H} , where \ell denotes the layer index and H the number of attention heads. We use the CLS token as input for the decision network since it encodes global context relevant for classification. Binary masks M_{\ell,i}\in\{0,1\} are obtained via Gumbel-Sigmoid sampling from a_{\ell,i} , where i denotes the head index. The masked attention is computed as follows: h_{\ell,i}=M_{\ell,i}\,\mathrm{Attn}(Q,K,V)_{\ell,i}. (1) The Vision Transformer and the decision network are trained jointly to preserve accuracy while enforcing head sparsity toward a target head usage ratio. After training, we extract x\in\mathbb{R}^{d} , the CLS token from the final layer’s residual input and use it to train a k -sparse autoencoder. Formally, the encoder and decoder are given by \displaystyle z \displaystyle=\mathrm{TopK}\!\left(W_{\mathrm{enc}}(x-b_{\mathrm{dec}})\right),\quad W_{\mathrm{enc}}\in\mathbb{R}^{n\times d}, (2) \displaystyle\hat{x} \displaystyle=W_{\mathrm{dec}}z+b_{\mathrm{dec}},\quad W_{\mathrm{dec}}\in\mathbb{R}^{d\times n}, (3) where z is the sparse latent representation of x , \hat{x} is its reconstruction, and b_{\mathrm{dec}} denotes the decoder bias. The parameters n and d represent the SAE latent and input embedding dimensions, respectively. Sparsity is enforced through a top- k activation, which preserves only the k largest dimensions. The SAE is trained with the following mean squared error (MSE) reconstruction objective: \mathcal{L}_{\mathrm{rec}}=\|x-\hat{x}\|_{2}^{2}, (4) which encourages \hat{x} to remain close to x . After training the SAE, we amplify the selected latent dimensions S by z_{i}^{\prime}=\begin{cases}z_{i}+\alpha,&i\in S,\\[3.0pt] z_{i},&i\notin S,\end{cases}\quad\tilde{z}=\mathrm{TopK}(z^{\prime}), (5) where \alpha is the amplification strength and \tilde{z} denotes the steered sparse latent vector obtained after amplification and top- k activation. The steered embedding is then reconstructed as \tilde{x}=W_{\text{dec}}\tilde{z}+b_{\text{dec}} , which is finally fed into the decision network to obtain the steered pruning mask. Experiments Dynamic pruning baseline. We fine-tune an ImageNet-pretrained ViT-Small ( 12 layers, 6 heads per layer, 384 -d embeddings) on CIFAR- 100 , reaching 91.27\% accuracy. Each layer employs a single decision network for head selection, as described in the Method section. The final model prunes 30\% of heads while maintaining 89.79\% accuracy. Sparse autoencoder. The SAE expands the 384 -d residual embedding into the 3072 -d latent space ( 8\times ) with top- k activation ( k=64 ). It is trained for 100 epochs, achieving an MSE loss of 0.0228 . Replacing the original embedding with its reconstruction yields only minor differences in accuracy ( -0.12\% ) and head usage ratio ( +0.025 ), showing that the SAE preserves essential information for effective pruning. Steering dynamic pruning. Inspired by the top- k masking experiments in PatchSAE ( 4 ) , we adapt this idea to dynamic head pruning using the latent steering defined in Eq. (5), which amplifies selected SAE latent dimensions. For each sample, we evaluate three strategies for defining the index set S based on training-set activation frequency statistics: (1) Per-class frequent — top- k most frequently activated latents within each class, (2) Global frequent — top- k most frequently activated latents across all classes, and (3) Random . Figure 2 shows that per-class steering reduces head usage while largely preserving accuracy, whereas global and random strategies lead to larger accuracy drops as \alpha increases. The low overlap between global and per-class top- k frequent latent dimensions ( 0.1641 ) indicates that the SAE captures class-discriminative concepts. Figure 3 further illustrates head usage patterns in the final layer under per-class steering. For bowl , accuracy rises ( 76\%\!\to\!82\% ) while head usage falls ( 0.72\!\to\!0.33 ), mainly relying on h_{2} and h_{5} . pine_tree shows a similar pattern ( 79\%\!\to\!84\% , 0.93\!\to\!0.35 ), relying on h_{2} and h_{3} . Interestingly, semantically related classes such as bowl and plate share similar head subsets h_{2} and h_{5} , indicating that per-class steering reveals class-level semantic relationships among heads. These examples suggest that amplifying per-class top- k frequent activations enriches class-specific signals in the decision network input, leading to class-specific pruning behaviors. Overall, these results demonstrate that the Sparse Autoencoder provides an effective way to interpret and control dynamic head pruning in ViTs. Figure 2: Accuracy (%) and head usage ratio in the final layer under different strategies as