Toy Combinatorial Interpretability Models Reveal Lottery Tickets in Early Feature Space
Alon Bebchuk, Nir Shavit · 2026
Read Toy Combinatorial Interpretability Models Reveal Lottery Tickets in Early Feature Space by Alon Bebchuk, Nir Shavit (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17704.txt.
Introduction
Alon Bebchuk and Nir Shavit probe the lottery ticket idea. It says dense neural networks hide small subnetworks that can match the full model when retrained alone. In a small, clause-based toy setting where features can be tracked clearly, winning tickets match starting spots already close to the final feature codes. Sparse retraining preserves a family of compatible codes rather than exact rows. Simple distance probes often beat standard weight-based ticket hunting.
Catalog record
| CREATOR | Alon Bebchuk, Nir Shavit |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82354 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.17704
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Toy Combinatorial Interpretability Models Reveal Lottery Tickets in Early Feature Space Alon Bebchuk, Nir Shavit Abstract The lottery ticket hypothesis posits that dense networks contain sparse subnetworks, “winning tickets,” that, when rewound to their initial weights and retrained in isolation, match the performance of the full model. We ask a more mechanistic question: what internal object does a winning ticket preserve? We work in a combinatorial, clause-structured toy setting that admits an interpretable feature-space representation with well-defined combinatorial distances between features. We show that winning tickets in weight space correspond to precursor locations in feature space that are already near, at initialization, to the final feature-channel codes. Dense SGD resolves these locations through structured selection: proximal locations either converge to final codes or are rejected, with rejection concentrated at more crowded neurons, implicating competition under superposition. A winning ticket is thus a family of compatible code locations that jointly balance proximity to final codes with low inter-feature interference. Sparse retraining often re-expresses the same clause/template family on a different row, so the preserved object is family-level rather than microscopic row identity. We validate this account with lightweight probes based on feature-space distance and motion; in our setting, these probes frequently outperform established weight-based ticket discovery methods in both accuracy and exact code recovery. Although these findings are grounded in a toy setting, they suggest that the lottery ticket structure is governed by hidden feature-space geometry rather than weight-space subnetwork identity. 1 Introduction The lottery ticket hypothesis (LTH) is often stated in functional language: prune a trained dense network, rewind the surviving weights to their original initialization, retrain the sparse subnetwork, and the sparse model can often approach dense accuracy ( 8 ) . Later work sharpened that picture by showing that rewinding to slightly later dense checkpoints often helps and by studying the early stability of winning tickets under optimization, meaning that sparse subnetworks found after early training can remain trainable and connected to good dense solutions ( 19 ; 10 ; 26 ) . But those results leave open a more mechanistic question: what object is the ticket preserving? One answer is that the ticket preserves a lucky set of weights. Another is that it preserves a mask, or a mask together with a favorable optimization basin. We study a different possibility: in the models considered here, a ticket preserves enough of an internal feature-space scaffold for sparse training to rebuild the relevant computation. The mask is visible in weight space, because that is where pruning is applied, but the computation itself is easier to read in this feature-space view. Existing work strongly suggests that masks encode more than a set of large coordinates ( 15 ) . In ordinary deep networks, however, the relevant hidden representation is usually too opaque to say exactly what object the mask is preserving. We therefore use a clause-structured toy setting in which the internal feature space can be read directly. The main story of the paper is that dense training makes certain locations in the initial feature space identifiable as useful ticket precursors. In our setting, SGD training of a DNF detector first constructs feature-channel codes: clause-local sign patterns that represent features using a small combinatorial alphabet, while multiple candidate features may share the same rows or embedded coordinates ( 1 ) . A key tool at our disposal is the ability to define combinatorial distances between codes. Some candidate locations begin close in code distance to the relevant collection of locations and are amplified into exact codes during training; others that are also plausible early are rejected. The rejection is structured: it occurs disproportionately on crowded rows, indicating that dense SGD is resolving competition among candidate carriers under superposition, in the sense of multiple features sharing limited representational dimensions ( 7 ) . Thus, ticket formation is not simply local code formation and not simply magnitude growth. It is a process of selecting a compatible family of feature-channel codes. Pruning changes this landscape. A mask is a sparse subset of coordinates in weight space, but applying that mask and rewinding does not merely remove a few displayed code locations. Because each clause-local entry is a sum over many embedded coordinates, zeroing the coordinates outside the mask changes the whole row pattern in feature space. The sparse initialization is therefore a new projected feature-space state. A winning ticket is a family of locations in this feature space that leaves enough degrees of freedom for sparse SGD to reconstruct compatible feature-channel codes. We propose a set of constructive rules for finding tickets. These rules are diagnostic: they ask whether the properties exposed by dense SGD—distance to codes, margin, early motion, and compatibility among row carriers—are sufficient to recover part of the same support. The benchmark sections make this point sharper: feature-space rules are best at finding the emerging code structure early, while later weight magnitude becomes strong precisely because dense training has already projected much of that structure into the weights’ magnitude ordering. The remaining gap is partly the translation problem: a promising row–clause code location must still be implemented by a sparse set of coordinates in weight space. In this clause-structured setting, we make the following claims: 1. A lottery ticket is a set of feature-space locations that are close to the eventual feature-channel codes for the relevant clauses. The corresponding object in weight space is a mask and its initial values, not just a mask. 2. Dense SGD exposes tickets in feature space by driving selected locations toward canonical codes while rejecting other plausible locations on heavily loaded rows. 3. Pruning changes the feature-space representation. The rewound sparse state is not a visual restriction of the dense initial state; removing coordinates changes the whole row pattern. 4. Mechanism-inspired detectors based on feature-space distance and motion recover a large part of the same structure and outperform raw weight-space signals for early code prediction. Their success supports distance and motion as crucial ticket elements, while their limitations reveal a separate site-to-weights translation problem. Limitations We are reminded of a story about Judah Folkman, the father of anti-angiogenesis therapy for cancer, a technique that today serves millions and generates billions in revenue. Some 30 years ago, after discovering the first two angiogenesis-inhibiting drugs that cured cancer in laboratory mice, a reporter asked what this astonishing result meant for cancer research. Folkman said: “Well, it says that if you are a mouse and you have cancer, we can probably cure you.” Our claim here is in that spirit: if you are a small neural network computing Boolean formulae, then we can probably understand your lottery tickets. Whether, when, and how these ideas extend to larger models and less discrete tasks is a research program. The paper should be read as a model-organism study that makes feature-space measurement possible and suggests a new path in interpretability research, not as a completed theory of lottery tickets in general. 2 Related Work The original lottery ticket hypothesis showed that dense networks can contain sparse subnetworks whose original initialization allows them to train to dense-like performance when isolated and retrained ( 8 ) . Subsequent work established rewinding and fine-tuning as practical routes to ticket recovery and emphasized that