Borrowed Geometry: Cross-Distribution Head-Importance Fingerprints of Frozen Pretrained Gemma 4 31B
Abay Bektursun · 2026
Read Borrowed Geometry: Cross-Distribution Head-Importance Fingerprints of Frozen Pretrained Gemma 4 31B by Abay Bektursun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00333.txt.
Introduction
Abay Bektursun probes whether a frozen text-trained model carries reusable structure into tasks it was never trained on. Working with an unmodified Gemma model, Bektursun studies a slice of its attention heads with two signals. One is an English text probe of copying behavior. The other is the effect of switching each head off on four pattern tasks that use no language.
Four heads ranked at the top on both signals, a coincidence the author shows is unlikely by chance. Switching the key head off in a trained game agent cut success sharply, far more than a matched control head. The paper also reports honest negatives: the four heads alone do not suffice, and some tasks recruit other layers. The claim is a slice-level fingerprint of shared importance, plus causal evidence on one target task.
Catalog record
| CREATOR | Abay Bektursun |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.00333
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Borrowed Geometry: Cross-Distribution Head-Importance Fingerprints of Frozen Pretrained Gemma 4 31B Abay Bektursun Abstract Frozen Gemma 4 31B weights pretrained exclusively on text, unmodified, transfer through a thin trainable interface to non-text modalities the substrate has never processed. On the L24–L29 slice (192 attention heads), an English-text TxtCopy attention probe (95 sentences) and per-head ablation impact on four non-language token-pattern tasks (binary copy, associative recall, 1D cellular automaton Rule 90, binary addition) jointly classify four heads — L26.28, L27.28, L27.2, L27.3 — as top-tier on both signals. The slice-level joint coincidence is significant under hypergeometric null ( P = 0.0013 P=0.0013 , N = 192 N{=}192 , K = 38 K{=}38 , n = 4 n{=}4 ) and survives permutation tests over probe primitive and threshold ( P V4 = 0.013 P_{\text{V4}}=0.013 ). Pretrained Gemma L26 reaches 60.22% success on OGBench cube-double-play-task1 vs ∼ 1 % \sim 1\% for random-init Gemma ( + 59 +59 pt at n = 3 n{=}3 , robust under a FrozenRandom-GPT2 control with correct 1 / d k 1/\sqrt{d_{k}} scaling that also fails). Head-level causal validation: zeroing L26.28 in the trained cube-task1 IQL agent drops success 63.3 % → 10.0 % 63.3\%\to 10.0\% vs 46.7 % 46.7\% for a layer-matched low-TxtCopy negative control ( 3.2 × 3.2\times specificity ratio at n = 30 n=30 ; replicated across n = 5 n=5 cube seeds, paired- t t p = 0.039 p=0.039 , mean ratio 1.82 × 1.82\times ). A full L26 head sweep places L26.28 at rank 4 of 32, with rank invariant after regressing out output-projection norm. Honest negatives: within-L26 Spearman gives ρ ( TxtCopy, drop ) = + 0.37 \rho(\text{TxtCopy, drop})=+0.37 ( opposite of a within-layer causal reading); single-head activation patching does not transfer the matching variable (E1); the 4 named heads alone do not suffice on any task (E2); Walker2d-DT and scene-task1 recruit L24 outside the named slice and show null head-ablation specificity. We frame the contribution as a cross-distribution importance fingerprint at the slice level plus causal head-level evidence on one cross-modality target, in the indirect-evidence and model-organism modes of Olsson et al. 2022 ; Wang et al. 2022 . Code, configs, raw metrics, and per-seed checkpoints at the project repository. 1. Introduction Pretraining a frontier text language model now costs hundreds of millions of dollars. The artifact is a set of weight matrices encoding attention patterns, feature decompositions, representational structure over tokens — and an industry that uses those weights for one downstream purpose: generating text, or fine-tuning the model for text-adjacent tasks. We ask whether the weights, frozen and unmodified, are a general computational substrate that can be borrowed across modality boundaries the substrate has never seen during pretraining. All transfer experiments use frozen Gemma 4 31B ( Gemma Team 2026 ) weights pretrained exclusively on text tokens (web text, code, math, multilingual, chat); robotic state-action sequences, joint kinematics, manipulation rewards, 2D bit-grid patterns, and continuous-control feedback do not appear in pretraining at any point. Why expect borrowed geometry to work at all? The hypothesis is older than transformers. Lin et al. 2017 argue that deep networks approximate natural-world functions cheaply because the laws of physics are low-order, symmetric, and hierarchical, and those three properties are exactly what deep hierarchical feature extraction is structured to exploit. If physics-level compositionality explains why deep learning works at all, then the computational primitives pretraining discovers should inherit that compositional structure and be reusable wherever it applies. We borrow exaptation from evolutionary biology ( Gould and Vrba 1982 ) as the label: structures selected for one function repurposed for another without redesign — feathers exapted from thermoregulation to flight; pretrained attention heads exapted from text-copying to whatever else token-copying solves. Our setup — frozen substrate plus thin trainable interface — structurally generalizes reservoir computing ( Jaeger 2001 ; Maass et al. 2002 ) : computation rides on a fixed high-dimensional substrate, only a small readout is learned. The most recent framing is the Platonic Representation Hypothesis ( Huh et al. 2024 ) : representations align across architectures and modalities under multitask scaling, capacity, and simplicity bias. The supervised proxies extend the Neural Turing Machine. The toy-experiment origin substituted a frozen Gemma slice for the LSTM controller in NTM ( Graves et al. 2014 ) : the original NTM tasks (binary copy, associative recall) became the first supervised stress-tests; 1D CA Rule 90 and binary addition extend the same shape (sequence-in, sequence-out, well-defined per-bit error). The dual-measurement protocol of §4 names heads on those four tasks because per-bit-error metrics support clean single-token-effect ablations; boundary tasks (Dyck-2, GoL, reservoir, BC ceilings; §7) fail in characterizable ways. A concrete sharp fact anchors what borrowing means here. Score each of the 192 attention heads in Gemma 4 31B’s L24–L29 slice on how strongly it copies tokens of matching lemma in 95 English sentences (one number per head). Separately, wrap the same frozen slice in a 113K-parameter linear interface, train it to copy bit strings, and zero each head’s output projection one at a time (one number per head). The two measurements share nothing: different input distribution, different objective, different signal. Head L26.28 wins both: it scores 3.7\times the slice mean for English token-copying (rank 4 of 192) and ranks #2 most-critical for binary copy ablation ( \Delta L30 =+0.221 ). Three further heads (L27.28, L27.2, L27.3) classify by the same dual measurement. Under uniform null, the joint coincidence has P=0.0013 (hypergeometric); under a multiplicity-aware permutation test over probe primitive and threshold, P_{\text{V4}}=0.013 (App C.3). On the same frozen weights at modality scale. OGBench cube-double-play-task1: pretrained Gemma L26 reaches 60.22% across n=3 seeds; random-init Gemma at matched depth and shape fails entirely ( \sim 1\% ) — a +59 pt substrate-isolation gap (§3.3). Head-level: zeroing the named L26.28 in the trained cube-task1 IQL agent collapses success to 10.0% at n=30 , versus 46.7\% for a layer-matched low-TxtCopy negative control — 3.2\times specificity, replicated across n=5 cube seeds (paired- t p=0.039 , mean ratio 1.82\times ; §5). A full L26 head sweep places L26.28 at rank 4 of 32 (top decile, not unique); within-L26 Spearman shows TxtCopy in the opposite direction of a within-layer causal reading. The protocol identifies a slice-level enrichment pattern with population-level cross-modality head importance, not an individual-head circuit identification (three-claim distinction, §6.4). Contributions. (i) A dual-measurement coincidence test: head h is named only if both an English text-attention probe (TxtCopy) and an independent non-language task ablation place it in the top tier on the L24–L29 slice. (ii) Application to Gemma 4 31B: four named heads, hypergeometric P=0.0013 , permutation P_{\text{V4}}=0.013 ; causal validation on OGBench cube-task1 ( 3.2\times specificity, n=5 paired- t ); honest negatives on supervised activation patching, head-subset sufficiency, and within-layer regression (§6). (iii) Boundary mapping: where the protocol fails (Dyck-2 plateau, 2D GoL, continuous regression, behavior-cloning ceilings, §7) and where the broader-manifold facet appears (Walker2d L24 outside the named slice, App F). Prior work in brief (full §8). Frozen Pretrained Transformers ( Lu et al. 2022 ) proposed frozen GPT-2 matches fine-tuning; Naik and Gupta 2021 narrowed the strong claim via LR sweeps. Our FrozenRandom-GPT2 control (§3.2) and matched-capacity Trained-Transformer (App B.5) close two confounds at frontier-Pareto substrate scale. Mechanistic interpretability (