Mask-Based Priors Are More Persistent than Query-Key Initializations

Mingze Ma, Hemanth Saratchandran, Cameron Gordon, Simon Lucey · 2026

Read Mask-Based Priors Are More Persistent than Query-Key Initializations by Mingze Ma, Hemanth Saratchandran, Cameron Gordon, Simon Lucey (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2608-00418.txt.

Introduction

Ma and co-authors test how to give Transformers a lasting structural prior. On some Boolean tasks, Transformers extrapolate in a systematically wrong way. They fit seen data but follow a simpler rule on unseen cases. Past structured starts set query and key projections to mimic a target attention pattern. Those priors wash out fast in training, they find.

Their fix initializes the attention mask itself. The mask is a finite, learnable logit bias built from task-level interaction structure. It sits apart from content-based attention scores, so it can persist through training. On Boolean reasoning tasks, mask starts reach near-perfect extrapolation. Vanilla and QK-started models stay trapped by the default bias. Low-data arithmetic also improves. Results stay competitive on vision and language benchmarks.

Catalog record

CREATORMingze Ma, Hemanth Saratchandran, Cameron Gordon, Simon Lucey
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2608.00418

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mask-Based Priors Are More Persistent than Query-Key Initializations Mingze Ma, Hemanth Saratchandran, Cameron Gordon, Simon Lucey Abstract Transformers do not merely lack data on some Boolean extrapolation tasks; they generalize in a systematically wrong way. Recent work on generalization on the unseen has shown that, despite fitting the observed domain, Transformers often extrapolate according to a simpler minimum-degree interpolator rather than the true target function. These Boolean tasks are not practical applications, but controlled stress tests for understanding Transformer inductive bias. We ask whether this failure mode can be corrected by injecting explicit structural priors into attention. Existing structured-initialization methods alter Transformer inductive bias indirectly, by choosing query and key projections whose similarity scores approximate a desired attention pattern. However, we find that when applied to Boolean extrapolation, these QK-based priors can be rapidly overwritten during training and fail to change the learned extrapolation rule. We propose a simpler alternative: initialize the additive attention mask directly. Unlike standard hard masks used for causality or locality attention, our mask is a finite, learnable attention-logit bias initialized from task-level interaction structure. This separates the structural prior from content-dependent attention scores, allowing it to persist throughout optimization. On Boolean reasoning tasks, mask-based initialization achieves near-perfect extrapolation where vanilla and QK-initialized Transformers remain trapped by the default inductive bias. The same mechanism also improves low-data arithmetic performance and remains competitive on vision and language benchmarks. These results show that attention masks can serve not only as architectural constraints, but as a simple substrate for encoding persistent inductive bias in Transformers. 1 Introduction Attention provides Transformers ( Vaswani et al., 2017 ) a powerful mechanism for learning long range token interactions directly from data. This flexibility has been central to their success across natural language processing ( Vaswani et al., 2017 ; Radford et al., 2019 ; Guo et al., 2025 ) and computer vision ( Dosovitskiy et al., 2020 ; Liu et al., 2021 ; Peebles and Xie, 2023 ) . However, this same flexibility can also make their inductive bias difficult to control. Recent work on generalization on the unseen (GOTU) ( Abbe et al., 2024 ) provides a sharp example: Transformers can fit the observed data while extrapolating according to the wrong rule. Boolean tasks make this failure especially clear. A model may match every training example yet prefer a simpler minimum-degree interpolator over the true target function on unseen inputs ( Abbe et al., 2024 ) . This raises a broader question: Can we modify self-attention so that its inductive bias better matches the structure of the target problem? A natural approach is to initialize self-attention with a structural prior. Rather than requiring the model to infer the relevant relational pattern entirely from data, structured initialization biases the initial attention mechanism toward interactions believed to be important for the task. Classical initialization methods mainly focus on optimization stability, often by analyzing signal propagation under assumptions on pseudo-inputs such as Gaussian-distributed features ( Sitzmann et al., 2020 ; Lindell et al., 2022 ; Kumar, 2017 ; He et al., 2015 ) . While effective for stabilizing deep networks, these methods are not designed to initialize the correlation structure learned by self-attention, and typically do not encode explicit relational priors. Figure 1 : Where the prior lives determines whether it changes extrapolation. Boolean GOTU tasks expose a failure mode in which Transformers fit the observed domain but extrapolate according to the wrong rule, often a simpler minimum-degree interpolator. We compare two ways of injecting the same structural prior into attention. QK-based initialization encodes the prior indirectly through content-dependent query-key projections, choosing Q and K so that \mathrm{Softmax}(QK^{\top}/\sqrt{d}) approximates the desired pattern. Mask-based initialization instead places the prior directly in the attention logits as a finite, learnable additive bias. After training, QK-based priors can be washed out, whereas mask-based priors remain explicit and persistent, leading to substantially improved extrapolation. More recent works have begun to study initialization directly for attention, either to improve training stability ( Ji et al., 2025 ) or to modify the model’s inductive bias ( Zheng et al., 2025 ) . These methods usually encode a desired attention pattern through the query and key projections, using either prior knowledge about the dataset ( Zheng et al., 2025 ) or posterior observations from well-trained attention maps ( Trockman and Kolter, 2023 ) . They have demonstrated improvements in settings where vanilla Transformers previously struggled. However, QK-based structural initialization is indirect: the prior must be represented through content-dependent similarity scores. Since the same query and key projections must also learn task-dependent representations during training, the injected structure may be rapidly repurposed or washed out. As a result, it is unclear whether QK initialization genuinely changes the learned extrapolation rule, particularly in settings such as GOTU or arithmetic problems where the failure mode is explicitly about inductive bias ( Abbe et al., 2024 ; Duan et al., 2024 ) . In this work, we first study Boolean GOTU problems ( Abbe et al., 2024 ) , where the extrapolation failure is explicit and the relevant variable interactions are known. This makes them a useful lens for testing whether an intervention genuinely changes the inductive bias of a Transformer, rather than merely improving optimization on the observed domain. We show that these logical structures can be naturally encoded as attention-mask patterns, guided by the principle that self-attention should expose the token interactions required by the target rule. Our main finding is that where the prior is placed matters. State-of-the-art QK-based structured initialization methods can produce the desired attention pattern at initialization ( Trockman and Kolter, 2023 ; Zheng et al., 2025 ) , but do not necessarily preserve this information during training and do not reliably alter the extrapolation behavior of the vanilla Transformer. In contrast, injecting the same structural patterns directly as learnable additive attention masks yields near-perfect extrapolation on Boolean GOTU tasks. This intervention does not introduce a new attention operation: it uses the standard additive attention-mask interface already present in Transformers. However, unlike causal or padding masks, which impose hard architectural constraints, our masks are finite, learnable attention-logit biases initialized from task-level interaction structure. Thus, the mask acts as a persistent inductive bias rather than merely a validity constraint. Beyond Boolean reasoning, we evaluate the same strategy on arithmetic, image classification, and small language-modeling benchmarks. Arithmetic tasks provide a natural extension from Boolean logic to compositional algorithmic structure, while vision and language tasks test whether mask-based priors remain compatible with settings where Transformers already perform well. Across these settings, mask-based initialization preserves prior information throughout optimization, improves low-data generalization on structured tasks, and remains competitive on standard vision and language benchmarks. These results suggest that additive attention masks can serve not only as architectural constraints, but also as a simple substrate for persistent inductive bias in Transformer attention. In summary, our main contributions are: • We use Boolean GOTU problems as a sharp test of Transformer extrapolation behavior, and show that appropriately designed attention-mask priors can move Transformers away from minimum-degree solutions. • We show that existing QK-based structured initialization methods may fail to preserve injected priors during training and may not reliably change the extrapolation behavior of vanilla Transformers on

Read the complete text