Block-Based Double Decoders
Vanessa Alexander, Benjamin Bradley, Chaitanya Harsha, Asher Labovich, Department of Computer Science · 2026
Read Block-Based Double Decoders by Vanessa Alexander, Benjamin Bradley, Chaitanya Harsha, Asher Labovich, Department of Computer Science (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18807.txt.
Introduction
Asher Labovich, Benjamin Bradley, Vanessa Alexander, and Chaitanya Harsha design block-based double decoders, a transformer build that mixes two training styles. Their doubly causal block masks allow full loss supervision and fixed sequence packing. In scaling tests they beat encoder-decoders and nearly match decoder-only models, while shrinking stored key-value data and work per token by at least two thirds at inference. The design keeps prefill caching and other decoder-only speed tricks.
Catalog record
| CREATOR | Vanessa Alexander, Benjamin Bradley, Chaitanya Harsha, Asher Labovich, Department of Computer Science |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31249 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18807
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Block-Based Double Decoders Vanessa Alexander, Benjamin Bradley, Chaitanya Harsha, Asher Labovich, Department of Computer Science Abstract Encoder-decoder models offer substantial inference-time savings over decoder-only models, but their pretraining objectives suffer from sparse supervision and dynamic sequence lengths, keeping them out of practice at scale. We propose block-based double decoders , a novel transformer architecture that utilizes doubly-causal block-based attention masks to train with full loss supervision and static sequence packing, combining decoder-only training efficiency with encoder-decoder inference efficiency. In scaling law experiments, block-based double decoders strongly outperform encoder-decoders and closely track decoder-only models across scales. At inference time, they cut KV-cache memory and per-token compute by at least 2 3 \frac{2}{3} without sacrificing prefill caching or other existing inference optimizations available to decoder-only models. 1 Introduction In recent years, the rise of the transformer [ 16 ] has led to advances in natural language modeling. Although the original transformer employed an encoder-decoder based architecture, with full attention for understanding in the encoder and causal attention for prediction in the decoder, decoder-only architectures have gained prominence for their scalability in text-generation settings [ 11 ] . Recently, however, there has been renewed attention in encoder-decoder architectures [ 3 ] for their gains in efficiency and their effectiveness in compute- and storage-constrained environments, as they achieve substantial reductions in KV-caching. Even with this work, it has been shown [ 12 ] that the architectural boundary between the encoder and decoder leads to inefficiencies, and that a 2P parameter encoder-decoder model incurs the same computational cost as a P parameter decoder-only architecture. An alternative single-transformer approach, PrefixLM, in which full attention is used for a prefix of the inputs, and causal attention after this, attempts to combine the bidirectional context of encoder-decoders with the sharing of parameters in decoder-only models. This method benefits from less dynamic batching, but leaves many tokens untrained on. But PrefixLM is outperformed by an encoder-decoder trained with span corruption, where the model is trained to predict a proportion of “corrupted” tokens. Although this achieves superior performance, it requires highly dynamic batching, and still leaves many tokens untrained on, as [ 12 ] found that a mere 15% corruption results in the best performance. We propose a novel attention mask for pre-training 1 1 1 Code found at https://github.com/ashlab11/block-based-double-decoder , which we call doubly-causal block-based masking . Our proposed architecture consists of two decoders. The first sees a standard causal mask and takes in the standard input. The second decoder conducts cross-attention on the output of the first decoder, but also on the input, masked with our approach. We split the input into “blocks,” with full self-attention within each block, and causal cross-attention between blocks. In this way, we receive loss signal from every token in the input, and we achieve constant token length. Figure 1: Graph comparing loss versus tokens for different size decoder, double decoder, and encoder-decoder models 2 Prior Work The original Transformer architecture [ 16 ] consists of a bidirectional self-attention encoder paired with a causal decoder for autoregressive generation. The encoder processes the full input sequence to produce contextualized representations for each token. In contrast, the decoder applies masked (causal) self-attention and additionally incorporates cross-attention, allowing each decoding position to attend to all of the encoded input representations. However, in recent years, it has been found that for generative modeling, decoder-only models are more scalable and achieve superior performance [ 11 , 1 ] . Meanwhile, encoder-only architectures [ 2 ] , trained using token masking, are performant when fine-tuned on downstream classification-type tasks. However, [ 12 ] demonstrated that any classification task can fundamentally be described as a text-to-text problem, putting it in reach of generative models. Their findings included a thorough search of several architectures and pre-training objectives, including decoder-only with the standard LM objective, decoder-only with the PrefixLM objective, and encoder-decoder with span corruption, among others. They found that across all the downstream tasks these models were tasked with, including question answering, summarizing, and translation, encoder-decoder with span corruption performed the best across a range of prediction tasks. In span corruption, contiguous spans of tokens are "corrupted" or replaced with sentinel tokens that are assigned a unique ID specific to that sentence. The target sequence consists of the concatenated missing spans, and the model is then trained to autoregressively generate those missing spans. Typically, only a fraction of the tokens (about 15%) are corrupted with major performance degradation occurring at 50% corruption. Empirically, this architecture combined with span corruption has been shown to outperform both decoder-only language models (LM) and prefix variants under comparable training settings. Additionally, span corruption as a pretraining objective has been shown to empirically outperform standard LM pretraining objectives under comparable training settings within the same model architectures. We note that, despite its high performance, that there are several issues with span corruption as a pre-training objective. First, [ 12 ] studied what rate of span corruption would be most performant, and confirmed [ 2 ] ’s findings that 15% span corruption led to optimal training. However, this means that only these 15% of tokens yield a loss signal, so most tokens are not being trained on. This creates a fundamental tradeoff between increasing prediction difficulty to enhance long-range reasoning and maximizing supervision density relative to the total computational cost. Additionally, span corruption requires dynamic batching, as for a given batch, the number of tokens predicted will not be constant across all training examples. This requires token padding, a waste of compute and memory. We hypothesize that an alternate pre-training objective can perform as well as span corruption, but without these inefficiencies. Additionally, prior work such as [ 18 ] demonstrated that attention patterns that capture both local dependencies and global information can achieve strong performance at reduced computational cost. Although we strive for train-time efficiency over O(n) attention, we take inspiration from this notion with an architecture that captures both local and global interdependencies. 3 Architecture Comparison 3.1 Block-based double decoders BOS BOS A A B B C C D D E E F F keys \rightarrow queries \rightarrow self-attention cross-attention Figure 2: Visual explanation of the decoder attention mask for an example sentence. Splitting up into three parts, we see that the decoder sees three context-response pairs in parallel: (1): empty context, [BOS, A] response (2): [BOS, A] context, [B, C, D] response (3): [BOS, A, B, C, D] context, [E, F] response Each token thus appears in the loss exactly once per forward pass. In this paper, we wish to create an architecture for next-token prediction that retains the benefits of encoder-decoder architectures – in particular, substantial KV-cache reductions and ease of use on edge device – without sacrificing the training efficiency of decoder-only transformers. We consider two criteria when creating such an architecture. First, loss-information density: every token in a packed training example should contribute a loss signal in every forward pass. Second, sequence length statis: post-packing sequence length should not change across training batches, so that throughput is predictable and maximally efficient. To motivate our architecture, we first consider a basic PrefixLM encoder-decoder training objective, which fails both criteria. To train one, we must randomly choose split-points at each batch, placing N\%