Sequence Repetition Enhances Token Embeddings and Improves Sequence Labeling with Decoder-only Language Models

Matija Luka Kukić, Marko Čuljak, David DukićMartin TutekJan ŠnajderTakeLab @ Faculty of Electrical Engineering and ComputingUniversity of Zagreb [email protected] · 2026

Read Sequence Repetition Enhances Token Embeddings and Improves Sequence Labeling with Decoder-only Language Models by Matija Luka Kukić, Marko Čuljak, David DukićMartin TutekJan ŠnajderTakeLab @ Faculty of Electrical Engineering and ComputingUniversity of Zagreb [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17585.txt.

Introduction

Matija Luka Kukić and co-authors fix a mismatch in sequence labeling with decoder-only models. Labeling tasks tag each token in a sequence. They gain from context on both sides of each token. Decoder models, though, read only the prefix. That one-way view has pushed labeling toward encoder models. The team uses sequence repetition to help.

Repeating the sequence boosts token embeddings in their tests. It gives the decoder a second view of each token. Better labeling serves search, health records, and language tools. If decoders can label well, one model family can do more jobs. The trick is simple, which aids uptake. The work tests it on labeling tasks.

Catalog record

CREATORMatija Luka Kukić, Marko Čuljak, David DukićMartin TutekJan ŠnajderTakeLab @ Faculty of Electrical Engineering and ComputingUniversity of Zagreb [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17585

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sequence Repetition Enhances Token Embeddings and Improves Sequence Labeling with Decoder-only Language Models Matija Luka Kukić, Marko Čuljak, David DukićMartin TutekJan ŠnajderTakeLab @ Faculty of Electrical Engineering and ComputingUniversity of Zagreb [email protected] Abstract Modern language models (LMs) are trained in an autoregressive manner, conditioned only on the prefix. In contrast, sequence labeling (SL) tasks assign labels to each individual input token, naturally benefiting from bidirectional context. This discrepancy has historically led SL to rely on inherently bidirectional encoder-only models. However, the rapid development of decoder-only models has raised the question of whether they can be adapted to SL. While causal mask removal has emerged as a viable technique for adapting decoder-only models to leverage the full context for SL, it requires considerable changes to the base model functionality. In this work, we explore sequence repetition (SR) as a less invasive alternative for enabling bidirectionality in decoder-only models. Through fine-tuning experiments, we show that SR inherently makes decoders bidirectional, improving the quality of token-level embeddings and surpassing encoders and unmasked decoders. Contrary to earlier claims, we find that increasing the number of repetitions does not degrade SL performance. Finally, we demonstrate that embeddings from intermediate layers are highly effective for SR, comparable to those from final layers, while being significantly more efficient to compute. Our findings underscore that SR alleviates the structural limitations of decoders, enabling more efficient and adaptable LMs and broadening their applicability to other token-level tasks. takelab/repetition-sl * * footnotetext: Equal contribution, order determined by ascending age $\dagger$ $\dagger$ footnotetext: Corresponding author 1 Introduction Figure 1: Attention weight matrix in a decoder block for an input sequence “ABC” repeated r times. Sequence repetition (SR) forms bidirectional attention block regions with no modifications to the naturally unidirectional architecture of decoder-only language models. The share of bidirectional blocks contributing to the attention outputs increases with subsequent repetitions, enabling the model with a higher capacity to process tokens to the right of the currently processed token. The NLP community has invested immense effort in training models for natural language understanding (NLU) tasks. Arguably, one of the more complex NLU tasks is sequence labeling (SL), where labels are assigned to individual tokens in a sequence. From named entity recognition and text chunking ( 41 ; 42 ) to more challenging tasks such as event extraction ( 6 ) and aspect-based sentiment analysis ( 33 ) , SL models rely on high-quality token embeddings. While encoder-only language models (LMs) such as BERT ( 5 ) and RoBERTa ( 29 ) have long been the standard for SL tasks, scaling has enabled decoder-only LMs to produce superior embeddings, surpassing their encoder-only counterparts on various token- and sequence-level tasks ( 2 ; 27 ) . This trend has motivated research on decoder-as-encoder adaptation, typically by removing the causal mask (CM) in combination with parameter-efficient fine-tuning ( 16 ; 8 ) . While this modification provides the model with a fully bidirectional context, it is a substantial architectural change ( 27 ) . A parallel line of work has shown that simply repeating the input sequence can improve sequence-level representations in decoders ( 48 ; 37 ; 7 ) . Sequence repetition (SR) offers a less invasive method for incorporating bidirectionality without modifying the decoder’s core architecture, as illustrated in Figure 1 . By repeating the input sequence several times, the decoder blocks form bidirectional regions inside attention matrices. However, despite its promise for sequence-level tasks, the effect of SR on token-level embeddings—and, by extension, on SL performance in a decoder-as-encoder setup—remains largely unexplored. To fill this gap, we explore the capacity of SR as a tool for token-level representation learning. Through controlled fine-tuning experiments, we show that SR implicitly enables bidirectionality within decoder-only models, yielding token-level embeddings that outperform both strong encoders and unmasked decoders across multiple SL benchmarks. In a follow-up analysis, we find that, contrary to previous findings, repeating the input more than once often results in performance gains rather than degradation ( 37 ) . 1 1 1 Reported in experiments not included in the main paper https://openreview.net/forum?id=Ahlrf2HGJR¬eId=zYNHEaQiWO . To mitigate the computational cost of sequence repetition, we utilize early exiting ( 47 ; 50 ; 35 ) and extract embeddings from intermediate layers. Fine-tuning the model only up to a selected layer yields representations that perform on par with those from the final layer, while being at least 1.39 \times more efficient to compute. These results reinforce findings that intermediate LM layers are critical for downstream tasks ( 9 ; 36 ) . The contributions of our work are threefold: (1) we find SR is a simple, practical, and effective strategy for SL with decoder-only models, outperforming more complex alternatives; (2) we show that, contrary to prior findings, performance improves as the number of sequence repetitions increases; (3) we demonstrate that early exiting mitigates the computational overhead of SR while maintaining performance competitive to last-layer embeddings. 2 Sequence repetition Previous works have shown that repeating the input sequence once is a simple method for bypassing the CM used in decoder-only models ( 48 ; 37 ) . In this way, tokens in the repeated sequence have access to the entire sequence on the left-hand side, allowing the decoder-only model to function bidirectionally. However, while prior work has found that additional repetitions yield no further gains on sequence-level tasks 48 ; 37 ; 7 , our findings show that the opposite is true for token-level tasks, where performance improves with each repetition (§ 4.2 ). We hypothesize that this occurs because additional repetitions increase the model’s effective processing capacity for each token. On the other hand, we posit that increasing the repetition count is not necessary for strong sequence-level performance because each token already encodes a compressed representation of the entire preceding sequence 32 . We further note that fine-tuning is necessary for models to learn how to fully utilize this increased capacity, which is underexplored by previous works who studied the effect of repetition count increase only in the zero-shot setting. We now outline our intuition of how repetitions enable bidirectional information flow during the computation of attention weights. Let s=[t_{1},\dots,t_{n}] be an input sequence of length n . We denote the repeated sequence as s^{k} , where k=r+1 is the total number of instances and r is the number of repetitions. The final sequence has a length of k\cdot n , as illustrated in Figure 1 . Let A be the attention weight matrix computed via standard scaled dot-product attention 43 using causal masking 34 : h_{a}(s)=A(s)V=\text{softmax}\left(\frac{QK^{T}+\mathit{CM}}{\sqrt{d}}\right)V, where Q=sW_{Q} , K=sW_{k} , and V=sW_{v} are query, key, and value matrices, respectively, d is the dimensionality of query and key vectors, and \mathit{CM} is the causal mask defined as: \mathit{CM}=\begin{bmatrix}0&-\infty&\cdots&-\infty\\ 0&0&\cdots&-\infty\\ \vdots&\vdots&\ddots&\vdots\\ 0&0&\cdots&0\end{bmatrix}. Due to causal masking, A is a lower-triangular n\times n matrix. For the repeated sequence s^{k} , which is composed of k instances of s , the dimensions of A grow to kn\times kn , and it can be viewed as a block matrix made of k^{2} individual n\times n submatrices: A(s_{k})=\begin{bmatrix}A_{11}&A_{12}&\cdots&A_{1k}\\ A_{21}&A_{22}&\cdots&A_{2k}\\ \vdots&\vdots&\ddots&\vdots\\ A_{k1}&A_{k2}&\cdots&A_{kk}\end{bmatrix}. Each submatrix A_{i,j} corresponds to the attention weights between tokens in the i -th and j -th instances of sequence s within the

Read the complete text