Bringing Emerging Architectures to Sequence Labeling in NLP
Ana Ezquerro, Carlos Gómez-Rodríguez, David Vilares · 2025
Read Bringing Emerging Architectures to Sequence Labeling in NLP by Ana Ezquerro, Carlos Gómez-Rodríguez, David Vilares (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25918.txt.
Introduction
Ana Ezquerro, Carlos Gomez-Rodriguez, and David Vilares test new neural designs on sequence labeling in a 2025 paper. Tagging each word in a sentence, for roles or entities, is dominated by pre trained transformer encoders. Rival designs such as long short term memory variants, state space models, diffusion models, and adversarial setups have shone in language modeling but were little tried here.
The team ran them on tagging tasks of varied structure and label size, across several languages. Strong results from simpler settings did not always carry over. Gains shrank on harder structured tasks and on some languages, a caution for swapping architectures in real tagging work.
Catalog record
| CREATOR | Ana Ezquerro, Carlos Gómez-Rodríguez, David Vilares |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-25918
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bringing Emerging Architectures to Sequence Labeling in NLP Ana Ezquerro, Carlos Gómez-Rodríguez, David Vilares Abstract Pretrained Transformer encoders are the dominant approach to sequence labeling. While some alternative architectures—such as xLSTMs, structured state-space models, diffusion models, and adversarial learning—have shown promise in language modeling, few have been applied to sequence labeling, and mostly on flat or simplified tasks. We study how these architectures adapt across tagging tasks that vary in structural complexity, label space, and token dependencies, with evaluation spanning multiple languages. We find that the strong performance previously observed in simpler settings does not always generalize well across languages or datasets, nor does it extend to more complex structured tasks. 1 Introduction Sequence labeling (SL) is a problem in machine learning, and particularly in NLP, where each element in a sequence is assigned exactly one output label. A feature of SL tasks is that labels are not predicted in isolation: they often depend on neighboring inputs. To address these dependencies, various architectures have been proposed over the years to model both short- and long-range interactions between input tokens, including Conditional Random Fields ( Lafferty et al. 2001 , CRF;) , sliding-window perceptrons Zhang and Clark 2008 , Long Short-Term Memory networks ( Hochreiter and Schmidhuber 1997 , LSTMs;) , and Convolutional Neural Networks ( LeCun et al. 1995 , CNNs;) . Today, Transformers Vaswani et al. 2017 , pretrained on large data, are the dominant approach for tagging tasks, offering superior ability to model relations across words. At the same time, research continues to explore both alternatives to Transformers and architectural variants of them across a range of machine learning tasks. This includes, for instance, enhanced contextualization mechanisms ( Beck et al. 2024 , xLSTM;) and structured state-space models ( Gu et al. 2022 , Mamba;) for language modeling, as well as diffusion models ( Ho et al. 2020 ) and generative adversarial networks ( Goodfellow et al. 2014 , GANs;) , which were originally developed for computer vision applications. Some of these architectures have shown competitive, albeit preliminary, performance compared to Transformers in metrics such as perplexity (see again Beck et al. 2024 ) and could be promising for tagging tasks. Meanwhile, others have already been adapted for sequence labeling in relatively simpler settings ( Huang et al. 2023 ; Tong et al. 2024 , e.g.,) , typically focusing on tasks like named-entity recognition (NER), part-of-speech (PoS) tagging, or (Chinese/Japanese/Korean) word segmentation. However, evaluations have mostly remained within these flatter, coarse-grained setups, leaving open questions about how well these models generalize to more fine-grained or complex scenarios, such as recent linearizations for structured tasks involving trees ( Gómez-Rodríguez and Vilares 2018 ; Kitaev and Klein 2020 ; Amini et al. 2023 ) or graphs ( Ezquerro et al. 2024b ) . Contribution We investigate alternative architectures beyond pretrained Transformers for sequence labeling, focusing on architectures not originally designed for token-level classification but with potential for this task. Specifically, we explore how these models can be adapted to capture linguistic structure in sequence-labeling problems, considering tasks of varying complexity, output label spaces, and dependency spans. Rather than aiming to outperform Transformers universally, the goal is to better understand the relative strengths and limitations of these architectures across different problem types. Our results show that bidirectional xLSTM architectures are generally superior to the traditional BiLSTMs across tasks and datasets, although they still fall behind Transformers. Diffusion tagging and state-space models trail the baseline, suggesting they may be less suited for NLP tagging. Finally, adversarial tagging yields a noteworthy result, consistently rivaling or surpassing the Transformer baseline across diverse tasks, including the most complex structured settings. Our code is publicly available at https://github.com/anaezquerro/separ . 2 Background We now outline concepts in tagging and modeling advances. 2.1 Sequence labeling for NLP Classical problems like PoS tagging or lemmatization naturally align with the definition of sequence labeling. Others, such as NER, chunking ( Ramshaw and Marcus 1995 ) , segmentation ( Hacioglu et al. 2004 ) , semantic role labeling Strubell et al. 2018 and slot filling ( Li et al. 2020 ) , can also be framed as tagging problems, typically using lightweight encoding schemes that assign token-level labels (e.g. IOB encoding). Despite gains from pretrained Transformers, many tasks, especially simpler ones, were already tractable with shallow, easier-to-deploy models. Similarly, previous efforts focused on reformulating tree- and graph-structured tasks as tagging through linearizations. However, pre-neural models struggled with these problems. Spoustová and Spousta 2010 showed that linguistically informed linearizations trained with pre-neural models performed impractically compared to the state of the art at the time. This limitation of earlier models started to change with context-aware encoders based on BiLSTMs or Transformers, which have recently revived sequence labeling for structured prediction, though effectively modeling such structure was not immediate Li et al. 2018 . In addition, they open up new possibilities for evaluating alternative sequence tagging architectures under more demanding testbeds, with large output spaces and long-range dependencies. In this context, linearization strategies have been proposed for both continuous Gómez-Rodríguez and Vilares 2018 ; Kitaev and Klein 2020 ; Amini and Cotterell 2022 and discontinuous constituent parsing Vilares and Gómez-Rodríguez 2020 , as well as for projective and non-projective syntactic dependency parsing Strzyz et al. 2019 ; Amini et al. 2023 and, recently, graph parsing Ezquerro et al. 2024b . 2.2 Sequence modeling While LSTMs and Transformers are the standard encoders for token-level classification tasks, recent years have seen growing interest in alternative techniques—some of which still leverage self-attention—and architectures that replace the Transformer with different contextualization systems. Although many of these methods were not initially designed for sequence labeling, we now examine their potential. GANs, created for image generation, have been extended to text generation in NLP, addressing the challenge posed by the discrete nature of language Kusner and Hernández-Lobato 2016 ; Yu et al. 2017 . However, fewer studies have explored their use in non-generative tasks, which require generating or refining structured outputs rather than free-form text. Notably, Parnow et al. 2021 trained a GAN-based tagging system to enhance grammatical error correction by enriching the learning process with generated errors. Tong et al. 2024 improved word segmentation and NER with a GAN-based framework, using the generator as a labeler and a discriminator to guide accurate sequences. Similarly, diffusion models Ho et al. 2020 are often used for generative NLP tasks He et al. 2023 ; Han et al. 2023 . While several studies improve denoising embeddings in latent space Gao et al. 2024 ; Zhou et al. 2024 to enhance text-to-text generation Shi et al. 2023 ; Liu et al. 2024 , few have explored their potential for tagging. Notably, some recent work has adapted diffusion processes for NER Shen et al. 2023 and PoS tagging Huang et al. 2023 . In addition, recent work has explored alternative architectures replacing self-attention. Gu et al. 2022 introduced structured-state space models (SSM) for linear-time language modeling, addressing the quadratic complexity of Transformers. Beck et al. 2024 introduced the xLSTM, a variant of the LSTM with parallelizable capabilities and better modeling of dependencies. Still, xLSTM has been mainly evaluated on language modeling and bioinformatics Heidari et al. 2025 ; Sun et al. 2025 ,