XLM : A Python package for non-autoregressive language models

Dhruvesh Patel Durga Prasad Maram Sai Sreenivas Chintha Benjamin Rozonoyer · 2025

Read XLM : A Python package for non-autoregressive language models by Dhruvesh Patel Durga Prasad Maram Sai Sreenivas Chintha Benjamin Rozonoyer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17065.txt.

Introduction

Dhruvesh Patel and coauthors release XLM, a Python package for non-autoregressive language models, which write text in parallel rather than word by word. Interest in such models has revived, but code was scattered and bespoke, blocking fair comparisons.

Each model family used to need its own data collation, loss, and prediction logic. XLM shares those parts so new methods can be built fast and tested alike. A companion package, xlm-models, ships small pre-trained models for the community to use at once. The authors aim to give non-autoregressive research the standard tooling that autoregressive work has long enjoyed.

Catalog record

CREATORDhruvesh Patel Durga Prasad Maram Sai Sreenivas Chintha Benjamin Rozonoyer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-17065

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

XLM : A Python package for non-autoregressive language models Dhruvesh Patel Durga Prasad Maram Sai Sreenivas Chintha Benjamin Rozonoyer Abstract In recent years, there has been a resurgence of interest in non-autoregressive text generation in the context of general language modeling. Unlike the well-established autoregressive language modeling paradigm, which has a plethora of standard training and inference libraries, implementations of non-autoregressive language modeling have largely been bespoke making it difficult to perform systematic comparisons of different methods. Moreover, each non-autoregressive language model typically requires it own data collation, loss, and prediction logic, making it challenging to reuse common components. In this work, we present the XLM python package, which is designed to make implementing small non-autoregressive language models faster. With a secondary goal of providing a suite of small pre-trained models (through a companion xlm-models package) that can be used by the research community. 1 1 1 Python package: https://pypi.org/project/xlm-core/ and https://pypi.org/project/xlm-models/ . Video: https://drive.google.com/file/d/1uP49n-Gi5sj-Qd6-N1hXaLolEYYKU275/view?usp=drive_link . Source code: https://github.com/dhruvdcoder/xlm-core . ythonline]pythonfontsize=, bgcolor=codebg,breakanywhere=false ashline]bashfontsize=, bgcolor=codebg,breakanywhere=false,autogobble=false 1 Introduction Autoregressive language models (ARLMs), which generate text sequentially from left to right by adding one token at a time, are well established with a plethora of standard training and inference libraries ( Wolf et al. 2020 ; OLMo et al. 2024 ) . However, recently, there has been a resurgence of research interest in non-autoregressive text generation due to its potential for faster inference speeds and better generation quality for certain tasks. Unlike left-to-right generation, non-autoregressive text generation can be done in many ways, for example, using masked diffusion language models ( Sahoo et al. 2024 ) , Gaussian diffusion language models ( Gulrajani and Hashimoto 2023 ) , insertion language models ( Patel et al. 2025 ) , edit-based language models ( Havasi et al. 2025 ) , etc. Moreover, each method typically requires its own data collation, loss, and prediction logic, making it challenging to reuse common components across different methods. The rapidly expanding landscape of these methods has led to many bespoke implementations, making it extremely difficult to compare them systematically. In this work, we present the XLM python package, which aims to provide a unified framework for developing and comparing small non-autoregressive language models. XLM uses Pytorch ( Paszke et al. 2019 ) as the deep learning framework, Pytorch Lightning ( Falcon and The PyTorch Lightning team 2019 ) for training utilities, and Hydra ( Yadan 2019 ) for configuration management. XLM is designed to make implementing small non-autoregressive language models faster without sacrificing flexibility. The rest of the paper is organized as follows. Section 2 discusses the core design principles of XLM . In section 3 , we discuss how XLM serves a unique purpose in the landscape of LLM libraries. Section 4 presents a high-level overview of the three core components of XLM , followed by section 5 that provides a step-by-step demonstration of how one would implement a new language model using XLM . Finally, Section 6 presents a set of benchmarking results where we implement three models in XLM to reproduce known results. 2 Design Principles XLM follows the principle of maximal independence. The core library provides access to a small number of shared components, which are designed to be model independent, and can be used by any kind of language model. Each model implementation lives in its own folder/package and is completely independent of other models. This allows researchers to keep their model code clean, self-contained, and easy to share. It also allows them to use their model outside the XLM framework without refactoring. Maximal independence is achieved by following design choices. Composition over inheritance. The maximal independence is achieved by using composition over inheritance ( Gamma et al. 1995 ) , wherein the core components delegate model specific logic to the specific model instance. For example, as shown in Figure 1 , the DataModule carries a collection of DatasetManager instances , one for each dataset, and the creation of dataloaders is delegated to the respective DatasetManager instance depending on the stage (train, val, test, or predict). Similarly, the Harness carries instances of Model , LossFunction , Predictor , to which it delegates the model specific logic for forward pass, loss computation, and generation respectively. Moreover, one can swap out one or more of the four components with a different but compatible implementation without having to change all four. Copy over branching. This principle goes against the common wisdom of not copying code. However, in case of research codebases, copying reduces code complexity and helps increase the speed of development ( Wolf et al. 2020 ) . It naturally creates independence. Moreover, it also allows templatizing the process of creating a brand new model which helps rapid prototyping by humans as well as LLMs ( Li et al. 2025 ) . Arbitrary code injection. Python is a highly dynamic language, which allows one to inject arbitrary code at runtime. In production and public facing codebases, this creates a security risk, but this flexibility is a boon for research codebases, as it allows rapid prototyping and experimentation. As we will discuss in section 4.3 , Hydra ( Yadan 2019 ) provides a powerful mechanism to inject arbitrary code at runtime by allowing one to fill a specific slot with an instance of any class. 3 Related Work Due to the rapid development of LLMs, there are many libraries for training ( Wolf et al. 2020 ; von Werra et al. 2020 ; Lightning-AI 2023 ; OLMo et al. 2024 ) and inference using auto-regressive LLMs ( Kwon et al. 2023 ) . On the other hand, there are only a handful of python libraries that support non-autoregressive sequence modeling like FairSeq ( Ott et al. 2019 ) , and AllenNLP ( Gardner et al. 2017 ) . Moreover, even these libraries are no longer actively maintained and do not support non-autoregressive language modeling. To the best of our knowledge, XLM is the only library that supports fast prototyping of small non-autoregressive language models, and is geared towards providing a suite of reference implementations for up and coming non-autoregressive language modeling methods. 4 Core Components of XLM Figure 1: Overview of XLM design. It consists of two classes of components: the core components ( Harness and DataModule ) and the model-specific components, whose implementations depend on the model logic. These components are defined in the configuration files managed by Hydra (see fig. 3 ), enabling arbitrary component swapping. The Harness component is responsible for instantiating all components (model, loss, predictor, etc.) and delegating their respective functionalities. The DataModule component manages multiple datasets across workflow stages using DatasetManager objects, each handling a dataset and an appropriate Collator . In this section, we will discuss the the Harness , the DataModule , and the configuration management, which together handle the execution flow of all the supported workflows ( section 5.7 ) like training, evaluation, prediction and debugging. In most use cases, these components need not be touched by the user, removing the need for most of the boilerplate code. 2 2 2 XLM also has some useful additional features described in appendix F . 4.1 DataModule The base TextDataModule , which builds on top of Lightning DataModule, provides a

Read the complete text