TESS 2 : A Large-Scale Generalist Diffusion Language Model

Jaesung Tae, Hamish Ivison, Sachin Kumar, Arman Cohan · 2025

Read TESS 2 : A Large-Scale Generalist Diffusion Language Model by Jaesung Tae, Hamish Ivison, Sachin Kumar, Arman Cohan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13917.txt.

Introduction

Jaesung Tae, Hamish Ivison, Sachin Kumar, and Arman Cohan present TESS 2, a diffusion-based language model, in a 2025 paper. Diffusion models build text by refining noisy drafts rather than writing word by word. Past instruction-following versions lagged behind standard models.

TESS 2 starts from a strong standard model, adapts it with continued training under a diffusion loss, and then tunes it on instructions. The team found that both the adaptation step and the choice of base model matter greatly. They also add reward guidance, a steering step used at generation time that needs no retraining. TESS 2 beat rival diffusion models and matched or topped strong standard ones. More computing at use time improved it further.

Catalog record

CREATORJaesung Tae, Hamish Ivison, Sachin Kumar, Arman Cohan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13917

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TESS 2 : A Large-Scale Generalist Diffusion Language Model Jaesung Tae, Hamish Ivison, Sachin Kumar, Arman Cohan Abstract We introduce TESS 2 , a general instruction-following diffusion language model that outperforms contemporary instruction-tuned diffusion models, as well as matches and sometimes exceeds strong autoregressive (AR) models. We train TESS 2 by first adapting an AR model via continued pretraining with the usual cross-entropy as diffusion loss, and then performing further instruction tuning. We find that adaptation training as well as the choice of the base model is crucial for training good instruction-following diffusion models. Furthermore, we propose reward guidance, a novel and modular inference-time guidance procedure to align model outputs without needing to train the underlying model. Finally, we show that TESS 2 further improves with increased inference-time compute, highlighting the utility of diffusion LMs in having fine-grained controllability over the amount of compute used at inference time. Code and models are available at https://github.com/hamishivi/tess-2 . 1 Introduction Existing language models are predominantly trained using an autoregressive (AR) paradigm. Prior works have suggested that AR models may have a number of limitations, especially in planning and self-correction ( Lin et al., 2021 ; Bachmann and Nagarajan, 2024 ; Huang et al., 2024 ) . While recent work has made large strides in improving AR model reasoning, this hinges on increasing test-time compute using specialized reinforcement learning-based training procedures ( Kumar et al., 2024 ; Laskin et al., 2022 ) and costly long generations ( DeepSeek-AI, 2025 ; Min et al., 2024 ) . An alternative paradigm lies in diffusion language models , which have shown promise in addressing some of the limitations of AR models ( Ye et al., 2024 ; Zhang et al., 2023 ) . Diffusion models provide plug-and-play controllability at inference time Li et al. (2022) , are naturally capable of flexible generation such as infilling, and enable precise control of inference compute, more so than recent long chain-of-thought approaches. However, diffusion LMs have remained relatively small scale and focused on improving intrinsic metrics such as perplexity ( Lou et al., 2024 ; Gulrajani and Hashimoto, 2023 ) , rather than examining common downstream tasks used to evaluate AR LMs. This constrains their broader impact and practical applicability. Figure 1: Overview of inference with TESS 2 . We provide a chat-template formatted query and iteratively denoise over the simplex space for a number of diffusion steps (typically 100). Optionally, we also incorporate reward guidance at each diffusion step by applying the gradient of the predicted reward to the intermediate logits, i.e., gradient ascent on reward. In this work, we aim to close this gap by introducing TESS 2 , a large-scale diffusion language model. We show that strong diffusion models can be trained by adapting from existing AR models, and that these models can be further instruction-tuned to serve as strong generalist models across varied downstream settings. We ablate a number of configurations and present a recipe for adapting existing AR LMs to diffusion models. Our final model outperforms contemporary diffusion LMs and performs close to or exceeds AR baselines, showing that continuous diffusion models can serve as generalist instruction-tuned LMs. To the best of our knowledge, we are the first work to provide a full recipe for training generalist instruction-following fully non-autoregressive continuous diffusion models, with prior work only examining performance after task-specific finetuning ( Karimi Mahabadi et al., 2024 ; Han et al., 2024a ) , using semi-autoregressive setups and only testing chat settings ( Han et al., 2023 ) , or using a discrete diffusion approach instead of a continuous one ( Gong et al., 2024 ) . We also underscore the unique advantages of diffusion LMs over AR models: the scalability of inference-time compute with increased diffusion steps and the ability to supplement the backward diffusion process with reward models. We summarize our contributions. 1. We propose a recipe for adapting LLMs into instruction-following diffusion LMs that significantly outperform prior text diffusion models and perform similarly to or better than AR models on QA and general instruction-following tasks. 2. We ablate components of the recipe and highlight salient elements , such as the choice of the base model, where possible. 3. We show that our resulting model can further improve with increased test-time compute by employing more diffusion steps. 4. We introduce reward guidance , a novel technique for steering diffusion LMs to generate text aligned with user preferences, enhancing downstream chat performance without any additional training . 2 Related Work Diffusion Language Models With the popularization of diffusion models in domains such as image ( Nichol and Dhariwal, 2021 ) , audio ( Kong et al., 2020 ; Tae et al., 2022 ; Shen et al., 2023 ) , video ( Ho et al., 2022 ) , and text-to-image generation ( Saharia et al., 2022 ; Ramesh et al., 2022 ) , there have been growing attempts to adapt and improve diffusion methods for language generation. One line of diffusion LMs proposes a diffusion process that adds Gaussian noise to word embeddings Li et al. (2022) ; Gulrajani and Hashimoto (2023) . Another approach uses discrete diffusion based on categorical jump probabilities and continuous-time Markov chains ( Lou et al., 2024 ; Richemond et al., 2022 ; Gong et al., 2024 ; Nie et al., 2025 ) , removing the need for mapping diffused embeddings to discrete inputs or outputs. In contrast, simplex diffusion language models maintain the continuity of the diffusion process, instead aiming to learn discrete data by modelling over a continuous simplex. We build off TESS ( Karimi Mahabadi et al., 2024 ) , which diffuses over the probability simplex and itself is a fully-non-autoregressive version of SSD-LM ( Han et al., 2022 ) . While Han et al. (2024b) explored scaling SSD-LM for instruction-following, it remained semi-autoregressive and focused on a novel collaborative decoding strategy without releasing code, models, or data. In contrast, we show that fully non-autoregressive generation is possible at context lengths of up to 2048 tokens while only using publicly available data and model checkpoints. We also take a different approach from LLaDA Nie et al. (2025) , a contemporary work that focuses on pretraining from scratch at scale, and instead explore how we can leverage already pretrained AR models to reformulate them as diffusion LMs. Adapting Diffusion Models Concurrent and prior work has also examined instruction-tuning diffusion models and adapting them from existing AR models. Han et al. (2024a) builds off SUNDAE ( Savinov et al., 2022 ) and shows that AR models can be successfully converted into diffusion models for further downstream finetuning. They focus on training diffusion checkpoints that can then be finetuned to perform specific tasks, as opposed to examining instruction tuning and general performance as done in this work. Ye et al. (2023) examines scaling reparameterized discrete diffusion models ( Zheng et al., 2024 ) by adapting XLM-R checkpoints ( Conneau et al., 2020 ) , but report that they are unable to adapt more modern LMs and find their models fall short on more complex reasoning tasks. Concurrent to our work, Gong et al. (2024) proposes DiffuLlama, an absorbing discrete diffusion model ( Austin et al., 2021 ) adapted

Read the complete text