Continuous Diffusion Models Can Obey Formal Syntax Jinwoo Kim, Taylor Berg-Kirkpatrick, Loris D’Antoni Abstract Diffusion language models offer a promising alternative to autoregressive models due to their global, non-causal generation process, but their continuous latent dynamics make discrete constraints—e.g., the output should be a JSON file that matches a given schema—difficult to impose. We introduce a training-free guidance method for steering continuous diffusion language models to satisfy formal syntactic constraints expressed using regular expressions. Our approach constructs an analytic score estimating the probability that a latent state decodes to a valid string accepted by a given regular expression, and uses its gradient to guide sampling, without training auxiliary classifiers. The denoising process targets the base model conditioned on syntactic validity. We implement our method in Diffinity on top of the PLAID diffusion model and evaluate it on 180 regular-expression constraints over JSON and natural-language benchmarks. Diffinity achieves 68-96% constraint satisfaction while incurring only a small perplexity cost relative to unconstrained sampling, outperforming autoregressive constrained decoding in both constraint satisfaction and output quality. 1 Introduction Diffusion-based language models are an increasingly attractive alternative to autoregressive generation, particularly in settings where bidirectional context or global revision is useful (e.g., infilling and long-horizon planning) ( 10 ; 13 ; 17 ; 12 ; 19 ; 29 ) . Because a diffusion model denoises an entire sequence jointly across timesteps, it can in principle model non-causal dependencies and enforce global structure in ways that left-to-right autoregressive decoding cannot. This aspect makes diffusion a natural candidate for structured generation , where outputs must satisfy discrete syntactic requirements—such as JSON schema, mathematical formats, or regular expressions. We show that continuous diffusion models can be steered to satisfy formal syntax (regular expressions), using a training-free guidance mechanism. Why is this problem challenging? In the autoregressive models, syntactic validity is typically enforced via constrained decoding , which filters next-token choices to ensure the final output is well formed ( 7 ) . However, a continuous diffusion model never maintains an explicit discrete prefix or in general a plain text representation of the output sequence: each timestep operates on a continuous latent representation of the entire sequence. As a result, the core primitive behind constrained decoding—masking next-token choices that would make the current prefix uncompletable—does not transfer directly to continuous diffusion. Existing constrained-decoding adaptations for diffusion ( 25 ; 18 ) therefore target specialized discrete diffusion variants that mimic autoregressive unmasking transitions ( 19 ; 29 ) (though they do not require unmasking to happen left-to-right). Other approaches for steering diffusion models to satisfy constraints use classifier guidance ( 4 ; 24 ) , which requires training an auxiliary model for each constraint. Because regular expressions induce highly nonlocal, discrete constraints, a classifier-free method for steering continuous diffusion models to obey formal syntax has remained out of reach—until now. Our solution. We address this gap by introducing a training-free guidance mechanism for continuous diffusion language models that targets discrete syntactic constraints. At a high level, our method mirrors classifier guidance ( 4 ) : rather than modifying the base diffusion model, we add a guidance term during sampling that pushes trajectories toward regions of latent space that decode to syntactically valid text. The key difference is that we do not train an auxiliary classifier; instead, we analytically compute the expected probability that a given latent state will decode to a sequence satisfying a target regular expression. Using the gradient of this probability to steer sampling yields two practical benefits: (1) Training-free guidance: The guidance score is computed analytically from the constraint, so applying a new regex does not require training (or storing) an additional classifier. (2) Principled conditioning: The resulting guidance approximates conditioning the diffusion sampler on syntactic validity rather than heuristically biasing token choices. We instantiate our framework for regular expressions (regexes), which capture a wide range of practical syntactic constraints and are substantially more expressive than the constraints supported by prior non-classifier diffusion steering methods ( 10 ) . Moreover, our guidance targets the conditional distribution of the base model given syntactic validity, enabling constraint-aligned generation in which samples remain faithful to the model rather than being distorted by ad hoc decoding heuristics. This stands in contrast to autoregressive constrained decoding, where enforcing validity can be at odds with preserving the original model distribution ( 20 ; 22 ; 1 ) . Results. We implement our method in Diffinity and apply it to PLAID, a a state-of-the-art continous diffusion language model ( 10 ) in terms of performance. 1 1 1 While there exist other diffusion models that display better performance ( 19 ; 29 ) , these are unmasking based discrete diffusion models and do not directly fit our framework. Across structured-generation benchmarks (including regular versions of JSONSchemaBench ( 6 ) and natural-language regex constraints), Diffinity achieves high constraint satisfaction—up to 70% on complex JSON-schema regexes—while incurring only a small perplexity cost relative to unconstrained sampling. In addition, Diffinity is more reliable in practice than existing autoregressive constrained-decoding toolkits ( 9 ; 28 ) , which can fail under finite token budgets ( max_tokens ) by stalling in low-probability parts of the automaton; we observe improvements in both validity and text quality against autoregressive models of comparable capacity. Contributions. We make three contributions: 1. We propose a training-free guidance method that steers continuous diffusion language models to satisfy discrete regex constraints (§ 2 ). 2. We provide a theoretical justification showing that the resulting sampler targets the base model conditioned on syntactic validity (§ 3 ). 3. We implement Diffinity and demonstrate strong validity–quality trade-offs on structured-generation benchmarks, outperforming autoregressive constrained decoding in both reliability and text quality under finite token budgets (§ 4 ). 2 Steering Diffusion Models to Satisfy Regular Constraints We now describe at a high level our approach to steering diffusion models towards generating outputs that satisfy the syntactic constraint imposed by a given regular expression. 2.1 A Crash Course on Diffusion Models for Text Diffusion models—more precisely, denoising diffusion probabilistic models (DDPMs) ( 11 ) —generate data by reversing a gradual noise-addition process. We denote the data as x , where x_{T} is pure Gaussian noise and x_{0} is the “clean” data. Critically, in continuous text diffusion models like PLAID ( 10 ) , x_{0} is not discrete text but a continuous embedding of a text sequence—specifically, a matrix of embedding vectors, one for each token position. Because x_{0} resides in a continuous space, we can model the generation process via Gaussian denoising. The “reverse” denoising step at each timestep t is defined by sampling x_{t-1} from a Gaussian distribution parameterized by the model: x_{t-1}=\mu_{\theta}(x_{t},t)+\sigma_{t}\epsilon (1) In Equation 1 , x_{t} is the current noisy latent embedding, x_{t-1} is the less-noisy latent to be computed, \epsilon is noise sampled from the standard normal distribution \mathcal{N}(0,I) , and \sigma_{t} is a fixed variance schedule. The term \mu_{\theta}(x_{t},t) represents the predicted mean of the posterior distribution given the current latent x_{t} . In practice, the model uses a neural network to predict the clean embedding x_{0} from x_{t} , and \mu_{\theta} is computed analytically as a weighted linear combination of x_{t} and this predicted x_{0} . Since the denoised output x_{0} is still a continuous latent representation, obtaining discrete text requires a decoder \operatorname{Dec} . This decoder maps the latent embedding x_{0} to a sequence of probability distributions over the vocabulary—a matrix of unigram distributions, where the distribution at each position is independent of the others. The final output sentence s is obtained by sampling from \operatorname{Dec}(x_{0}) , where each token in s is sampled independently from the corresponding unigram distribution at its position. Following previous work, we rely on this interpretation of the latent space as a decoder-induced unigram distribution to derive the expected probabilities discussed in the following sections. q_{0} q_{1} q_{2} q_{3} q_{4} c a u t t \underbrace{\begin{array}[]{r ccc}&c_{1}&c_{2}&c_{3}\\ \begin{matrix}{\mathsf{a}}\\ {\mathsf{c}}\\ {\mathsf{r}}\\ {\mathsf{t}}\\ {\mathsf{u}}\end{matrix}&\left(\begin{matrix}0.1\\ {\color[rgb]{0,0.5,1}0.7}\\ 0.2\\ 0.0\\ 0.0\end{matrix}\right)&\left(\begin{matrix}{\color[rgb]{0.72,0,0}0.3}\\ 0.0\\ 0.1\\ 0.1\\ {\color[rgb]{0,0.88,0}0.5}\end{matrix}\right)&\left(\begin{matrix}0.0\\ 0.2\\ 0.3\\ {\color[rgb]{1,0.49,0}0.5}\\ 0.0\end{matrix}\right)\\ \end{array}}_{x_{\mathit{t}}} \begin{array}[]{cc}\underbrace{\begin{array}[]{cccccc}&q_{0}&q_{1}&q_{2}&q_{3}&q_{4}\\ q_{0}&0.0&{\color[rgb]{0,0.5,1}0.7}&0.0&0.0&0.0\\ q_{1}&0.0&0.0&0.1&0.0&0.0\\ q_{2}&0.0&0.0&0.0&0.0&0.0\\ q_{3}&0.0&0.0&0.0&0.0&0.0\\ q_{4}&0.0&0.0&0.0&0.0&0.0\\ \end{array}}_{{\large M_{1}}}&\hskip 8.50012pt\underbrace{\begin{array}[]{cccccc}&q_{0}&q_{1}&q_{2}&q_{3}&q_{4}\\ q_{0}&0.0&0.1&0.0&0.0&0.0\\ q_{1}&0.0&0.0&{\color[rgb]{0.72,0,0}0.3}&{\color[rgb]{0,0.88,0}0.5}&0.0\\ q_{2}&0.0&0.0&0.0&0.0&0.1\\ q_{3}&0.0&0.0&0.0&0.0&0.1\\ q_{4}&0.0&0.0&0.0&0.0&0.0\\ \end{array}}_{M_{2}}\\ \underbrace{\begin{array}[]{cccccc}&q_{0}&q_{1}&q_{2}&q_{3}&q_{4}\\ q_{0}&0.0&0.2&0.0&0.0&0.0\\ q_{1}&0.0&0.0&0.0&0.0&0.0\\ q_{2}&0.0&0.0&0.0&0.0&{\color[rgb]{1,0.49,0}0.5}\\ q_{3}&0.0&0.0&0.0&0.0&{\color[rgb]{1,0.49,0}0.5}\\ q_{4}&0.0&0.0&0.0&0.0&0.0\\ \end{array}}_{M_{3}}&{\small\begin{split}&\mathds{E}_{s\sim\mathsf{Dec}(x_{\mathit{t}})}[s\in\mathcal{L}]\\ &(\begin{pmatrix}1&0&0&0&0\end{pmatrix}\cdot M_{1}\cdot M_{2}\cdot M_{3})[q_{4}]=\\ &1*{\color[rgb]{0,0.5,1}0.7}*({\color[rgb]{0.72,0,0}0.3}*{\color[rgb]{1,0.49,0}0.5}+{\color[rgb]{0,0.88,0}0.5}*{\color[rgb]{1,0.49,0}0.5})=\\ &0.28\end{split}}\end{array} Figure 1 : The automaton for the regex c(a|u)t describing the regular constraint \mathcal{L}=\{\texttt{cat},\texttt{cut}\} , the unigram distribution \mathsf{Dec}(x_{\mathit{t}}) defined by the current latent x_{\mathit{t}} on a vocabulary \{{\texttt{a, c, r, t, u}}\} of size 5, and transition matrices for this automaton and latent space for a sequence of length 3. Valid transitions inside the automaton and their probabilities are color-coded. 2.2 Denoising to Satisfy Regular Constraints Our goal is to steer the denoising process defined by Equation 1 so that the generated output sentence s belongs to a regular language \mathcal{L} , specified by a regular expression or a deterministic finite automaton (DFA). Previous work has, to some extent, succeeded in guiding the diffusion process towards simple constraints such as “The sentence should start with the word ‘Apple’,” by simply maximizing the probability that the token “Apple” occurs at the start of the sequence ( 10 ) . However, these existing approaches fail to extend to more complex structural constraints such as regexes. The standard approach to provide more complex structural guidance in other modalities is through classifier guidance ( 4 ) . This approach steers the denoising process with input gradients from a pretrained control target classifier. However, training a classifier is often resource-heavy and difficult for syntactic constraints—e.g., a new classifier would need to be trained for each new regex. The key idea in this paper is that, for syntactic constraints, we can construct an analogous steering mechanism via dynamic programming without ever training a classifier . More specifically, because we can interpret the current noisy latent x_{t} as defining a unigram distribution \operatorname{Dec}(x_{t}) over the vocabulary, we can analytically compute the probability that a sentence s sampled from this distribution satisfies \mathcal{L} . We use this expectation as a proxy for our “classifier” score, and show how to compute its input gradient. Specifically, our modified steering procedure uses the following guided denoising step: x_{t-1}=\mu_{\theta}(x_{t},t)+\gamma\sigma_{t}^{2}\nabla_{x_{t}}\log\mathbb{E}_{s\sim\operatorname{Dec}(x_{t})}[s\in\mathcal{L}]+\sigma_{t}\epsilon (2) In this equation, \mu_{\theta}(x_{t},t) represents the original model’s predicted mean (the unconditioned update). The guidance term is defined by \nabla_{x_{t}}\log\mathbb{E}_{s\sim\operatorname{Dec}(x_{t})}[s\in\mathcal{L}] , which is the gradient of the log-expected probability that a sentence s sampled from the unigram distribution \operatorname{Dec}(x_{t}) satisfies the constraint \mathcal{L} , taken with respect to x_{t} . This gradient is scaled by \sigma_{t}^{2} (the variance) to ensure the guidance signal is proportional to the noise level, and by \gamma , a guidance scale hyperparameter that controls the strength of the constraint enforcement. Intuitively, Equation 2 shifts the center of the sampling distribution at each step toward regions of the continuous latent space that decode into valid sentences in \mathcal{L} . Maximizing this expected probability biases the reverse process to ultimately result in a valid discrete output s , while the \sigma_{t}\epsilon term maintains the necessary stochasticity for high-quality generation. This formulation admits a natural interpretation through the lens of Langevin dynamics ( 24 ) , where the guidance term acts as an external potential. Specifically, the gradient of the expected probability exerts a force that steers the latent trajectory toward the manifold of embeddings that decode to strings in \mathcal{L} . 2.3 Computing the Expected Probability To compute \mathds{E}_{s\sim\mathsf{Dec}(x_{t})}[s\in\mathcal{L}] , we take advantage of the fact that any regular language \mathcal{L} can be represented as a deterministic finite automaton (DFA). Our approach uses the DFA representing the regular constraint to build a series of weighted transition matrices M_{1},\cdots,M_{n} such that M_{k}[i][j] represents the probability that the automaton will transition from state i to state j by consuming the k -th token in the generated sequence, according to the unigram distribution given by the current latent. Example 2.1 (Transition Matrices) . Consider the regex ‘ c(a|u)t ’, which describe the regular language consisting of the words \{\texttt{cat},\texttt{cut}\} , and a diffusion model with a vocabulary \{{\texttt{a, c, r, t, u}}\} . Figure 1 illustrates the DFA representation of this regular language (top-left), a distribution \mathsf{Dec}(x_{\mathit{t}}) that might occur from a latent x_{\mathit{t}} during the diffusion process, and transition matrices M_{1} , M_{2} , and M_{3} based on \mathsf{Dec}(x_{\mathit{t}}) when sequence length l=3 . Note how \mathsf{Dec}(x_{\mathit{t}}) is a unigram distribution: each column c_{i} denotes probabilities that a certain token appears at the i -th position, independent of other positions. Observe how in the first transition matrix we have that M_{1}[q_{0}][q_{1}]=\mathsf{Dec}(x_{\mathit{t}})[\mathsf{c}][1]=0.7 , i.e., the probability that the automaton will take q_{0}\xrightarrow{\mathsf{c}}q_{1} on the first token is the probability that \mathsf{c} will appear at the first position according to the current latent point x_{\mathit{t}} . Similarly, for the second matrix: M_{2}[q_{1}][q_{2}]=0.3 and M_{2}[q_{1}][q_{3}]=0.5 , as \mathsf{Dec}(x_{\mathit{t}})[\mathsf{a}][2]=0.3 and \mathsf{Dec}(x_{\mathit{t}})[\mathsf{u}][2]=0.5 . Example 2.2 (Computing Expected Probability) . Given the transition matrices M_{1},M_{2},M_{3} , we start with \mathbf{p}_{0}=(1\ \overbrace{0\cdots 0}^{l-1}) —i.e., the automaton starts at the initial state with probability 1 . The vector \mathbf{p}_{0}\cdot M_{1}\cdot M_{2}\cdot M_{3} represents the probability distribution over the states of the automaton after reading all three tokens according to the current latent x_{\mathit{t}} . We take the probability of q_{4} (the only accepting state) in this distribution to obtain the expected probability. The bottom-right of Figure 1 illustrates the computation, where 1*0.7*0.3*0.5 represents the probability that we take the path \mathsf{c,a,t} , and 1*0.7*0.5*0.5 represents the probability that we take the path \mathsf{c,u,t} . In particular, observe that M_{1}[q_{1}][q_{2}]=0.1 because the token \mathsf{a} can occur with probability 0.1 in the first position according to \mathsf{Dec}(x_{\mathit{t}}) . However, this probability gets multiplied by 0 when computing \begin{pmatrix}1&0&0&0&0\end{pmatrix}\cdot M_{1} , as the probability of the automaton being in state q_{1} is 0 at the first token—and thus does not contribute to the final expected probability (as it should, as a sentence starting with \mathsf{a} violates the regex). 3 Formal Framework First, we address the mismatch between regular-expression constraints and model tokenizers by constructing a vocabulary-aligned automaton ( Section 3.1 ). Then, we present a dynamic programming algorithm that uses the vocabulary-aligned automaton to compute the expected probability that a sample satisfies the constraint ( Section 3.2 ). Finally, we connect our approach to standard classifier guidance ( 4 ; 24 ) and show how our approach approximates the desired conditional distribution ( Section 3.3 ). 3.1 Tokenizer-Automaton Alignment Before presenting the steering algorithm, we need to address a mismatch between the character-level definition of regular languages and the token-level representation used by diffusion language models. While a regular constraint specifies a set of valid strings, models generate sequences of tokens from a fixed vocabulary, and a single string may admit multiple valid tokenizations. To correctly compute expected probabilities and their gradients, the automaton representing the constraint must accept all token sequences whose concatenation forms a valid string; otherwise, valid probability mass would be systematically ignored. Formally, if a regular language \mathcal{L} accepts a string x , then the DFA A for \mathcal{L} must accept every tokenization of x . For example, for the string cat , if the automaton accepts the token sequence ca, t but not c, at , our approach in Section 2 would ignore the probability of the latter and thus compute an incorrect expected probability. We introduce a function \mathsf{Align}(A,V) that constructs a vocabulary-aligned automaton A_{V} from a given DFA A and vocabulary V . The algorithm constructing \mathsf{Align}(A,V) (which is formalized in Appendix D and proven correct in Appendix E ) first transforms A into a character-level automaton A_{C} , whose transitions are labeled by individual characters. It then augments A_{C} by adding transitions corresponding to multi-character tokens in V whenever the underlying character path exists in A_{C} . For instance, if V contains the token ca , the algorithm adds a transition q_{0}\xrightarrow{\texttt{ca}}q_{2} whenever a path labeled c \to a exists from q_{0} to q_{2} . Tokens that do not correspond to any valid character path (e.g., rt in Figure 1 ) are ignored. Finally, the algorithm removes character-level transitions not present in V , yielding a token-level automaton compatible with the model vocabulary. 3.2 Computing the Expected Probability Algorithm 1 Computing the Expected Probability Input: DFA A=(\Sigma,Q,\mathit{q_{0}},\delta,F) , latent x , seq. length l 1: (Q_{V},q_{V_{0}},\delta_{V},F_{V})\leftarrow\mathsf{Align}(R,V) 2: \mathbf{p}_{0}\leftarrow\overbrace{\begin{pmatrix}1&0&\cdots&0\end{pmatrix}}^{|Q_{V}|} 3: for k=1 to l do 4: M_{k}\leftarrow 0_{|Q_{V}|,|Q_{V}|} 5: for (q,q^{\prime})\in\delta_{V} do 6: M_{k}[q][q^{\prime}]\leftarrow\sum_{\mathit{tok}\text{ where }\delta(q,\mathit{tok})=q^{\prime}}\mathsf{Dec}(x)[\mathit{tok}][k] 7: end for 8: \mathbf{p}_{k}\leftarrow\mathbf{p}_{k-1}M_{k} 9: end for 10: return \sum_{q\in F}\mathbf{p}_{l}[q] Having solved the alignment issue, Algorithm 1 formalizes the algorithm used to compute the expected probability \mathds{E}_{s\sim x_{t}}[s\in\mathcal{L}] given a latent x_{t} and a regular constraint \mathcal{L} . Most concepts have already been illustrated in Section 2.2 : \mathbf{p}_{0} denotes the initial probability distribution over the states Q_{V} of the (aligned) automaton, while lines 4 to 7 formalize the construction of the transition matrices M_{1},\cdots,M_{l} . Theorem 3.1 states the correctness of Algorithm 1 . Theorem 3.1 (Expected Probability) . Let A be a DFA that represents the regular language \mathcal{L} . Then Algorithm 1 returns the expected probability \mathds{E}_{s\sim x}[s\in\mathcal{L}] . The proof is by induction on the fact that \mathbf{p}_{i} denotes the correct distribution over the states of the automaton at step i . The full proof is in the Appendix, Appendix E . 3.3 Connection to Classifier Guidance Rather than deriving a new convergence proof, we observe that our approach is a training-free instantiation of Classifier Guidance ( 4 ; 24 ) . In the standard framework, conditioned generation is achieved by modifying the score function (the drift term in the reverse SDE) with the gradient of a log-likelihood term: \nabla_{x_{t}}\log p(x_{t}\mid\mathcal{L})=\nabla_{x_{t}}\log p(x_{t})+\nabla_{x_{t}}\log p(\mathcal{L}\mid x_{t}) (3) The first term is provided by the base diffusion model, while the second term—the “guidance”—is typically approximated by training a separate classifier network on noisy latents x_{t} . Our method replaces this learned classifier with the analytical expectation derived in Algorithm 1 : \log p(\mathcal{L}\mid x_{t})\approx\log\mathbb{E}_{s\sim\operatorname{Dec}(x_{t})}[s\in\mathcal{L}] (4) This substitution interprets the noisy latent x_{t} as parameterizing a valid unigram distribution via the decoder. While this is an approximation at high noise levels ( t\gg 0 ), it becomes exact as t\to 0 (where x_{t}\approx x_{0} ). By using this analytical proxy, we leverage the robust theoretical foundations of guided diffusion without the cost or instability of training auxiliary classifiers. The convergence properties of our sampler thus follow directly from the standard results for score-based conditional generation established by 2 ; 24 , contingent on the quality of this unigram approximation. 4 Evaluation We implement our approach as a tool Diffinity using the PLAID diffusion model ( 10 ) as the base model. PLAID internally uses a 32-dimensional word2vec model as the latent space, and is equipped with a 1.3B transformer that interprets points in this space as unigram probability distributions (this transformer is trained alongside PLAID as part of its denoising network). Diffinity takes the gradient of this transformer composed with our implementation of the expected probability to obtain gradients on the word2vec latent space, then steers the diffusion process by adding the gradient term to the DDPM sampling equation that PLAID uses for denoising. At the end of the diffusion process, PLAID (and also Diffinity ) samples using argmax from the final unigram distribution to produce the final concrete sentence. We reiterate that PLAID is a continuous diffusion model, and the decoding to a concrete sentence happens only once at the end of the denoising process; as opposed to discrete models ( 17 ; 19 ) which decode to a discrete sentence at each step of denoising. Benchmarks We consider 180 regular-expression constraints spanning two categories. JSON : 70 derived from JSONSchemaBench ( 6 ) (schemas converted to regexes when possible, then filtered for non-regular constraints and overly large tokenizer-aligned automata). Natural Language : 110 synthetically generated natural-language patterns covering Prefix, Suffix, Appearance, Between- n , Between (unbounded), and Word-length templates, each instantiated by sampling one or two words from the most common words in the English vocabulary ( 26 ) and an integer parameter (e.g., position or distance). Details on the benchmark construction are in Appendix F . We set the sequence length for all benchmarks to 64, which is a sufficiently long bound for generating sequences that satisfy our regex constraints. We run JSON generation benchmarks for 256 timesteps, because they result in complex automata that take a long time to compute the gradient over. For all other benchmarks, we run generation for 1024 timesteps. All other hyperparameters are set to the default values of the PLAID model ( 10 ) . Table 1 : Comparison of constraint satisfaction and Pass@10 rates for Diffinity and GPT2 variants for JSON schema benchmarks. Rates for Diffinity are reported with guidance scale 2.5 . Diffinity GPT2-Small-GCD GPT2-Medium-GCD GPT2-Large-GCD Metric Sat. Pass@10 Sat. Pass@10 Sat. Pass@10 Sat. Pass@10 JSON 68.4 91.4 79.3 98.0 75.3 97.1 77.0 98.6 Table 2 : Combined comparison of constraint satisfaction (Sat.), pass@10, perplexity (PPL), and fluency (Flu.) across natural language benchmarks. Perplexity and fluency are computed only on samples that satisfy the regex. Rates for Diffinity use guidance scale 2.5 . Diffinity GPT2-Medium-GCD GPT2-Large-GCD Benchmark Sat. Pass@10 PPL \downarrow Flu. \uparrow Sat. Pass@10 PPL \downarrow Flu. \uparrow Sat. Pass@10 PPL \downarrow Flu. \uparrow Prefix 95.7 100.0 66.5 32.2 42.8 95.0 539.9 25.3 43.5 95.0 401.7 30.9 Suffix 96.8 100.0 59.3 36.8 17.0 70.0 115.4 27.2 17.5 55.0 70.3 31.9 Appearance 92.5 100.0 58.5 34.7 2.0 5.0 133.8 33.8 3.2 10.0 69.7 40.9 Between- n 85.5 95.0 58.7 33.3 0.2 0.0 1946.8 5.0 0.5 10.0 68.1 35.0 Between (ubd.) 93.8 100.0 57.4 33.3 0.2 5.0 132.6 33.3 3.5 10.0 71.2 47.1 Word Length 95.0 100.0 60.7 43.3 90.5 100.0 169.4 40.2 89.5 100.0 125.0 44.2 Metrics and Baselines We report constraint satisfaction rate (the fraction of generated samples that satisfy the regex constraint) and pass@10 (the fraction of benchmarks for which at least one of the first 10 samples satisfies the constraint), along with perplexity (geometric mean over all constraint-matching samples), and fluency (arithmetic mean over all constraint-matching samples). Perplexity is computed using LLama3.1-8B ( 8 ) as the reference distribution, while fluency was measured on a scale of 0 to 100 by using Claude Sonnet 4.5-20250929 as an external LLM judge (prompts to derive fluency are adapted from 15 and can be found in the Appendix, Appendix C ). Our baseline is constrained generation for autoregressive models using the Guidance library ( 9 ) with GPT2-Small, Medium, and Large. This choice follows the original PLAID ( 10 ) paper ( 10 ) , which demonstrates that the capacity of the base PLAID model lies in between GPT2-Small and Medium. We evaluate Diffinity using different guidance scales \gamma ( Equation 2 ); \gamma=2.5 is most effective. The reported metrics use \gamma=2.5 unless otherwise noted. We also compare our computational overhead against PLAID’s. The PLAID model also has native guidance capabilities that allow it to loosely model some of our regular expressions, so we make comparisons where applicable. 4.1 JSON Schema Benchmarks Table 1 summarizes the results for the 70 JSON schema benchmarks. When generating regexes, Diffinity is based on a diffusion model and must generate a fixed number of tokens for each sentence: so we use the regex \texttt{.}^{\texttt{*}}\texttt{schema}\texttt{.}^{\texttt{*}} , where wildcards are padded around the schema regex, to allow additional padding tokens. For the GPT models, we use the non-padded regex (i.e., schema only). Diffinity has comparable performance to the constrained GPT-2 models when generating JSON schema, with around a 10% decrease in satisfaction rate and 7% difference in pass@10 rate. Complex structured formats pose a fundamental challenge for smaller diffusion language models such as PLAID. Our JSON Schema experiments demonstrate that Diffinity overcomes this limitation without the need of fine-tuning the model, enabling reliable generation of highly structured outputs outside the model’s training distribution. 4.2 Natural Language Benchmarks Table 2 summarizes the results for the natural language benchmarks. Due to space limitations, we present GPT2-Small-GCD’s results in Appendix G (they are similar to GPT2-Medium-GCD). In terms of constraint satisfaction and pass@10 rates, Diffinity outperforms existing autoregressive constrained decoding, achieving an average constraint satisfaction rate of 92.9% compared to 20.5% for GPT2-Large-GCD. The constraint satisfaction rates for the GPT2 models are low in these benchmarks primarily because they fall into the ‘infinite repetition trap’: for example, given a regex \texttt{[A-Za-z ] cat [A-Za-z ,.]}^{\texttt{*}} , an autoregressive model might continuously append characters to the first word if it finds that cat is an unlikely continuation to the first word it guessed until the max_tokens budget runs out. Diffinity is free of this problem, it will guide the denoising process towards an entire, coherent sentence that contains cat as the second word. Diffinity also outperforms all GPT2 variants in terms of perplexity, consistently outperforming even GPT2-Large-GCD, with the smallest perplexity difference being 9.4 for the between (udb.) category. Diffinity ’s fluency (35.8) surpasses those of GPT2 Small (25.0) and Medium (32.0) and is comparable to that of GPT2-Large (37.2). We find this result remarkable; the original PLAID paper ( 10 ) indicates that the base PLAID model sits in between GPT2-Small and Medium in terms of perplexity. The average perplexity of all samples generated by Diffinity for the natural language benchmarks is 60.0 while the average fluency is 35.8 at scale 2.5 . Our experiments indicate that the base PLAID model displays 61.6 perplexity and 46.0 fluency for unconditioned generation under the same hyperparameters (sequence length of 64 with 1024 diffusion timesteps). This shows that the perplexity gap induced by Diffinity is virtually nonexistent at scale 2.5 , while the fluency gap is also relatively low —confirming that our approach in Section 2 allows diffusion models to obey formal syntax while preserving information from the original distribution. We provide an additional experiment that showing that Diffinity preserves the underlying probability distribution in the Appendix, Section G.1 . 1 1.5 2 2.5 0 20 40 60 80 100 Guidance Scale Satisfaction Rate (%) Satisfaction (NL) Satisfaction (JSON) PPL (NL) Fluency (NL) 0 10 20 30 40 50 60 70 80 90 100 PPL / Fluency (a) Guidance-scale trade-offs for satisfaction, perplexity, and fluency. 0 20 40 60 80 100 0 20 40 60 80 100 Diffinity (PPL) PLAID Native Guidance (PPL) Appearance Prefix Suffix (b) Minimum perplexity (PPL) comparison: Diffinity vs. native PLAID guidance. 0 0.5 1 1.5 2 2.5 0 200 400 600 800 Unconstrained (batch size 1): 33.3 s Transitions ( \times 10^{6} ) Time per Sample (s) Batch Size 1 Batch Size 4 Batch Size 16 (c) Generation latency vs. automaton size. Figure 2 : Guidance-scale trade-offs, PLAID perplexity comparison, and computational overhead. 4.3 The Effect of Different Guidance Scales We study how the guidance scale trades off constraint satisfaction and generation quality in Diffinity . Figure 2(a) shows how varying the guidance affects satisfaction rates, perplexity, and fluency. As expected, increasing the guidance scale monotonically improves constraint satisfaction. The effect is particularly pronounced for JSON, where satisfaction rises from 30.4\% at scale 1.0 to 68.4\% at scale 2.5 . We conjecture that the relatively large increase in satisfaction rate for JSON is because well-formed JSON is relatively further from the training distribution of the base PLAID model: at low guidance scales the diffusion trajectory frequently drifts into syntactically invalid regions, whereas stronger guidance anchors the reverse process to the JSON schema. Surprisingly, stronger guidance does not degrade language quality. On the contrary, perplexity measured with Llama-3.1-8B decreases substantially, from 73.6 at scale 1.0 to 60.0 at scale 2.5 . Fluency exhibits only a mild drop (from 39.4 to 35.8 ), indicating that the improvement in structural correctness is not achieved by sacrificing readability. We hypothesize that this counterintuitive perplexity improvement arises because higher guidance suppresses unstable denoising trajectories that violate the regex early and subsequently require large updates. By constraining the diffusion path to remain close to the target regular language, Diffinity produces sequences that are both syntactically valid and more predictable under an external language model. 4.4 Comparison with PLAID Generation quality. PLAID provides native guidance mechanisms that enable token-level control over simple syntactic properties, such as increasing the probability that a specific token appears at a given position, by injecting gradients of token log-probabilities during the denoising process. However, these mechanisms do not extend to structured constraints such as regular expressions. In particular, PLAID operates at the tokenizer level and is therefore incompatible with regex constraints due to the automaton–token alignment problem discussed in Section 3.1 . If we temporarily ignore this limitation, PLAID can express a strict subset of our natural-language benchmarks—namely the Prefix , Suffix , and Appearance categories. We therefore compare PLAID and Diffinity on this common subset, using a guidance scale of 2.5 for both methods. PLAID achieves an average satisfaction rate of 59.4\% on these benchmarks, whereas Diffinity reaches 92.9\% , demonstrating that automaton-level guidance is substantially more effective than token-level control even when both are applied at comparable strength. Figure 2(b) reports a scatter plot of the minimum-perplexity samples obtained for benchmarks that both PLAID and Diffinity could solve. The scatter plot shows that Diffinity is advantageous in generating lower-perplexity samples, where Diffinity ’s minimum perplexity is on average 10.4 compared to 25.7 for PLAID. PLAID does have a slightly lower average perplexity for all generated samples that meet the constraint, at 55.65 compared to 60.27 for Diffinity ; we attribute this difference to the fact that Diffinity has a much higher constraint satisfaction rate. Computational overhead. We next examine the computational overhead introduced by Diffinity . Figure 2(c) plots per-sample generation latency for 14 representative benchmarks, across batch sizes of 1 , 4 , and 16 , as a function of the number of transitions in the aligned automaton. Compared to unconditional generation, Diffinity incurs substantial overhead: the smallest benchmark shown contains 71{,}735 transitions and requires 145.6 seconds per sample, versus 33.3 seconds for unconditional generation in PLAID, and 55.9 seconds for conditional generation. This overhead scales approximately linearly with the number of transitions, caused by Line 6 of Algorithm 1 , which computes a dense sum over all automaton transitions. Although this computation is parallelized on the GPU, we call torch.autograd to compute the gradient of Algorithm 1 , restricting further optimization opportunities, and the large number of transitions also exterts significant pressure on memory bandwidth. As a result, performance gains from batching quickly saturate, which explains the relatively small per-sample latency difference between batch sizes 4 and 16 in Figure 2(c) . PLAID, on the other hand, displays constant latency across regexes of different size because it does not have to compute a gradient over an automaton. Our primary goal in this work is to establish that diffusion models can reliably obey formal syntactic constraints—often more robustly than autoregressive approaches, as reflected in our perplexity and fluency results. Designing memory-efficient data structures and custom gradient kernels for Algorithm 1 is a promising direction for future work, and we expect optimizations like those used in constrained decoding ( 21 ) to reduce the overhead of Diffinity by one to two orders of magnitude. 5 Related Work Diffusion models, including DDPMs ( 11 ) and Variational Diffusion Models ( 14 ) , were originally developed for continuous data and have recently been adapted to text using both continuous and discrete paradigms. Continuous variants 16 ; 10 ; 12 perform diffusion in a continuous latent space that is decoded to text; for example, PLAID diffuses in a 32-D word2vec space and decodes to a unigram distribution over sentences. Discrete variants operate directly on token sequences by defining the forward noise-adding process as a probabilistic transition matrix ( 17 ) . Masking diffusion models (D3PM ( 3 ) , Dream ( 29 ) , LLaDA ( 19 ) ) introduce a special mask token and corrupt sequences by replacing tokens with masks, while others such as SEDD-Uniform ( 17 ) instead replace tokens with other vocabulary items. Our work focuses on continuous diffusion models (PLAID in particular), modifying the denoising equation for continuous diffusion models a là classifier guidance. Because classifier guidance in theory is applicable to discrete models as well, we expect to be able to apply our approach to discrete diffusion models by developing a variant of Algorithm 1 that efficiently computes the expected probability for latent spaces of discrete models. We leave this idea as an interesting venue for future work. The standard approach for steering diffusion models is classifier guidance ( 4 ; 24 ) , which relies on a separately trained classifier. Our method follows the same principle but replaces the classifier with Algorithm 1 , which computes the exact expected probability induced by a regular constraint. PLAID ( 10 ) also supports limited steering using token-level probability gradients (e.g., enforcing the presence or absence of specific tokens at fixed positions). Such methods cannot express global or sequential constraints (e.g., ‘Orange comes after Apple’), which require tracking dependencies across the full sequence. Several approaches support constrained decoding for discrete masking diffusion models. DINGO ( 25 ) enforces regular constraints, and 18 supports context-free constraints (which reduce to regular constraints for bounded length). These methods filter out unmaskings that must violate the constraint, but are limited to masking-based discrete models and distort the underlying model distribution. They do not apply to continuous diffusion models (our target) or mask-free discrete models such as SEDD ( 17 ) . Grammar-constrained decoding (GCD) ( 7 ) and its many implementations ( 9 ; 21 ; 27 ; 28 ) enforce syntactic constraints for autoregressive models by filtering out tokens that are guaranteed to eventually result in a string that violates the given grammar. GCD distorts the model distribution and can lead to the infinite repetition trap for complex grammars ( 20 ) . While recent sampling methods reduce this distortion ( 20 ; 22 ; 1 ) , their benefits mainly arise when drawing hundreds of samples for the same task. As demonstrated in our evaluation, Diffinity targets the desired conditional distribution and does not suffer from the infinite repetition trap. 6 Conclusion We presented Diffinity , a method for steering continuous diffusion language models to satisfy formal syntactic constraints specified by a finite automaton (or a regex). Diffinity guides diffusion sampling using gradients of the exact automaton acceptance probability under the model’s current latent distribution, avoiding training a separate classifier. Using PLAID as the base model, we find that Diffinity is capable of successfully enforcing complex regular constraints while maintaining competitive generation quality. While Diffinity ’s current computational overhead is relatively high compared to other methods, we believe existing work on succinctly representing automata ( 21 ) , can lift to Diffinity ; a promising direction of future work. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. Appendix A Hardware and Software Our experiments were conducted on a Ubuntu 20.04.5 LTS server equipped with a AMD EPYC 7282 CPU (16-cores at 2.8GHz) and NVIDIA RTX A6000 GPUs. We ran all experiments using a single GPU. Our implementation is based on Python 3.10.19, PyTorch 2.0.1 with CUDA 11.8, and Flash Attention 1.0.9. Appendix B Model Checkpoint We use the following models in our evaluation: • PLAID : From the PLAID github repository https://github.com/igul222/plaid (commit e7d5cf8 ) • GPT2 : 23 with specific models being: – GPT2-Small : https://huggingface.co/openai-community/gpt2 (commit 607a30d ) – GPT2-Medium : https://huggingface.co/openai-community/gpt2-medium (commit 6dcaa7a ) – GPT2-Large : https://huggingface.co/openai-community/gpt2-large (commit 32b71b1 ) • Llama-8B : Llama-3.1-8B ( 8 ) : https://huggingface.co/meta-llama/Llama-3.1-8B (commit d04e592 ) • Claude : Claude Sonnet 4.5-20250929. Appendix C Fluency Evaluation Prompt We use the following prompt when querying Claude Sonnet 4.5 for fluency, for the generated sample sample . """Read the text below. Then, indicate the fluency of the text, on a scale from 1 (poor fluency) to 100 (excellent fluency). In your assessment, consider factors such as grammatical correctness, naturalness of language, and overall smoothness. Text to evaluate: < Text > {sample} < /Text > On a scale from 1 (poor fluency) to 100 (excellent fluency), how fluent is this text? Respond in JSON format with two fields: - "score": an integer from 1 to 100 - "reasoning": a brief explanation (1-2 sentences) Example response: {{"score": 75, "reasoning": "The text is mostly fluent with natural phrasing, though there is one minor grammatical error."}} Your response (JSON only, no other text):""" Appendix D Vocabulary-DFA Alignment Algorithm Details We first formally define deterministic finite-state automata. Definition D.1 (Deterministic Finite-State Automaton) . A deterministic finite-state automaton (DFA) is defined as a 5-tuple A=(\Sigma,Q,\mathit{q_{0}},\delta,F) , where \Sigma is a finite set of inputs called the alphabet, Q is a finite set of states, \mathit{q_{0}}\in Q is the initial state, \delta is a transition function \delta:Q\times\Sigma\rightarrow Q , and F\subseteq Q is a set of final states. The extended transition function \hat{\delta} of A is defined as the function \hat{\delta}:Q\times\Sigma^{*}\rightarrow Q such that \hat{\delta}(q,\epsilon)=q for q\in Q and the empty string \epsilon , and \hat{\delta}(q,wc)=\delta(\hat{\delta}(q,w),c) where w\in\Sigma^{*} and c\in\Sigma . We say that a accepts a string x\in\Sigma^{*} iff \hat{\delta}(\mathit{q_{0}},x)\in F . Algorithm 2 Vocabulary-DFA Alignment \mathsf{Align}(A,V) Input: DFA A=(\Sigma,Q,q_{0},\delta,F) , vocabulary V 1: // Split A into char-level DFA A_{C} 2: (C,Q_{C},q_{C_{0}},\delta_{C},F_{C})\leftarrow\textsc{CharSplit}(A) 3: \delta_{V}\leftarrow\emptyset 4: for \mathit{tok}\in V do 5: // Split \mathit{tok} into char-level sequence c_{\mathit{tok}} 6: c_{\mathit{tok}}\leftarrow\textsc{Charify}(v) 7: for q_{C}\in Q_{C} do 8: // Traverse c_{\mathit{tok}} in DFA starting from q_{C} 9: q^{\prime}_{C}\leftarrow\textsc{Traverse}(c_{\mathit{tok}},q_{C},\delta_{C}) 10: if q^{\prime}_{C}\neq\mathsf{null} then 11: \delta_{V}\leftarrow\delta_{V}\cup\{(q_{C},\mathit{tok},q^{\prime}_{C})\} 12: C\leftarrow C\cup\mathit{tok} 13: end if 14: end for 15: end for 16: // Remove char-level transitions not in V 17: \delta_{V}\leftarrow\textsc{Remove}(\delta_{V},V) 18: return DFA A_{V}=(C,Q_{C},q_{C_{0}},\delta_{V},F_{C}) In Algorithm 2 , Charify is a function that takes as input a token v and splits it into characters, which is the smallest unit of strings that is common to both the automaton and the tokenizer. For example, a tokenizer might contain the word cat ; this token can be Charify ed into c, a, t . The idea is that the character-level automaton A_{C} will now be able to accept this character-level string, because all transitions inside the automaton have been factorized into character-level transitions. Traverse in Algorithm 2 performs this operation, checking for every state in the automaton there exists a path to another state that consumes the characters in c_{\mathit{tok}} . If there is, then there is a valid path inside the automaton for the token \mathit{tok} and we add the path as a transition. Theorem E.1 in Appendix E states the correctness of Algorithm 2 . Appendix E Proofs We provide simple proofs for Theorem E.1 and Theorem 3.1 . Theorem E.1 (Vocabulary Alignment) . Let \mathcal{L} be a regular constraint and A be the DFA representation of \mathcal{L} . Then \mathsf{Align}(A,V) as defined by Algorithm 2 returns a DFA A_{V} such that A_{V} accepts a sequence of tokens \mathit{tok}_{1},\cdots,\mathit{tok}_{l} iff \mathit{tok}_{i}\in V for all 1\leq i\leq l and the concatenation \mathsf{concat}(\mathit{tok}_{1},\cdots,\mathit{tok}_{l})\in\mathcal{L} . Proof. That A_{V} accepts \mathit{tok}_{1},\cdots,\mathit{tok}_{l} if the conditions hold follows by construction of A_{V} , as the character-level automaton A_{C} accepts \textsc{Charify}(\mathsf{concat}(\mathit{tok}_{1},\cdots,\mathit{tok}_{l})) . That A_{V} accepts \mathit{tok}_{1},\cdots,\mathit{tok}_{l} only if the condition holds follows from the fact that ( i ) if there exists \mathit{tok}_{i} such that \mathit{tok}_{i}\not\in V , then there cannot be a transition in A_{V} that consumes \mathit{tok}_{i} by construction; and ( ii ) if \mathsf{concat}(\mathit{tok}_{1},\cdots,\mathit{tok}_{l})\not\in\mathcal{L} , the Charify of this string is not accepted by A_{C} and thus cannot be accepted by A_{V} as well. ∎ Theorem 3.1 (Expected Probability) . Let A be a DFA that represents the regular language \mathcal{L} . Then Algorithm 1 returns the expected probability \mathds{E}_{x\sim\theta}[x\in\mathcal{L}] . Proof. By induction on the correctness of the intermediate state distributions \mathbf{p}_{i} . As the base case, \mathbf{p}_{0} clearly represents the correct distribution as the automaton always starts from the initial state. Assume that \mathbf{p}_{i} has the correct distribution. Then \mathbf{p}_{i+1} also has the correct distribution, if the transition matrix M_{i+1} is correct. By Theorem E.1 , we have that A accepts all possible tokenizations of a string x\in\mathcal{L} , which are summed in line 6 of Algorithm 1 , and thus M_{i+1} correctly captures the probability that A can make a transition between states when accepting any tokenization of a string x\in\mathcal{L} . It follows from induction that \mathbf{p}_{i} contains the correct distribution of states for every i , and thus that Algorithm 1 returns the expected probability. ∎ Appendix F Detailed Benchmark Description JSON schema benchmarks For JSONSchemaBench, we begin by sampling 10 schemas from each of the 10 benchmark subcategories ( GitHub-trivial , GitHub-easy , etc.), for a total of 100 schemas. We then convert each schema into a regular expression when possible. We exclude 10 schemas whose constraints are not expressible using regular languages (e.g., nested objects or unbounded lists), yielding 90 regexes. Our current implementation of tokenizer-aligned automata ( Section 3.1 ) is not yet optimized and can produce prohibitively large automata for certain patterns, particularly those with many wildcard transitions. To ensure tractable evaluation, we further exclude 20 regexes whose aligned automata exceed 230 states or 7.5M transitions. This results in a final set of 70 JSONSchemaBench-derived regexes. Developing a more compact representation for tokenizer-aligned automata (e.g., using symbolic automata ( 5 ) ) is an interesting direction for future work. Natural language benchmarks The 110 natural-language regular expressions are generated synthetically using a fixed vocabulary and a small set of template patterns. We first take the 100 most frequent English words according to ( 26 ) . To generate a benchmark regex in each of the following templates, we uniformly sample one or two words uniformly from this list (denoted WORD , or WORD1 / WORD2 when two words are required) and a value of n that satisfies the constraint: Prefix. WORD appears exactly at the n -th position from the beginning of the sentence, for 1\leq n\leq 5 . Suffix. WORD appears exactly at the n -th position from the end of the sentence, for 1\leq n\leq 3 . Appearance. Both WORD1 and WORD2 appear somewhere in the sentence, in any order. Between- n . WORD1 appears before WORD2 , with exactly n words in between, for 1\leq n\leq 3 . Between (unbounded). WORD1 appears before WORD2 , with an arbitrary number of intervening words. Word-length. The sentence contains at least one word of exactly n characters, for 1\leq n\leq 10 . From the first five template categories, we sample 20 regexes per category. We additionally include the 10 word-length regexes (one for each length n\in\{1,\dots,10\} ), resulting in 110 natural-language benchmarks in total. For evaluation, we generate 20 samples for each natural-language benchmark and 10 samples for each JSON schema benchmark, mainly because some JSON benchmarks result in very large regexes and aligned automata that consume significant amounts of compute when generating many samples. Appendix G Additional experiments and results Table 3 and Table 4 display the summarized results for the natural language benchmarks, for which we compute constraint satisfaction rate, pass@10 rate, average perplexity (geometric mean over all matching samples), and average fluency (arithmetic mean over all matching samples). Table 3 : Comparison of constraint satisfaction rates (Sat.) and pass@10 rates across natural language benchmarks. 20 samples are generated for each natural language regex, so the satisfaction rate might be non-zero even if pass@10 is zero (as in GPT2-Small-GCD for Between (ubd.)). Diffinity consistently outperforms constrained decoding with GPT2 variants. Rates for Diffinity are reported with guidance scale 2.5 . Diffinity GPT2-Small-GCD GPT2-Medium-GCD GPT2-Large-GCD Benchmark Sat. Pass@10 Sat. Pass@10 Sat. Pass@10 Sat. Pass@10 Prefix 95.7 100.0 42.8 95.0 42.8 95.0 43.5 95.0 Suffix 96.8 100.0 19.5 60.0 17.0 70.0 17.5 55.0 Appearance 92.5 100.0 1.8 5.0 2.0 5.0 3.2 10.0 Between- n 85.5 95.0 0.0 0.0 0.2 0.0 0.5 10.0 Between (ubd.) 93.8 100.0 1.5 0.0 0.2 5.0 3.5 10.0 Word Length 95.0 100.0 88.0 100.0 90.5 100.0 89.5 100.0 Table 4 : Comparison of perplexity (PPL) and fluency (Flu., on scale of 0 to 100 ) across benchmarks. Perplexity and fluency are only computed for samples that successfully meet the regex; non-matching samples are excluded. Rates for Diffinity are reported with guidance scale 2.5 . Lower perplexity ( \downarrow ) and higher fluency ( \uparrow ) indicate better distribution alignment. Diffinity GPT2-Small-GCD GPT2-Medium-GCD GPT2-Large-GCD Benchmark PPL \downarrow Flu. \uparrow PPL \downarrow Flu. \uparrow PPL \downarrow Flu. \uparrow PPL \downarrow Flu. \uparrow Prefix 66.5 32.2 505.1 22.9 539.9 25.3 401.7 30.9 Suffix 59.3 36.8 123.8 21.5 115.4 27.2 70.3 31.9 Appearance 58.5 34.7 124.3 20.7 133.8 33.8 69.7 40.9 Between- n 58.7 33.3 – – 1946.8 5.0 68.1 35.0 Between (ubd.) 57.4 33.3 123.3 22.2 132.6 33.3 71.2 47.1 Word Length 60.7 43.3 312.9 28.8 169.4 40.2 125.0 44.2 G.1 Additional experiment showing that Diffinity preserves the underlying probability distribution As an additional experiment that Diffinity preserves the underlying distribution of the diffusion model, we test Diffinity and GPT2-Small-GCD on the following three regexes: • (The [A-Za-z .,;:"?!()-]+.|It [A-Za-z .,;:"?!()-]+.) • (The first [A-Za-z .,;:"?!()-]+.|It [A-Za-z .,;:"?!()-]+.) • (The first rule of Fight Club is [A-Za-z .,;:"?!()-]+.|It [A-Za-z .,;:"?!()-]+.) Here, the LLM is forced to select between a sentence that starts with The and a sentence that starts with It , with sentences that start with The getting increasingly longer prefixes. The idea is that the probability distribution between sentences that start with The and those that starts with It should be roughly equal to the following ratios: • Probability of a sentence starting with The : Probability of a sentence starting with It • Probability of a sentence starting with The first : Probability of a sentence starting with It • Probability of a sentence starting with The first rule of fight club is : Probability of a sentence starting with It Following Llama3.1-8B, the ratios of these probabilities are roughly 10.10 , 0.11 , and 1.26*10^{-6} . Diffinity , when generating 200 samples from each regex, generates a roughly matching distribution of sentences that start with The and It : 169:30 (ratio 5.63 ) for the first regex, 5:176 (ratio 0.03 ) for the second, and 0:180 (ratio 0 ) for the third (for the matching regexes). The ratio of sentences starting with The decreases as the prefix grows larger, following the distribution. On the other hand, GPT2-Small-GCD fails to generate a distribution that adapts to the relative likelihood of the sentence prefixes: it generates sentences with a ‘Starting with The ’:‘Starting with It ’ ratio of 110:90 (ratio 1.2 ) for the first regex, 126:74 (ratio 1.7 ) for the second, and 112:88 (ratio 1.27 ) for the third. This illustrates how autoregressive constrained decoding distorts the underlying distribution: the ratios remain largely similar, dominated by the probability of the first token in the sequence.