Sampling out-of-distribution chemical spaces via Bayesian flow Authors: Nianze Tao, Minori Abe Published: 2026 ABSTRACT Generating novel molecules with higher properties than the training space, namely the out-of-distribution generation, is important for de novo drug design. However, it is not easy for distribution learning-based models, for example diffusion models, to solve this challenge as these methods are designed to fit the distribution of training data as close as possible. In this paper, we show that Bayesian flow network, especially ChemBFN model, is capable of intrinsically generating high quality out-of-distribution samples that meet several scenarios. A reinforcement learning strategy is added to the ChemBFN and a controllable ordinary differential equation solver-like generating process is employed that accelerate the sampling processes. Most importantly, we introduce a semi-autoregressive strategy during training and inference that enhances the model performance and surpass the state-of-the-art models. A theoretical analysis of out-of-distribution generation in ChemBFN with semi-autoregressive approach is included as well. Scientific contribution Benchmarked on both small molecule and protein generative tasks, ChemBFN method showed outstanding out-of-distribution performance without complex modification, proving its feasibility of exploring chemical spaces outside the training data and its value as a tool to accelerate drug design and materials discovery.Graphic Abstract Supplementary InformationThe online version contains supplementary material available at https://doi.org/10.1186/s13321-026-01248-9. Introduction The chemical space is large, and even the sub-space is too large to be explored completely. For instance, the number of small drug-like molecules is believed to be over \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$10^{60}$$\end{document}1060 [1–3], amongst which, however, only a much smaller sub-region, e.g., macrocycles [4–6], were tested in the laboratories and applied to real world challenges; when we start considering larger systems like proteins, the space grows exponentially [7]. Recent emerging deep generative models that virtually search the chemical space are attractive alternatives to trial-and-error processes conducted by human scientists [8–12]. Despite the success of generative models, especially the distributional learning models, in molecule generation, some researchers have pointed out the limitations of these methods: because the models and benchmarks were designed to optimise and test the in-distribution performance, i.e., how close the generated molecules are to the training data, (1) the models were poor at generating highly novel samples with desired properties [9, 13]; (2) multi-objective optimisation was difficult [9]; (3) the sampling space could change to the false-positive region volatilely when an overconfident guidance appeared [14]. To overcome these problems, it is important to improve the performance of out-of-distribution (OOD) generation, i.e., generating compounds with higher properties than that of molecules in the training dataset. Pioneering works have introduced a few methods to enhance the diffusion models’ OOD performance including introducing a dedicatedly designed control method to steer towards the OOD space [9], and utilising unlabelled data to expand the training domain and regularise the model [14]. In this research, we, on the other hand, show that the Bayesian flow network, another branch of deep generative methods, is a natural OOD sampler. Moreover, we demonstrate that by introducing a semi-autoregressive (SAR) behaviour our method outperformed several state-of-the-art (SOTA) models in OOD multi-objective optimisation tasks. Methods Bayesian flow networks Similar to denoising diffusion models [15–20] (DMs) and more general flow matching methods [21–29], Bayesian flow networks [30] (BFNs) splits the generative process into a sequential steps. Some research has pointed out that the generative process of BFN is similar to and thus can be approximated as a reversed stochastic differential equation (SDE) that used in DMs though [31–33], BFN method does not require to define a diffusion process nor learns the noise distribution: instead, BFN directly optimises the parameters of a distribution towards a more informative direction, which makes it applicable to continuous, discretised, and discrete data as the parameters of any real-world distribution is continuous [30]. The previous studies have already applied the idea of BFN to 3-dimensional molecular conformation generation [33, 34] (discretised for atom types and continuous cases for coordinates), text-based molecule generation [35] (discrete case for structural tokens), and protein sequence generation [32] (discrete case for amino acid types), which proved its capability of understanding the chemical space. In this research, we employ ChemBFN [35], a BFN model with encoder-only transformer [36] architecture handling the 1-dimensional molecular representation originally designed to generate SMILES [37] and SELFIES [38] strings, to study BFN’s potentials of OOD sampling, i.e., generating samples with high properties out of a low-property training space. ChemBFN introduces a new accuracy schedule β (t)β(t) for discrete BFN that reduces the reconstruction loss and improves the model performance [35]. However, limitations remained, e.g., to generate high quality samples, the number of sampling steps should be as large as 1000 in some cases [35] which limited the feasibility of generating large molecules; the distributional distance between the generated samples and the training data is large compared with baselines [35]. in the latter sections, we introduce our solution to solve the first limitation. We also realise that the second mentioned limitation can be transformed into an advantage in OOD generating. We show that a small change in the training or sampling process of ChemBFN can significantly enhance the OOD generative performance. Efficient sampling with BFN Although ChemBFN succeeded in generating diverse molecules, the ratio of valid SMILES was low when fewer sampling steps were used [35]. To reduce the sampling steps while retaining the high validity ratio, here we propose two methods. I. Auxiliary reinforcement learning term. For any parameters of categorical distribution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\theta }$$\end{document}θ fed into the neural network, we define the following reinforcement learning (RL) term, inspired by REINFORCE algorithm [39, 40], added into the training loss1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L^{RL} = \eta \mathbb {E}_{t\sim U(0,1)}\left( \boldsymbol{e}^{(k)}(\hat{\boldsymbol{\theta }};t)\cdot (1-\delta _{c}(\boldsymbol{e}(\hat{\boldsymbol{\theta }};t)))\right) , \end{aligned}$$\end{document}LRL=ηEt∼U(0,1)e(k)(θ^;t)·(1-δc(e(θ^;t))),where ηη is a scaling constant, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k=argmax(\boldsymbol{e}(\hat{\boldsymbol{\theta }};t))_{dim=-1}$$\end{document}k=argmax(e(θ^;t))dim=-1, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{e}(\hat{\boldsymbol{\theta }};t)$$\end{document}e(θ^;t) is the estimated categorical distribution (i.e., output distribution in terms of BFN) at time t via the neural network, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\delta _{c}(.) = 1$$\end{document}δc(.)=1 when criterion c is satisfied but 0 elsewhere. To increase valid ratio of generated molecules, a naïve strategy is to raise the possibilities that at any time step t the output distributions correspond to valid molecules. Therefore, we set the criterion c as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c {:=} \{\boldsymbol{e}(\hat{\boldsymbol{\theta }};t)\mathrm {~corresponds~to~a~valid~molecule}\}$$\end{document}c:={e(θ^;t)correspondstoavalidmolecule}. In practice, we found that η =0.01η=0.01 was optimal in most cases whereas η <0.001η<0.001 or η >0.1η>0.1 led to degraded performance. II. ODE-like generating process. The linkage between SDE and BFN generating process was first discovered by Xue et al. [31], who stated that by directly operating in the latent space \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Omega _{z}$$\end{document}Ωz instead of the distribution parameter space \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Omega _{\theta }$$\end{document}Ωθ, an ordinary differential equation (ODE) solver or an SDE solver that updated the latent variable \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{z}_{i}=\boldsymbol{z}_{i-1}+\alpha _{i}(K\boldsymbol{e}(\hat{\boldsymbol{\theta }}_{i-1};t_{i-1})-1)+\sqrt{K\alpha _{i}}\boldsymbol{\epsilon }$$\end{document}zi=zi-1+αi(Ke(θ^i-1;ti-1)-1)+Kαiϵ (where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{z}\in \Omega _{z}$$\end{document}z∈Ωz, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\theta }\in \Omega _{\theta }$$\end{document}θ∈Ωθ, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\epsilon }\sim \mathcal {N}(\boldsymbol{0},\boldsymbol{I})$$\end{document}ϵ∼N(0,I), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{i}=\beta (t_{i})-\beta (t_{i-1})$$\end{document}αi=β(ti)-β(ti-1), β (t)β(t) is a monotonic accuracy schedule [30, 35], and K is the number of categories) could accelerate the generating process. A simplified alternative method proposed by Atkinson et al. [32] and Qu et al. [33] reduced the complexity of the algorithm, based on which we found that scaling the randomness by a temperature coefficient τ >0τ>0 further benefited the valid ratio of generated objects. Our ODE-like sampling algorithm is shown in Algorithm 1. In the later text, we show that the combination of these two methods significantly speeds up the molecular generation. Algorithm 1ODE-like sampling algorithm ODE-like sampling algorithm SAR training and sampling In the original ChemBFN [35] model and any BERT [41]-like model, tokens are updated bidirectionally as illustrated in Fig. 1 (left), where for an arbitrary token in a finite fixed-length sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{j}\in (s_{0}, s_{1}, s_{2},..., s_{N})$$\end{document}sj∈(s0,s1,s2,...,sN) a function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{BI}$$\end{document}fBI maps \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{j}$$\end{document}sj from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i^{th}$$\end{document}ith layer to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(i+1)^{th}$$\end{document}(i+1)th layer based on itself and tokens from both left side (previous tokens) and right side (subsequent tokens), i.e., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s^{i+1}_{j}=f_{BI}(s^{i}_{j};s^{i}_{0:j-1},s^{i}_{j+1:N})$$\end{document}sji+1=fBI(sji;s0:j-1i,sj+1:Ni). In the case of autoregressive models, however, since the sequence is extended step by step (as shown in Fig. 1 (middle)), the next token \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{j+1}$$\end{document}sj+1 only comes from the previous tokens via a function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{AR}$$\end{document}fAR, i.e., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s^{i+1}_{j+1}=f_{AR}(s^{i}_{0:j})$$\end{document}sj+1i+1=fAR(s0:ji). For models employing self-attention mechanism, e.g., the decoder of autoencoder transformer [36] model and GPT [42] model, a causal mask that maps entities of the attention matrix above the main diagonal to zero is applied to implement this autoregressive behaviour. We found that in a trained ChemBFN model, the entries that were far away from the main diagonal in the attention matrices were extremely close to zero, which enlightened the possibility of applying causal masks to ChemBFN models without breaking models’ generative capability. By employing the causal mask, we introduce a SAR behaviour to ChemBFN models, in which all tokens are updated together as a block but subsequent tokens are not used to update the current token (Fig. 1 (right)), i.e., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s^{i+1}_{j}=f_{SAR}(s^{i}_{j};s^{i}_{0:j-1})$$\end{document}sji+1=fSAR(sji;s0:j-1i). To be simple, we denote applying causal masks to models as ‘SAR’ while ‘normal’ stands for disenabling causal masks. Depending on whether the causal masks are used in training or sampling processes, four strategies (strategy 1–4) are proposed in Table 1 and a workflow is in Fig. 2. An overview of integration of above mentioned methods are shown in Fig. 3. We show how these settings affect the generative behaviours of our model in later texts.Fig. 1Visualisation of bidirectional, autoregressive, and SAR token update methods. Note that to make the equations clear, we use t+1t+1, i+1i+1, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{i+1}$$\end{document}ti+1 in a interchangeable way in the main text when the discretised time is discussedFig. 2The flowchart showing that whether SAR is applied or not based on selected strategies listed in Table 1. ‘Switch on SAR’ refers to applying causal masks to attention mechanism and ‘switch off SAR’ refers to applying no causal masks. The guidance vector can be a null vector ϕϕ as described in the original paper of ChemBFN [35]Fig. 3The overview schematic of integrating SAR, online RL, and ODE-like solver into ChemBFN model. Purple-highlighted parts are newly added methods in this work. The dashed path is the loop involved in inference process. The solid path leading to ‘KL loss + RL’ strands for the training processTable 1Training and sampling strategies\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{a }$$\end{document}aStrategyTrainingSampling1NormalNormal2NormalSAR3SARNormal4SARSAR\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{a}$$\end{document}aSAR is ‘semi-autoregressive’ Visualisation of bidirectional, autoregressive, and SAR token update methods. Note that to make the equations clear, we use t+1t+1, i+1i+1, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{i+1}$$\end{document}ti+1 in a interchangeable way in the main text when the discretised time is discussed The flowchart showing that whether SAR is applied or not based on selected strategies listed in Table 1. ‘Switch on SAR’ refers to applying causal masks to attention mechanism and ‘switch off SAR’ refers to applying no causal masks. The guidance vector can be a null vector ϕϕ as described in the original paper of ChemBFN [35] The overview schematic of integrating SAR, online RL, and ODE-like solver into ChemBFN model. Purple-highlighted parts are newly added methods in this work. The dashed path is the loop involved in inference process. The solid path leading to ‘KL loss + RL’ strands for the training process Table 1Training and sampling strategies\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{a }$$\end{document}aStrategyTrainingSampling1NormalNormal2NormalSAR3SARNormal4SARSAR\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{a}$$\end{document}aSAR is ‘semi-autoregressive’ Training and sampling strategies\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{a }$$\end{document}a \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{a}$$\end{document}aSAR is ‘semi-autoregressive’ Datasets and benchmarks MOSES [43] as well as GuacaMol [44] are widely-used benchmarks in the studies of small molecule generation. Apart from testing the validity, uniqueness, diversity, and novelty of generated molecules, MOSES benchmark focuses more on the distribution learning performances of tested models, including Tanimoto similarity (SNN), fragment similarity (Frag), scaffold similarity (Scaf), and Fréchet ChemNet Distance [45] (FCD). Although the purpose of MOSES is to estimate how close the learnt distribution is to the training space, in this study it was utilised to showcase how far away the generated space of our method could be from the training space while retaining the chemical meaningfulness. Lee et al. [9], on the other hand, proposed ZINC250k dataset, which consists of 249,455 molecules collected from ZINC [46] database and their paired quantitative estimate of drug-likeness [47] (QED), synthetic accessibility [48] (SA), and docking scores (DS) unit in kcal/mol to five different proteins (PARP1, FA7, 5HT1B, BRAF, and JAK2) calculated via QuickVina 2 [49], and the corresponding metrics to test the OOD multi-objective guided molecular generating performance. In this benchmark, a group of filters that should be applied to generated molecules are defined as2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left\{ \begin{array}{lcl} {\text {QED}} & {>} & {0.5}\\ {\text {SA}} & {<} & {5}\\ {\text {DS}} & {<} & \widetilde{\textrm{DS}}(\text {molecules in training data})\\ {\text {SNN}} & {<} & {0.4},\\ \end{array}\right. \end{aligned}$$\end{document}QED>0.5SA<5DS