Data-Dependent Smoothing for Protein Discovery with Walk-Jump Sampling

Srinivas Anumasa, Barath Chandran.C, Tingting Chen, Dianbo Liu · 2025

Read Data-Dependent Smoothing for Protein Discovery with Walk-Jump Sampling by Srinivas Anumasa, Barath Chandran.C, Tingting Chen, Dianbo Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02069.txt.

Introduction

Srinivas Anumasa, Barath Chandran C, Tingting Chen, and Dianbo Liu add data dependent smoothing to diffusion based protein generation. Diffusion models create samples by reversing a noising process step by step. Protein data are sparse and uneven, with a few dense clusters and vast thin regions. Standard methods ignore that unevenness when picking noise scales.

Their walk-jump framework first estimates density around each data point to set its own noise scale. A score model then trains with those per point scales baked in. Local geometry thus shapes the denoising process directly. Evaluations show consistent gains across multiple metrics over uniform smoothing. The lesson is that data aware noise prediction matters for sparse, high dimensional generation.

Catalog record

CREATORSrinivas Anumasa, Barath Chandran.C, Tingting Chen, Dianbo Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-02069

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Data-Dependent Smoothing for Protein Discovery with Walk-Jump Sampling Srinivas Anumasa, Barath Chandran.C, Tingting Chen, Dianbo Liu Abstract Diffusion models have emerged as a powerful class of generative models by learning to iteratively reverse the noising process. Their ability to generate high-quality samples has extended beyond high-dimensional image data to other complex domains such as proteins, where data distributions are typically sparse and unevenly spread. Importantly, the sparsity itself is uneven. Empirically, we observed that while a small fraction of samples lie in dense clusters, the majority occupy regions of varying sparsity across the data space. Existing approaches largely ignore this data-dependent variability. In this work, we introduce a Data-Dependent Smoothing Walk-Jump framework that employs kernel density estimation (KDE) as a preprocessing step to estimate the noise scale σ \sigma for each data point, followed by training a score model with these data-dependent σ \sigma values. By incorporating local data geometry into the denoising process, our method accounts for the heterogeneous distribution of protein data. Empirical evaluations demonstrate that our approach yields consistent improvements across multiple metrics, highlighting the importance of data-aware sigma prediction for generative modeling in sparse, high-dimensional settings. 1 Introduction (a) (b) (c) (d) Figure 1 : Illustration of different contour behaviours: (a) small \sigma=0.4 oversharpens, (b) large \sigma=1.5 oversmooths, (c) With Potential Optimal \sigma=0.8 (d) Our adaptive \sigma mapping. Sampling in high-dimensional spaces is widely regarded as a challenging task due to the curse of dimensionality bishop2006pattern . Recently, diffusion models have emerged as a powerful class of generative models capable of producing high-quality samples across diverse domains, including images ho2020denoising ; dhariwal2021diffusion , videos ho2022video , graphs jo2022score , and biological sequences frey2023protein . The central training objective of diffusion models is to learn the score function, the gradient of the log-probability density function (PDF) using neural networks song2020sliced . Usually, the training relies on estimating the score of noisy distributions, obtained by convolving the data distribution with known noise distributions (typically Gaussian). The neural network thus learns to predict score values by denoising perturbed samples. At inference time, starting from pure noise, the learned denoiser iteratively refines samples through a reverse diffusion process, ultimately recovering clean data samples that resemble the training distribution. Protein discovery is both highly valuable and particularly challenging for generative models, largely due to the extreme sparsity of the protein sequence distribution romero2009exploring . In this work, we focus on the task of antibody protein sequence generation. An antibody sequence of length L is defined as a valid combination over an alphabet of 20 amino acids, implying a vast combinatorial search space of size 20^{L} . Identifying valid sequences within this exponentially large space is a formidable challenge. Existing autoregressive models struggle to generate valid protein sequences, suffering from error accumulation across long sequences as well as high inference latency. Latent diffusion models romero2009exploring , as discrete generalizations of variational diffusion, alleviate some of these issues but remain memory-intensive and slow at inference frey2023protein . More recently, the Discrete Walk-Jump Sampling (dWJS) frey2023protein method, grounded in the theory of Neural Empirical Bayes saremi2019neural , has been proposed as a memory-efficient alternative that also reduces inference latency. The dWJS framework is conceptually simple: it trains a score model at a single noise level ( \sigma ), optimized through denoising, and uses it during inference to guide samples in continuous space (the “walk”), before projecting them back to the discrete domain via a “jump” operation. While effective and efficient, the performance of dWJS is highly sensitive to the choice of \sigma . A large \sigma oversmooths the data manifold, while a small \sigma leads to sharp, fragmented regions, both of which might degrade performance. Careful tuning of this hyperparameter is therefore critical. Although existing models acknowledge the sparsity of protein data, we empirically observe that the degree of sparsity itself varies across the data space. Thus, a single global noise level may be insufficient to capture such heterogeneity. In this work, we propose a simple yet effective extension, Data-Dependent Scaling with Discrete Walk-Jump Sampling (DDS-dWJS) , which replaces the global \sigma with data-dependent noise levels estimated from the training data. The only additional complexity introduced to dWJS is the estimation of these local noise scales, after which the score model is trained in the same manner but with sample-specific \sigma values. Figure 1 provides an overview of the energy surface around data points under different choices of \sigma . While Figure 1 (a) and (b) illustrate the limitations of using a single global \sigma —either oversharpening or oversmoothing the landscape. Figure 1 (d) demonstrates that with adaptive, data-dependent \sigma\in\{0.2,1.5\} , the energy surface naturally varies across regions according to local density. Here, the local density at a point x is approximated by the inverse of its mean distance to the 4 nearest neighbors. This heterogeneity enables the sampler to more effectively explore the diverse regions (varying density). Our contributions are summarized as follows: • We empirically demonstrate that antibody protein sequences reside in regions of varying sparsity, motivating the need for data-dependent noise levels. • On a synthetic toy dataset mimicking protein sequence distributions, we show that DDS-dWJS successfully discovers the true underlying distribution by generating valid samples. • On real antibody protein sequences, we evaluate DDS-dWJS against standard dWJS and demonstrate that data-dependent scaling improves model performance. 2 Background Neural Empirical Bayes (NEB) saremi2019neural is an approach that combines non-parametric smoothing with empirical Bayes estimation to perform denoising in a least-squares sense. Let x denote a clean sample. A smoothed version is obtained by adding Gaussian noise with standard deviation \sigma (the noise level): y=x+\mathcal{N}(0,\sigma^{2}). (1) From the noise, the clean sample can then be estimated using the empirical Bayes formula \bar{x}=y+\sigma^{2}\nabla\log f(y), (2) where f(y) is the density of the smoothed samples. Although the density f(y) is generally unknown, its score function \nabla\log f(y) can be approximated using a neural network. The neural network is trained via a denoising objective, which encourages the model to map noisy samples back to their clean versions. Formally, the optimization problem is given by L(\phi)\;=\;\mathbb{E}_{x\sim p(x),\,y\sim p(y|x)}\,\big\|x-\hat{x}_{\phi}(y)\big\|^{2}, (3) where p(x) is the distribution of the data, p(y|x)=\mathcal{N}(x,\sigma^{2}) , and the function, \hat{x}_{\phi}(y)=y+\sigma^{2}g_{\phi}(y) with g_{\phi}(y) serving as the neural approximation of the score function \nabla\log f(y) . The dWJS method adapts this framework to the task of protein–antibody sequence generation. In their formulation, each amino acid sequence is represented as a concatenated one-hot encoding: a sequence of length 297 is mapped to a binary vector of size 297\times 21 . Starting from this discrete representation, each sample is projected into a continuous space by adding Gaussian noise. A neural network is then trained to approximate the score function by learning to denoise these noisy embeddings. Once the score model is trained, inference proceeds as follows: a random sequence is generated, converted into its concatenated one-hot vector form, and perturbed with Gaussian noise to project it into the continuous space. In this space, the sample is iteratively updated along the direction of the learned score function (the “walk” step), effectively following the energy surface. After T steps, the updated sample is projected back into the data space and discretized to yield a valid protein sequence (the “jump”

Read the complete text