Supervised Score-Based Modeling by Gradient Boosting
Changyuan Zhao, Hongyang Du, Guangyuan Liu, Dusit Niyato · 2024
Read Supervised Score-Based Modeling by Gradient Boosting by Changyuan Zhao, Hongyang Du, Guangyuan Liu, Dusit Niyato (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01159.txt.
Description
Abstract Score-based generative models can effectively learn the distribution of data by estimating the gradient of the distribution. Due to the multi-step denoising characteristic, researchers have recently considered combining score-based generative models with the gradient boosting algorithm, a multi-step supervised learning algorithm, to solve supervised learning tasks.
Catalog record
| CREATOR | Changyuan Zhao, Hongyang Du, Guangyuan Liu, Dusit Niyato |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-01159
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Supervised Score-Based Modeling by Gradient Boosting Changyuan Zhao, Hongyang Du, Guangyuan Liu, Dusit Niyato Abstract Score-based generative models can effectively learn the distribution of data by estimating the gradient of the distribution. Due to the multi-step denoising characteristic, researchers have recently considered combining score-based generative models with the gradient boosting algorithm, a multi-step supervised learning algorithm, to solve supervised learning tasks. However, existing generative model algorithms are often limited by the stochastic nature of the models and the long inference time, impacting prediction performances. Therefore, we propose a Supervised Score-based Model (SSM), which can be viewed as a gradient boosting algorithm combining score matching. We provide a theoretical analysis of learning and sampling for SSM to balance inference time and prediction accuracy. Via the ablation experiment in selected examples, we demonstrate the outstanding performances of the proposed techniques. Additionally, we compare our model with other probabilistic models, including Natural Gradient Boosting (NGboost), Classification and Regression Diffusion Models (CARD), Diffusion Boosted Trees (DBT), and non-probabilistic GBM models. The experimental results show that our model outperforms existing models in both accuracy and inference time. Introduction Generative artificial intelligence (GAI) models aim to create highly realistic simulations or reproductions of data by learning from the characteristics of the original data. Recently, score-based models (diffusion models) have become the most powerful families of GAI due to their excellent generation performance ( Yang et al. 2023 ; Dhariwal and Nichol 2021 ) . In these models, a series of noises is first added to the original data through a forward diffusion process. The model then learns the added noise and continuously denoises it to generate new data that follows the distribution of the original dataset. At present, many researchers have applied score-based models to generation tasks in many fields, including image generation, audio generation, video generation, etc., all with excellent performance ( Rombach et al. 2021 ; Kong et al. 2020 ; Ho et al. 2022 ) . Supervised learning is another important research area in machine learning different from generative learning. In supervised learning, a model is trained on labeled data with the goal of learning a mapping from inputs to outputs, allowing it to predict the output for unseen data accurately ( Cunningham, Cord, and Delany 2008 ) . Regression and classification problems are two basic types of supervised learning tasks, which aim to train a model for prediction. For regression problems, this prediction result is the target output corresponding to the input data, while for classification problems, it is the accurate label. The learning performance of this data relationship is closely related to the learning ability of the sophisticated structure of neural network models ( He et al. 2016 ) . Particularly, the Gradient Boosting Machine (GBM) is a supervised learning model that combines several weak learners into a strong learner ( Bentéjac, Csörgő, and Martínez-Muñoz 2021 ) . The sequential weak learners reduce errors from previous iterations to obtain accurate predictions, especially with large and complex datasets. Recently, GBM algorithms such as XGBoost ( Chen and Guestrin 2016 ) and LightGBM ( Ke et al. 2017 ) have demonstrated state-of-the-art performance in tabular competitions on platforms, including Kaggle. Due to the multi-step denoising characteristic, researchers have recently considered combining score-based generative models with the gradient boosting algorithm to solve supervised learning tasks ( Han and Zhou 2024 ; Beltran-Velez et al. 2024 ) . However, these previous methods are mainly based on diffusion models, whose random denoising process may lead to unstable prediction results and reduce the robustness. Moreover, diffusion models often require longer denoising steps, increasing model inference time. If accelerated algorithms such as DDIM ( Song, Meng, and Ermon 2020 ) are used, the quality of the generated results will usually decrease, resulting in lower prediction accuracy. Therefore, in this paper, our goal is to propose a score-based gradient boosting model for supervised learning to enable quick and accurate inference. To achieve this goal, we consider the score-based generative models via denoising score matching ( Song and Ermon 2019 ) since its inference process does not have a specific number of denoising steps. Based on this, we propose Supervised Score-based Model (SSM) , a score-based model by gradient boosting. We summarize our main contributions as follows: 1) We utilize noise-free Langevin Dynamics to establish a connection between score-based generative models and GBM algorithms. 2) we present SSM to predict the maximum log-likelihood estimation point, which estimates errors between the target and input data through a score network. 3) For the training process and inference process, we provide a theoretical analysis to balance inference time and prediction accuracy. 4) Experiments on regression and classification tasks show that SSM can achieve better performance than existing methods and significantly shorten the inference time. Related Work Score-based generative models have demonstrated powerful learning capabilities for complex data distribution in many fields ( Croitoru et al. 2023 ; Du et al. 2023 ; Liu et al. 2024 ) . There are generally two categories of such models: diffusion-based and score matching-based models ( Yang et al. 2023 ) . Diffusion-based models consist of a forward diffusion process, which adds Gaussian noises into the original data, making it converge to the standard normal distribution, and a denoising process, which removes the added noises from a random variable following normal distribution ( Ho, Jain, and Abbeel 2020 ; Nichol and Dhariwal 2021 ) . The score matching-based model can be regarded as an energy-based model ( Song and Kingma 2021 ) , which aims to learn explicit probability distributions of data by a score function. Denoising score matching is a state-of-the-art approach for score estimation, which also estimates the score function from the disturbed dataset and then denoises through Langevin dynamics ( Song and Ermon 2019 ) . From a general perspective, these two types of models can also be viewed as the discretization of stochastic differential equations ( Song et al. 2020b ) . The GBM is a powerful ensemble learning technique that builds models sequentially, with each new model aiming to correct the errors of its predecessors ( Friedman 2001 ) . By combining the predictions of multiple weak learners, GBM creates a strong predictive model that can handle complex data patterns and interactions ( Natekin and Knoll 2013 ) . XGBoost, LightGBM, and NGBoost are typical decision trees-based gradient boosting techniques that extend the foundational principles of GBM to enhance efficiency and performance. XGBoost and LightGBM focus on computational speed and scalability ( Chen and Guestrin 2016 ; Ke et al. 2017 ) , while NGBoost introduces probabilistic predictions, offering a novel approach to uncertainty estimation in gradient boosting models ( Duan et al. 2020 ) . Recently, via setting input data in supervised learning as conditions, conditional score-based models have been utilized to solve supervised learning problems ( Amit et al. 2021 ; Rahman et al. 2023 ; Beltran-Velez et al. 2024 ) . ( Zimmermann et al. 2021 ) proposed a score-based generative classifier for classification tasks. This model predicts classification results through maximum likelihood estimation of different label generation distributions and target values instead of direct prediction. In ( Han, Zheng, and Zhou 2022 ) , the authors proposed CARD,