Course Project Report: Comparing MCMC and Variational Inference for Bayesian Probabilistic Matrix Factorization on the MovieLens Dataset
Ruixuan Xu, Xiangxiang Weng · 2025
Read Course Project Report: Comparing MCMC and Variational Inference for Bayesian Probabilistic Matrix Factorization on the MovieLens Dataset by Ruixuan Xu, Xiangxiang Weng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-09928.txt.
Introduction
Ruixuan Xu and Xiangxiang Weng compare two ways to fit a Bayesian recommendation model in a 2025 course project report. Matrix factorization predicts ratings by giving each user and film a hidden profile. The probabilistic form adds distributions over those profiles, so uncertainty comes along for the ride.
The catch is that the true posterior over profiles cannot be computed in closed form. Two approximation routes go head to head on the MovieLens dataset: Markov chain Monte Carlo, which samples its way to the answer, and variational inference, which fits a simpler stand-in distribution. The report carries full methods, experiments, references, and math derivations. Sampling gave the sharper posterior estimates, while the variational route converged faster, a classic speed-versus-fidelity trade-off the authors document with care.
Catalog record
| CREATOR | Ruixuan Xu, Xiangxiang Weng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-09928
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Course Project Report: Comparing MCMC and Variational Inference for Bayesian Probabilistic Matrix Factorization on the MovieLens Dataset Ruixuan Xu, Xiangxiang Weng Abstract This is a course project report with complete methodology, experiments, references and mathematical derivations. Matrix factorization [ 1 ] is a widely used technique in recommendation systems. Probabilistic Matrix Factorization (PMF) [ 2 ] extends traditional matrix factorization by incorporating probability distributions over latent factors, allowing for uncertainty quantification. However, computing the posterior distribution is intractable due to the high-dimensional integral. To address this, we employ two Bayesian inference methods: Markov Chain Monte Carlo (MCMC) [ 3 , 4 ] and Variational Inference (VI) [ 5 , 6 ] to approximate the posterior. We evaluate their performance on MovieLens dataset [ 7 ] and compare their convergence speed, predictive accuracy, and computational efficiency. Experimental results demonstrate that VI offers faster convergence, while MCMC provides more accurate posterior estimates. 1 Introduction Collaborative filtering [ 8 ] is an essential technique in recommendation systems, where the goal is to predict user preferences based on sparse observed ratings. Matrix factorization [ 1 ] has been widely adopted due to its ability to model user-item interactions effectively. However, traditional matrix factorization relies on point estimates, which may lead to overfitting and lack of uncertainty quantification. Probabilistic Matrix Factorization (PMF) [ 2 ] mitigates this issue by considering a probabilistic approach, where user and item latent matrices are treated as random variables with prior distributions. The challenge in PMF is computing the posterior distribution of latent matrices, which is intractable. To approximate the posterior, we explore two Bayesian inference methods: 1. Markov Chain Monte Carlo (MCMC) [ 3 , 4 ] : A sampling-based approach that provides asymptotically exact posterior estimates. 2. Variational Inference (VI) [ 5 , 6 ] : An optimization-based approach that approximates the posterior using a parameterized distribution. In this report, we implement both methods on MovieLens dataset [ 7 ] and compare their performance. 2 Problem Setting 2.1 Matrix Factorization Model In mathematics, a sparse matrix refers to a matrix in which most of the elements are zero. A sparse rating matrix is a common data structure in recommendation systems, typically used to represent users’ rating data for items. Collaborative filtering [ 8 ] relies on user behavior data such as ratings to predict items a user might like. Due to the high sparsity of rating matrices, directly storing and computing them would waste a lot of memory and computational resources. Matrix factorization [ 1 ] addresses this by decomposing the user-item rating matrix into two low-dimensional matrices, and then predicting ratings via their dot product, thereby achieving collaborative filtering. Given an N\times M sparse rating matrix R . Suppose there are L observed ratings ( L\ll N\times M ), for predicting all unrated items, we need to estimate N\times M-L\approx N\times M parameters. An N\times M rating matrix R can be approximated by the multiplication of an N\times K matrix and a K\times M matrix. Here, K is feature dimension, U\in\mathbb{R}^{N\times K} is user feature matrix, and V\in\mathbb{R}^{M\times K} is item feature matrix. With matrix factorization, rather than estimating N\times M-L\approx N\times M parameters, we only need to estimate (N+M)\times K , where K\ll\min(N,M) . Our goal is to find matrices U and V such that UV^{T} can accurately represent the original rating matrix. In this way, the number of parameters to be estimated is greatly reduced, and the entire prediction matrix can be obtained simply by matrix multiplication, making computation efficient. Finding the target parameter matrices U and V involves optimization methods based on gradient descent. For example, minimizing the following loss function: \min\limits_{U,V}\sum\limits_{(i,j)\in O}(r_{ij}-\mathbb{u}_{i}\mathbb{v}_{j}^{T})^{2} where: - O is the set of all rated items. - r_{ij} is the true rating given by user i to item j . - \mathbb{u}_{i}\mathbb{v}_{j}^{T} is the predicted rating. Gradient descent update rules: \displaystyle\mathbb{u}_{i} \displaystyle\leftarrow\mathbb{u}_{i}+\alpha\cdot\sum\limits_{j\in O_{i}}(r_{ij}-\mathbb{u}_{i}\mathbb{v}_{j}^{T})\mathbb{v}_{j} \displaystyle\mathbb{v}_{j} \displaystyle\leftarrow\mathbb{v}_{j}+\alpha\cdot\sum\limits_{i\in O_{j}}(r_{ij}-\mathbb{u}_{i}\mathbb{v}_{j}^{T})\mathbb{u}_{i} where \alpha is the learning rate. However, traditional matrix factorization has many limitations, such as poor generalization ability, inability to capture the uncertainty of latent vectors and lack of probabilistic interpretation. Therefore, we adopt Bayesian Probabilistic Matrix Factorization (BPMF) [ 4 , 6 ] on the basis of matrix factorization. 2.2 Bayesian Probabilistic Matrix Factorization Instead of estimating matrices U and V to compute each rating r_{ij} based on deterministic approaches, one can use Bayesian inference. Consider each row of U and V , i.e., \mathbb{U}_{i} and \mathbb{V}_{j} , as a multivariate random variables. \mathbb{U}_{i} and \mathbb{V}_{j} are assumed to be standard normal random vectors. \displaystyle\begin{cases}U_{ik}\sim N(0,1)\quad\text{ for any }1\leq k\leq K\\ V_{ik}\sim N(0,1)\quad\text{ for any }1\leq k\leq K\end{cases} The prior distributions of \mathbb{U}_{i} and \mathbb{V}_{j} have the following PDFs \displaystyle\begin{cases}f_{\mathbb{U}_{i}}(\mathbb{u}_{i})=\frac{1}{(2\pi)^{\frac{K}{2}}}\exp\left(-\frac{1}{2}\mathbb{u}_{i}\mathbb{u}_{i}^{T}\right)\\ f_{\mathbb{V}_{j}}(\mathbb{v}_{j})=\frac{1}{(2\pi)^{\frac{K}{2}}}\exp\left(-\frac{1}{2}\mathbb{v}_{j}\mathbb{v}_{j}^{T}\right)\end{cases} The likelihood of each observed rating r_{ij} is defined as a normal distribution: R_{ij}|\mathbb{U}_{i},\mathbb{V}_{j}\sim N\left(\text{sigmoid}\left(\mathbb{u}_{i}\mathbb{v}_{j}^{T}\right),\sigma^{2}\right) Note: as sigmoid function returns a value into [0,1] , during training, we need to first normalize ratings via r_{ij}\leftarrow\frac{r_{ij}-1}{R-1} , where original ratings are defined in \{1,2,…,R\} , and \sigma^{2} is a hyperparameter. The PDF of likelihood is \displaystyle f_{R_{ij}|\mathbb{U}_{i},\mathbb{V}_{j}}(r_{ij}|\mathbb{u}_{i},\mathbb{v}_{j})=\frac{1}{\sqrt{2\pi\sigma^{2}}}\exp\left(-\frac{\left(r_{ij}-\text{sigmoid}\left(\mathbb{u}_{i}\mathbb{v}_{j}^{T}\right)\right)^{2}}{2\sigma^{2}}\right). Based on conditional independence and Bayes’ rule, the posterior of \mathbb{U} and \mathbb{V} can be inferred as \displaystyle f_{\mathbb{U},\mathbb{V}|\{\mathbb{R}_{ij}\}}(U,V|\{r_{ij}\}) \displaystyle\propto\ \displaystyle f_{\{\mathbb{R}_{ij}\}|\mathbb{U},\mathbb{V}}(\{r_{ij}\}|U,V)f_{\mathbb{U},\mathbb{V}}(U,V) \displaystyle\propto\ \displaystyle\prod\limits_{i=1}^{N}\prod\limits_{j=1}^{M}\left[f_{R_{ij}|\mathbb{U}_{i},\mathbb{V}_{j}}(r_{ij}|\mathbb{u}_{i},\mathbb{v}_{j})\right]^{I_{ij}}\exp\left(-\frac{1}{2}\mathbb{u}_{i}\mathbb{u}_{i}^{T}\right)\exp\left(-\frac{1}{2}\mathbb{v}_{j}\mathbb{v}_{j}^{T}\right). where I_{ij} is an indicator function that is equal to 1 if user i rated movie j , otherwise 0 . If the likelihood adopts a simple Gaussian distribution, minimizing the negative log of the posterior (loss function) combined with gradient descent to obtain the most likely U and V , and then directly multiplying them to generate the prediction matrix are still available, just as researchers do in PMF [ 2 ] . However, we introduce nonlinearity by applying a sigmoid function to the mean of the Gaussian likelihood distribution, in order to simulate the complex likelihoods that are likely to occur in real-world applications. In addition, a simple Gaussian likelihood may produce predicted ratings outside the valid range (e.g., from 1 to 5), while sigmoid can compress the predicted ratings into valid range. In this case, the gradient descent becomes infeasible. Therefore, we adopt Bayesian framework, retain posterior distributions of U and V , and optimizing the predictive distribution over unrated data. To make a prediction on an unobserved rating r_{ab} : \displaystyle f_{R_{ab}|\{R_{ij}\}}(r_{ab}|\{r_{ij}\}) \displaystyle=\ \displaystyle\int_{\mathbb{u}_{a}\in\mathbb{R}^{K}}\int_{\mathbb{v}_{b}\in\mathbb{R}^{K}}f_{R_{ab}|\mathbb{U}_{a},\mathbb{V}_{b}}(r_{ab}|\mathbb{u}_{a},\mathbb{v}_{b})f_{\mathbb{U}_{a},\mathbb{V}_{b}|\{R_{ij}\}}(\mathbb{u}_{a},\mathbb{v}_{b}|\{r_{ij}\})d\mathbb{u}_{a}d\mathbb{v}_{b} \displaystyle=\ \displaystyle\mathbb{E}_{\mathbb{U}_{a},\mathbb{V}_{b}|\{R_{ij}\}}\left[f_{R_{ab}|\mathbb{U}_{a},\mathbb{V}_{b}}(r_{ab}|\mathbb{u}_{a},\mathbb{v}_{b})\right]. In general, we can use MAP to generate a predicted rating in [0,1] : \displaystyle r_{ab}^{*}=\arg\max\limits_{r_{ab}}f_{R_{ab}|\{R_{ij}\}}(r_{ab}|\{r_{ij}\}). Then map back to the original rating using r=(R-1)r^{*}+1 . However, we still cannot compute f_{R_{ab}|\{R_{ij}\}}(r_{ab}|\{r_{ij}\}) , because there are two challenges: 1. Although we have derived the parameter posterior distribution f_{\mathbb{U}_{a},\mathbb{V}_{b}|\{R_{ij}\}}(\mathbb{u}_{a},\mathbb{v}_{b}|\{r_{ij}\}) , we cannot accurately compute the normalization constant, since integrating a high-dimensional Gaussian distribution with a sigmoid is difficult even for computers; 2. Even if we obtain the parameter posterior, the integral involved in the predictive distribution itself is high-dimensional and difficult to compute. To address this, we adopt some Bayesian inference methods. 3 Bayesian Inference Methods Consider Bayesian inference: \displaystyle f_{\Theta|X}(\theta|x)=\frac{f_{\Theta}(\theta)f_{X|\Theta}(x|\theta)}{Z(x)}. - f_{\Theta|X}(\theta|x) : Posterior. - f_{\Theta}(\theta) : Prior. - f_{X|\Theta}(x|\theta) : Likelihood. - Z(x) : Normalization constant, where Z(x)=\int_{\theta}f_{\Theta}(\theta)f_{X|\Theta}(x|\theta)d\theta . For multiple observations, let D=\{X_{1},\ldots,X_{n}\} be the joint random variables and d=\{x_{1},\ldots,x_{n}\} their values. Bayesian inference can be written as: \displaystyle f_{\Theta|D}(\theta|d) \displaystyle=\frac{f_{\Theta}(\theta)f_{D|\Theta}(d|\theta)}{Z(d)}. Bayesian prediction: \displaystyle f_{X|D}(x^{*}|d)=\int_{-\infty}^{+\infty}f_{X|\Theta}(x^{*}|\theta)f_{\Theta|D}(\theta|d)d\theta=\mathbb{E}_{\Theta|D=d}[f_{X|\Theta}(x^{*}|\theta)]. Two major