Generating with Fairness: A Modality-Diffused Counterfactual Framework for Incomplete Multimodal Recommendations

Jin Li, Shoujin Wang, Qi Zhang, Shui Yu, Fang Chen · 2025

Read Generating with Fairness: A Modality-Diffused Counterfactual Framework for Incomplete Multimodal Recommendations by Jin Li, Shoujin Wang, Qi Zhang, Shui Yu, Fang Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11916.txt.

Introduction

Jin Li and co-authors present a recommender systems paper, published in 2025, on fair suggestions when some product information is missing. Multimedia recommendations use images and text as well as ratings, but items often lack one channel, and plain items then get buried even when users would love them. That visibility bias, the authors argue, is a fairness problem as much as an accuracy problem.

Their framework first generates the missing channel with a diffusion model that learns each channel's distribution, refining the fill in over rounds. A second module reasons counterfactually to cancel the causal effect of visibility bias on rankings. Together the parts aim to recommend the items users truly prefer, complete or not.

Catalog record

CREATORJin Li, Shoujin Wang, Qi Zhang, Shui Yu, Fang Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-11916

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generating with Fairness: A Modality-Diffused Counterfactual Framework for Incomplete Multimodal Recommendations Jin Li, Shoujin Wang, Qi Zhang, Shui Yu, Fang Chen Abstract . Incomplete scenario is a prevalent, practical, yet challenging setting in Multimodal Recommendations (MMRec), where some item modalities are missing due to various factors. Recently, a few efforts have sought to improve the recommendation accuracy by exploring generic structures from incomplete data. However, two significant gaps persist: 1) the difficulty in accurately generating missing data due to the limited ability to capture modality distributions; and 2) the critical but overlooked visibility bias, where items with missing modalities are more likely to be disregarded due to the prioritization of items’ multimodal data over user preference alignment. This bias raises serious concerns about the fair treatment of items. To bridge these two gaps, we propose a novel Modality-Diffused Counterfactual (MoDiCF) framework for incomplete multimodal recommendations. MoDiCF features two key modules: a novel modality-diffused data completion module and a new counterfactual multimodal recommendation module. The former, equipped with a particularly designed multimodal generative framework, accurately generates and iteratively refines missing data from learned modality-specific distribution spaces. The latter, grounded in the causal perspective, effectively mitigates the negative causal effects of visibility bias and thus assures fairness in recommendations. Both modules work collaboratively to address the two aforementioned significant gaps for generating more accurate and fair results. Extensive experiments on three real-world datasets demonstrate the superior performance of MoDiCF in terms of both recommendation accuracy and fairness. The code and processed datasets are released at https://github.com/JinLi-i/MoDiCF . DOI: 10.1145/3696410.3714606 CCS: Information systems Retrieval tasks and goals 1. Introduction Multimodal data has recently shown massive growth across various multimedia applications, including social media, news and e-commerce platforms ( Song et al., 2023a ) . To address information overload and help users discover interesting items, Multimodal Recommendations (MMRec) ( Liu et al., 2024b ; Liu et al., 2024a ; Deldjoo et al., 2021 ; Zhou et al., 2023b ; Zhou, 2023 ) have emerged as a prevalent research topic. MMRec integrates items’ diverse modalities (e.g., item images, review texts) into user preference modeling, thereby delivering more accurate recommendations. However, most MMRec methods heavily rely on the assumption that all modalities are complete (i.e., fully observed) ( Bai et al., 2024 ) , which doesn’t always hold in the real world. Factors like privacy concerns, data sparsity, and acquisition costs often result in incomplete items, with some modalities missing. This poses a significant challenge to existing methods, as they struggle to exploit useful information from limited data. Figure 1. Examples of visibility bias. (a): Exposure of incomplete items is largely suppressed compared to complete scenarios. (b): Suppression aggravates as missing rates increase. In recent years, a few efforts ( Bai et al., 2024 ; Lin et al., 2023 ; Wang et al., 2018 ; Malitesta et al., 2024 ) have been made to handle incomplete data in MMRec. Among them, two primary strategies have been researched: 1) data completion, which aims to recover missing data using different imputation methods, such as autoencoder ( Wang et al., 2018 ) , feature propagation ( Malitesta et al., 2024 ) , clustering-based hypergraph convolution and cross-modal transport ( Lin et al., 2023 ) , and 2) modality-invariant learning ( Bai et al., 2024 ) , which aims to learn inherent content preferences that is invariant to modality missing. Despite achieving promising accuracy improvements, we identify two major gaps confronted by these existing methods. Gap 1 : they often struggle to accurately recover multimodal missing data due to their limited capability in capturing modality-specific distributions. Existing methods typically focus on learning generic data structures without explicitly exploring data distributions within each modality and their differences across modalities ( Daras et al., 2023 ) . This is particularly crucial in incomplete MMRec, where severe information loss disrupts data structure learning ( Gong et al., 2023 ) , and, moreover, high heterogeneity commonly exists across modalities ( Liang et al., 2024 ) , making the learning process more challenging. For instance, the appearance image and description texts of an item may convey similar semantics but belong to distinct distributions. Failing to accurately capture such modality-specific distributions can greatly diminish data completion quality, leading to suboptimal recommendation performance ( Liu et al., 2024a ) . Recently, diffusion models ( Cao et al., 2024 ; Ho et al., 2020 ; Wang et al., 2023a ) have gained widespread attention for their ability to capture complex data distributions. Though several diffusion-based recommendation methods have been developed ( Yang et al., 2023 ; Ma et al., 2024a ) , they are not applicable in incomplete MMRec due to their unimodal design and inability to handle incomplete data. Thus, there is an urgent need for a specifically devised diffusion module tailored for modality completion in MMRec to fill this gap. Gap 2 : they largely overlook a critical fairness issue, namely the visibility bias of items . This bias refers to the phenomenon that items with missing modalities are more likely to be overlooked or receive less exposure from recommenders, regardless of their genuine alignment with users’ preferences. As an example in Figure 1 (a), the exposure of items in the Baby dataset by an existing MMRec method, MMSSL ( Wei et al., 2023 ) , significantly decreases in incomplete scenarios, confirming the existence of visibility bias. Moreover, Figure 1 (b) shows that as the data missing rate increases, the number of suppressed items rises rapidly, indicating a growing severity of visibility bias. This bias often leads to the unfair treatment of items and, consequently, inaccurate and biased recommendation results. For example, on e-commerce platforms, niche or artisan products may have limited visual content, which reduces their visibility and sales opportunities. This not only affects the sellers but also deprives the users of potentially better-suited products. Such critical yet overlooked bias demands careful attention and a targeted solution. However, addressing it is a non-trivial task that requires a special focus on the causal impacts ( Pearl, 2009 ) of incomplete multimodal content on recommendation outcomes. Most existing debiasing methods ( Wei et al., 2021 ; Shang et al., 2024 ) overlook incomplete scenarios and fail to identify the unique cause of visibility bias. Therefore, they are not ready to address this particular problem. To this end, in this paper, we propose a novel Mo dality- Di ffused C ounter- F actual ( MoDiCF ) framework for incomplete MMRec to address the two aforementioned significant gaps. MoDiCF features the following two well-designed modules. 1) Modality-diffused data completion module. Built within a generative framework, this module excels at capturing and generating missing modalities from complex modality-specific distributions. To effectively harness cross-modal correlations and inject them into the diffusion process, we introduce modality-aware conditioning. It provides complementary modality information to enhance modality-specific distribution learning and facilitate cross-modal alignment. Additionally, we devise a novel iterative refinement strategy to continuously update and improve the generated missing data, further enhancing the quality of data completion. 2) Counterfactual multimodal recommendation module. In this module, we perform counterfactual adjustments on the MMRec process to mitigate the visibility bias and deliver accurate and fair recommendations. We first examine the visibility bias problem through a causal

Read the complete text