Spectrum-based Modality Representation Fusion Graph Convolutional Network for Multimodal Recommendation Conference: Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining; March 10–14, 2025; Hannover, Germany Proceedings of the Eighteenth ACM International Conferenc

Rongqing Kenneth Ong, Andy W. H. Khong · 2024

Read Spectrum-based Modality Representation Fusion Graph Convolutional Network for Multimodal Recommendation Conference: Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining; March 10–14, 2025; Hannover, Germany Proceedings of the Eighteenth ACM International Conferenc by Rongqing Kenneth Ong, Andy W. H. Khong (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14978.txt.

Introduction

Rongqing Kenneth Ong and Andy Khong fuse image, text, and other signals for recommendations without letting noise swamp the mix. Each modality carries its own noise, and plain concatenation or attention can amplify cross-modality noise. Their system, SMORE, moves multi-modal features into the frequency domain and fuses them there. A filter damps modality-specific noise while keeping shared patterns.

A multi-modal graph learning module mines links among similar items, and a preference module weighs modalities per user. The design aims to capture both single-modality taste and fused taste at once. The work appeared at the ACM Web Search and Data Mining conference in Hannover.

Catalog record

CREATORRongqing Kenneth Ong, Andy W. H. Khong
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-14978

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Spectrum-based Modality Representation Fusion Graph Convolutional Network for Multimodal Recommendation Conference: Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining; March 10–14, 2025; Hannover, Germany Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining (WSDM ’25), March 10–14, 2025, Hannover, Germany DOI: 10.1145/3701551.3703561 ISBN: 979-8-4007-1329-3/25/03 CCS: Information systems Recommender systems Rongqing Kenneth Ong, Andy W. H. Khong Abstract . Incorporating multi-modal features as side information has recently become a trend in recommender systems. To elucidate user-item preferences, recent studies focus on fusing modalities via concatenation, element-wise sum, or attention mechanisms. Despite having notable success, existing approaches do not account for the modality-specific noise encapsulated within each modality. As a result, direct fusion of modalities will lead to the amplification of cross-modality noise. Moreover, the variation of noise that is unique within each modality results in noise alleviation and fusion being more challenging. In this work, we propose a new S pectrum-based Mo dality Re presentation (SMORE) fusion graph recommender that aims to capture both uni-modal and fusion preferences while simultaneously suppressing modality noise. Specifically, SMORE projects the multi-modal features into the frequency domain and leverages the spectral space for fusion. To reduce dynamic contamination that is unique to each modality, we introduce a filter to attenuate and suppress the modality noise adaptively while capturing the universal modality patterns effectively. Furthermore, we explore the item latent structures by designing a new multi-modal graph learning module to capture associative semantic correlations and universal fusion patterns among similar items. Finally, we formulate a new modality-aware preference module, which infuses behavioral features and balances the uni- and multi-modal features for precise preference modeling. This empowers SMORE with the ability to infer both user modality-specific and fusion preferences more accurately. Experiments on three real-world datasets show the efficacy of our proposed model. The source code for this work has been made publicly available at https://github.com/kennethorq/SMORE . 1. Introduction In the rapidly expanding world of e-commerce, recommendation systems play a critical role in assisting users to identify products of interest. The standard user browsing experience encompasses exposure to various forms of multi-modal content ( Ma et al. 2024 ) intended to captivate and entice users. As user preferences are generally driven by a mixture of modality content, research into multi-modal recommender systems (MRSs) that leverage modalities to infer user interest has been gaining popularity. Studies have shown that the use of modalities outperforms general recommenders that rely solely on users’ historical interaction ( He and McAuley 2016b ; Zhang et al. 2021 ; Zhou et al. 2023b ; Zhou and Shen 2023 ; Kim et al. 2024 ) . The central focus of MRSs involves the integration of different modalities within the collaborative filtering (CF) framework. Earlier works such as VBPR ( He and McAuley 2016b ) and DeepStyle ( Liu et al. 2017 ) focus on fusing visual-specific modality by first projecting the modality features into a lower-dimensional space before combining it with the item identifier (ID) embeddings using concatenation and summation, respectively. Since user-item interactions can naturally be depicted as a bipartite graph ( He et al. 2020 ; Dai et al. 2022 ; Zhao et al. 2022 ; Ong et al. 2023 ; Chun et al. 2024 ; Zheng et al. 2024 ) , graph neural networks (GNNs) have been adapted to capture the high-order connectivity of both multi-modal and behavioral interactions. To this end, LATTICE ( Zhang et al. 2021 ) constructs the latent structures associated with modalities by first constructing different views of the item graph. Thereafter, it performs modality fusion using an attention mechanism to weigh each modality. As an extension, FREEDOM ( Zhou and Shen 2023 ) reduces redundancy in training the latent graphs by freezing them prior to training. More recently, to mitigate challenges associated with modality noise that may be introduced through pre-trained encoders, MGCN ( Yu et al. 2023 ) integrates behavioral features with modality features in an attempt to reduce such noise. It then employs average pooling to fuse each of the modality features. Figure 1. An illustrative example of user multi-modality preferences and issues related to modality-specific contamination. (i) User multi-modal preferences, (ii) irrelevant textual descriptions resulting in an unexpectedly high similarity score of 69.35% even though item pairs are unrelated, and (iii) blurred images resulting in a low similarity score of 11.27% even though pairs are related. While existing MRSs achieve notable success in incorporating multiple modalities, they suffer from a significant drawback—amplification of cross-modality noise during fusion ( Zhang et al. 2024 ; Yang et al. 2023 ) . Although it is essential to infer fusion preferences, existing works ( He and McAuley 2016b ; Liu et al. 2017 ; Zhang et al. 2022 ; Zhang et al. 2021 ) combine modalities directly without taking into account the impact of modality-specific noise, which may detrimentally affect the quality of item representations learned. Consider an illustrative case study constructed from the Amazon Clothing ( He and McAuley 2016a ) dataset. Fig. 1 (i) highlights the importance of fusion preferences, where users make purchase decisions from an image and text description. However, capturing fusion preferences via modality fusion carries the risk of modality-specific contamination. For instance, consider a pair of items in (ii), which differ in terms of their functionality—the left item being a toy hammer, while the right being a jumpsuit. Due to the irrelevant description for the latter (which mentioned Hammer ), these items yield a surprisingly high similarity score of 69.35% despite them being uncorrelated. In terms of visual contamination, as shown in Fig. 1 (iii), a similar pair of items (e.g., travel pouches) yields an unexpectedly low similarity score of 11.27% when one of the images suffers from a blurring effect. As a result, the noise within each modality will be amplified further during fusion, thereby corrupting the item encoding process. Nonetheless, reducing noise contamination within each modality is highly challenging due to their unique and dynamic contamination characteristics ( Izadi et al. 2023 ; Anwar and Barnes 2019 ) . To reliably ascertain the multi-modal preferences of a user, a fusion module must be designed meticulously to mitigate the undesired effect of noise within each modality before fusion so as to capture universal patterns effectively while preserving essential uni-modal features. In this work, we draw inspiration from the field of signal processing, which has shown to be effective for modality denoising ( Rao et al. 2021 ; Michelashvili and Wolf 2019 ) . By projecting signals to the frequency domain via the Fourier transform ( Heideman et al. 1984 ) , a sparse frequency spectrum is generated. This unique characteristic facilitates the acquisition of discriminative spectrum ( Lao et al. 2024 ) , which enables the distillation of critical modality features by means of effective attenuation. Furthermore, the frequency domain offers a comprehensive global perspective ( Rao et al. 2021 ; Xu et al. 2024 ) , thereby enabling each spectral component to attend to all spatial domain features efficiently and effectively. Inspired by the discriminative spectral property and the global perspective by the frequency domain and to overcome the aforementioned drawbacks, we propose a new S pectrum-based Mo dality Re

Read the complete text