GeMi : A G raph-bas e d, M ult i modal Recommendation System for Narrative Scroll Paintings This work was done prior to the author joining his current affiliation.

Haimonti Dutta, Pruthvi Moluguri, Jin Dai, Saurabh Amarnath Mahindre · 2026

Read GeMi : A G raph-bas e d, M ult i modal Recommendation System for Narrative Scroll Paintings This work was done prior to the author joining his current affiliation. by Haimonti Dutta, Pruthvi Moluguri, Jin Dai, Saurabh Amarnath Mahindre (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00854.txt.

Introduction

Dutta and three colleagues design a graph-based recommender for narrative scroll paintings, an endangered art form with text and image content. Their system learns from item links and viewer tastes to suggest similar works for watching or buying, drawing on field-collected data. It also doubles as a storehouse for fragile works and a conservation aid in a field the authors say lacked any dedicated recommender.

Catalog record

CREATORHaimonti Dutta, Pruthvi Moluguri, Jin Dai, Saurabh Amarnath Mahindre
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86250
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.00854

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GeMi : A G raph-bas e d, M ult i modal Recommendation System for Narrative Scroll Paintings This work was done prior to the author joining his current affiliation. Haimonti Dutta, Pruthvi Moluguri, Jin Dai, Saurabh Amarnath Mahindre Abstract Recommendation Systems are effective in managing the ever-increasing amount of multimodal data available today and help users discover interesting new items. These systems can handle various media types such as images, text, audio, and video data, and this has made it possible to handle content-based recommendation utilizing features extracted from items while also incorporating user preferences. Graph Neural Network (GNN)-based recommendation systems are a special class of recommendation systems that can handle relationships between items and users, making them particularly attractive for content-based recommendations. Their popularity also stems from the fact that they use advanced machine learning techniques, such as deep learning on graph-structured data, to exploit user-to-item interactions. The nodes in the graph can access higher-order neighbor information along with state-of-the-art vision-language models for processing multimodal content, and there are well-designed algorithms for embedding, message passing, and propagation. In this work, we present the design of a GNN-based recommendation system on a novel data set collected from field research. Designed for an endangered performing art form, the recommendation system uses multimodal content (text and image data) to suggest similar paintings for viewing and purchase. To the best of our knowledge, there is no recommendation system designed for narrative scroll paintings – our work therefore serves several purposes, including art conservation, a data storage system for endangered art objects, and a state-of-the-art recommendation system that leverages both the novel characteristics of the data and preferences of the user population interested in narrative scroll paintings. keywords Graph Neural Networks, content-based recommendation, multimodal vision language models, folk art recommendation 1 Introduction Recommendation Systems (RS) ( 15 , 139 ) help organize a large number of items (such as books, music, appliances) available to users. They find items of interest by predicting the preference for candidate items and recommending the most pertinent ones using historical data. Data-driven RS falls into three main categories: collaborative filtering (CF), Content-based Filtering (CBF), and Hybrid. In CF models, the emphasis is on a community of users, their behavioral patterns and preferences. The CBF methods depend on the user’s behavioral data along with information about the items, such as words in a text document, color in images, and audio content. Some recommendation systems are known to combine collaborative filtering methods with content-based methods for added benefits. All three categories of recommendation systems use multimodal data (images, text, audio, video) extensively. Conventional methods for integrating data from different modalities in RS include concatenation or summation ( 31 ; 62 ). However, with the advent of deep learning techniques with non-linear modeling capabilities, each modality can be represented separately and combined subsequently to generate a suitable representation. The recent increase in interest in graph-based recommendation systems has inspired a new line of work that explores higher-order interactions between users and items by leveraging the power of graph neural networks ( 32 ; 115 ; 129 ; 131 ; 141 ; 140 ; 110 ; 109 ; 101 ; 140 ). For example, MMGCN ( 110 ) exploits user-item interactions to guide representation learning in each modality by constructing graph convolutional networks with nodes having topological features and aggregating features of neighbors. GRCN ( 109 ) refines this technique by designing a graph refining layer that is designed to identify noisy edges with high confidence in user preference. However, user and item representations learned by graph aggregations are often inherently noisy, since user-item interactions are sparse. These systems also suffer from cold start problems, where there is not enough data (such as clicks, ratings, or purchases) to learn preferences. One solution to the user-item sparsity problem is to reinforce node (users and items) features. However, the noise present in the nodes often makes it difficult to extract useful information for recommendations. Noise in item features (such as blurry images, errors in spelling in text, damaged soundtracks, or misleading descriptions) causes uncertainty in representation. The item features may also contain irrelevant information, such as background color or distracting objects in the image. In multimodal settings, complementary information from different modalities is leveraged to deal with sparsity, but noisy features can lead to undesirable results. If certain modalities are missing or absent, the multimodal system must accommodate this. To address these challenges in multimodal learning, Variational Auto Encoders (VAEs) have become popular ( 122 ; 52 ; 66 ; 50 ) in which modality-specific encoders learn a Gaussian variable parameterized by a mean and variance vector for each node. The mean vector is modeled to represent semantic information, and the variance vector captures the uncertainty of the corresponding modality. In this paper, we describe a multimodal content-based recommendation system augmented with user preference information, that is designed for a unique, sociocultural application to conserve and study an endangered performing art from eastern India. Singing painters are a group of itinerant storytellers who paint scrolls and use a song or poem to explain the artwork. The art is usually inspired by epics, regional folktales, and events that transpire in the day-to-day lives of the painters ( 69 ). Songs or poems are typically learned by word-of-mouth. Sporadic, ad hoc efforts have been made to store samples of artwork or song texts in non-digitized formats, but to the best of our knowledge, a meticulous study of this multimodal data from images and text, or the design of a recommendation system to organize it, has never been undertaken. In recent work ( 19 ), we have prepared a custom dataset of digitized images, aligning lyrics of songs with images, and documenting, to a good extent, regional variations in style, theme, color choice, and other relevant details. We use an enhanced version of this dataset 1 1 1 More data is available from subsequent field studies in other districts of the state. to design the recommendation engine, which can suggest artwork similar to what appreciative users like. In doing so, it was found that while there have been many techniques for the construction of multimodal recommendation systems for known benchmark datasets (such as Amazon ( 75 ), MovieLens ( 30 ) or Netflix ( 4 )), the application of these to a novel data set can quickly become tedious if careful analysis and principled experimentation are not adhered to. We report our findings on the design of the multimodal recommendation system, GeMi, carefully benchmarking it with state-of-the-art systems found in the literature. The dataset used in this research is available from: https://github.com/Haimonti/Folk-Art-Recommendation-System . The primary contributions of this paper are: • A unique, multimodal dataset collected from field surveys and curated for research • The GeMi Recommendation System - A content-based recommendation using state-of-the-art vision-language models with sophisticated multimodal fusion, graph structure learning, and incorporating user preferences through collaborative filtering style plug-and-play modules. This paper is organized as follows: Section 2 presents Related Work; the data collection procedure along with non-digitized sources are described in Section 3 ; the recommendation system is presented in Section 4 ; empirical results comparing the performance of GeMi to state-of-the-art transductive and

Read the complete text