Towards Unbiased Cross-Modal Representation Learning for Food Image-to-Recipe Retrieval

Qing Wang, Chong-Wah Ngo, Ee-Peng Lim · 2025

Read Towards Unbiased Cross-Modal Representation Learning for Food Image-to-Recipe Retrieval by Qing Wang, Chong-Wah Ngo, Ee-Peng Lim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-15201.txt.

Introduction

Qing Wang, Chong-Wah Ngo, and Ee-Peng Lim tackle a bias in systems that match a food photo to its recipe. A recipe causes the dish, but a photo never shows every step or ingredient equally: cooking, plating, and camera angle all hide detail. Standard training treats the recipe as a flat description of the picture, so models latch onto the most visible matches and miss subtle cues that decide the right pairing.

The authors model that mismatch with causal theory, separating what the image truly shows from what the recipe merely says. Learning then corrects for the misleading pull of dominant alignments. The causal framing should make recipe retrieval fairer to dishes whose photos understate their contents.

Catalog record

CREATORQing Wang, Chong-Wah Ngo, Ee-Peng Lim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-15201

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Unbiased Cross-Modal Representation Learning for Food Image-to-Recipe Retrieval Qing Wang, Chong-Wah Ngo, Ee-Peng Lim Abstract This paper addresses the challenges of learning representations for recipes and food images in the cross-modal retrieval problem. As the relationship between a recipe and its cooked dish is cause-and-effect, treating a recipe as a text source describing the visual appearance of a dish for learning representation, as the existing approaches, will create bias misleading image-and-recipe similarity judgment. Specifically, a food image may not equally capture every detail in a recipe, due to factors such as the cooking process, dish presentation, and image-capturing conditions. The current representation learning tends to capture dominant visual-text alignment while overlooking subtle variations that determine retrieval relevance. In this paper, we model such bias in cross-modal representation learning using causal theory. The causal view of this problem suggests ingredients as one of the confounder sources and a simple backdoor adjustment can alleviate the bias. By causal intervention, we reformulate the conventional model for food-to-recipe retrieval with an additional term to remove the potential bias in similarity judgment. Based on this theory-informed formulation, we empirically prove the oracle performance of retrieval on the Recipe1M dataset to be MedR=1 across the testing data sizes of 1K, 10K, and even 50K. We also propose a plug-and-play neural module, which is essentially a multi-label ingredient classifier for debiasing. New state-of-the-art search performances are reported on the Recipe1M dataset. 1 Introduction Cross-modal recipe retrieval is potentially a more scalable approach than training a classifier for food image or ingredient recognition [ 20 , 46 ] . Given a food image query, the task is to retrieve the corresponding recipe to provide the necessary information, such as dish name and ingredients, for the estimation of nutrition and calorie content. In the literature [ 30 , 1 , 3 , 9 , 47 , 37 , 48 , 34 , 10 , 29 , 31 ] , the above task is posed as a cross-modal representation learning problem, where every recipe and its food image are transformed and projected to a shared embedding space to maximize their pairwise similarity. Under this formulation, a recipe is assumed to provide textual descriptions of the food’s visual content. The relationship between a recipe and its food image, however, is beyond visual-textual alignment: a recipe narrates the process of preparing a dish while a food image visualizes the outcome of cooking. The cause-and-effect alignment between a textually described cooking process and a static visual snapshot of the cooking outcome, in general, characterizes the underlying challenge of recipe retrieval using an image as the query. We postulate that, due to pairwise alignment, the joint space tends to capture the dominant content of food images and recipes, while overlooking the subtle visual changes due to the cooking process. For example, the recipe “drunken chicken” lists the major ingredient “chicken” and other minor ingredients such as “ginger” and “goji”. While the major ingredient will visually present in the food image, the minor ingredients such as “ginger” or “goji” may not be visible subject to their sizes, occlusion, and image-capturing conditions. Due to the inconsistency between a recipe and its paired image, learning image representations that can capture all details of ingredients for alignment is practically difficult. Therefore, the existing representation learning techniques fall short in capturing the subtle variations to distinguish the recipes of visually similar food images (e.g., drunken chicken, steamed ginger chicken, herbal chicken). This paper addresses the bias in representation learning from a causal view that treats food image as the effect of manipulating culinary elements by following cooking instructions in a recipe. The causal view allows us to identify the potential confounders that introduce spurious correlation in representation learning. In general, culinary elements such as ingredients, cooking, and cutting methods are potential confounders. In this paper, we mainly consider ingredients to formulate the problem. By backdoor adjustment [ 23 ] , a causal-informed equation is derived to alleviate the potential bias arising from ingredients. We discuss the challenge of realizing this equation for recipe retrieval and propose a neural module implementation to approximate this equation. The module is essentially a multi-labeled ingredient classifier that predicts the distribution of ingredients in an image for readjustment of image representation during learning. This debiasing module can be readily plugged into most of the existing neural architectures for recipe retrieval, including the SoTA models such as H-T [ 29 ] , TFood [ 31 ] , VLPCook [ 32 ] for cross-modal learning. A significant contribution of our work is that the causal view provides a causal theory-informed upper-bound performance for the problem of image-to-recipe retrieval. By simulating the proposed debiasing module with the desired classification performance, a near-perfect retrieval performance is attained on the Recipe1M dataset [ 30 ] . When applying the module to debiase the representation learning of SoTA models such as VLPCook [ 32 ] , the new state-of-the-art performances are also reported on Recipe1M. To the best of our knowledge, this is the first paper addressing the bias in representation learning based on the causal theory for cross-modal recipe retrieval. 2 Related Work The past research efforts in recipe retrieval are mostly devoted to learning to represent the highly structured information in recipes [ 6 , 30 ] . The early works [ 1 , 30 ] have established the basis for encoding the three sections of information in a recipe (dish title, ingredients, instructions) independently using LSTM. The encoded representations of each section are then concatenated and projected to align with the visual representation. Variants of methods have been proposed based on this LSTM-based architecture, including attentional mechanism [ 3 , 38 , 16 ] , multi-task learning [ 2 ] , hierarchical modeling of semantics [ 24 ] , and adversarial learning [ 47 , 37 , 10 , 44 ] . As reported by a comparative study in [ 46 ] , these efforts, which explore textually rich descriptions in recipes to align food images, have incrementally boosted search performance. Transformer-based neural architectures have also been investigated for image-and-recipe alignment. Representative works include H-T (hierarchical recipe transformer) [ 29 ] , T-Food (transformer decoders for food) [ 31 ] and VLPCook (visual-language pre-training) [ 32 ] . This line of works is first demonstrated by H-T [ 29 ] via replacement of LSTM with a transformer to demonstrate significant improvement in search. Following early works, H-T also encodes the three sections of the recipe independently. T-Food [ 31 ] improves H-T by having additional layers of transformers to model interactions among the representations encoded from the title, ingredient, and cooking instruction sections. Furthermore, different from [ 30 , 1 , 9 , 42 ] which typically employ ResNet-50 as an image encoder, T-Food employs transformers to encode image features as well as modeling the inter-dependencies between image and recipe features. More recently, foundational models have also been employed in [ 32 , 12 ] . VLPCook [ 32 ] leverages CLIP [ 26 ] to extract the local (ingredient) and global (cooking instructions) contexts from food images. Vision transformer (ViT) is then employed to jointly embed the food images and contexts for image representation learning.

Read the complete text