SalientFusion: Context-Aware Compositional Zero-Shot Food Recognition

Jiajun Song, Xiaoou Liu✉ · 2025

Read SalientFusion: Context-Aware Compositional Zero-Shot Food Recognition by Jiajun Song, Xiaoou Liu✉ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-03873.txt.

Introduction

Jiajun Song and one co-author devote a 2025 research paper to recognising food dishes never seen during training, even in new cuisine and ingredient mixes. New dishes appear all the time, so a recogniser cannot require examples of every category. Their new task frames cuisines as attributes and ingredients as objects, matching the structure of compositional zero-shot learning, where known parts recombine into unseen wholes.

Three obstacles are named: distracting backgrounds, mix-ups between staple and side dishes, and semantic bias toward a single attribute. Their method, SalientFusion, answers with two components. SalientFormer strips background redundancy and resolves role confusion using depth features. DebiasAT reduces that bias by aligning text prompts with visual features. Two new benchmarks are introduced, with categories numbering in the hundreds. SalientFusion reaches leading results on those benchmarks and on the most popular general datasets as well. Code is released publicly.

Catalog record

CREATORJiajun Song, Xiaoou Liu✉
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-03873

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SalientFusion: Context-Aware Compositional Zero-Shot Food Recognition Jiajun Song, Xiaoou Liu✉ Abstract Food recognition has gained significant attention, but the rapid emergence of new dishes requires methods for recognizing unseen food categories, motivating Zero-Shot Food Learning (ZSFL). We propose the task of Compositional Zero-Shot Food Recognition (CZSFR), where cuisines and ingredients naturally align with attributes and objects in Compositional Zero-Shot learning (CZSL). However, CZSFR faces three challenges: (1) Redundant background information distracts models from learning meaningful food features, (2) Role confusion between staple and side dishes leads to misclassification, and (3) Semantic bias in a single attribute can lead to confusion of understanding. Therefore, we propose SalientFusion , a context-aware CZSFR method with two components: SalientFormer, which removes background redundancy and uses depth features to resolve role confusion; DebiasAT, which reduces the semantic bias by aligning prompts with visual features. Using our proposed benchmarks, CZSFood-90 and CZSFood-164, we show that SalientFusion achieves state-of-the-art results on these benchmarks and the most popular general datasets for the general CZSL. The code is avaliable at https://github.com/Jiajun-RUC/SalientFusion . 1 Introduction Food is integral to daily life and underpins many societal processes. Despite significant advances in food recognition [ 11 , 16 , 22 ] —a cornerstone of food computing [ 10 ] —existing methods struggle with novel, unseen dishes. To overcome this, we introduce Zero-Shot Food Learning (ZSFL), a framework for recognizing food categories absent from training data. Given the limited research and lack of dedicated datasets for ZSFL [ 23 , 6 ] , we propose the task of Compositional Zero-Shot Food Recognition (CZSFR). In CZSFR, the inherent structure of food is exploited by aligning “cuisines” and “ingredients” with “attributes” and “objects” in Compositional Zero-Shot Learning (CZSL). As illustrated in Fig. 1 , our model is trained on seen compositions (e.g., “braise pork” and “stir-fry chicken”) and evaluated on unseen ones (e.g., “braise chicken” and “stir-fry pork”). This task not only directs future research in ZSFL but also highlights the critical role of compositional analysis in food recognition. Figure 1: An example diagram of compositional zero-shot tasks in the food domain: rows represent food cuisines, and columns represent food ingredients. Within the Cartesian product of cuisines and ingredients, the non-overlap between the training set and the testing set making it challenging to achieve out-of-distribution generalization. Existing CZSL methods face three key challenges in CZSFR: (1) Redundant background information. Food images often include irrelevant elements, such as plates and tables, which distract models from meaningful features. As shown in Fig. 2 (a), Vision-Language Models (VLMs), for instance, may describe a grilled beef image as “grilled beef on a round plate placed on a table”, where “plate” and “table” are irrelevant. Suppressing such distractions can help models focus on the food itself. (2) Role confusion between the staple dish and the side dish. Accurate classification requires distinguishing the staple dish from side dishes. As shown in Fig. 2 (a), misidentifying a side dish, such as vegetables or noodles, as the staple dish can lead to errors. Emphasizing the staple’s role is crucial for precise categorization. (3) Semantic bias in a single attribute can lead to confusion of understanding. The single attribute like “stew” can differ in meaning across categories. As shown in Fig. 2 (b) and (c), “stew” in stew sparerib aligns with “braised”, while in stew seafood, it aligns with “boil”. Addressing these biases requires aligning visual features with contextual prompts. By tackling these issues, models can better handle CZSFR and generalize to unseen combinations. To address the above three identified challenges, we propose SalientFusion , a context-aware method for CZSFR, comprising two key components: SalientFormer and DebiasAT module. First, SalientFormer tackles redundant background information and role confusion by combining image segmentation [ 20 ] and depth detection [ 1 ] . The segmentation removes irrelevant elements like plates and tables, while depth information can capture the volume and distance of objects, which helps distinguish the staple dish from side dishes. These features are fused to produce a salient representation focused on meaningful food regions. Second, to address the semantic bias in a single attribute across different compositions, DebiasAT module aligns static text representations with salient visual features extracted by SalientFormer. By dynamically guiding text learning based on visual context, this module ensures more accurate semantic alignment and reduces attribute misinterpretation. Together, these components enable SalientFusion to effectively address CZSFR’s unique challenges, redundant information, role confusion and semantic bias in a single attribute, setting the stage for improved generalization to unseen food compositions. Finally, we re-divide and re-annotate two popular food image datasets, ETH Food-101 [ 2 ] and VireoFood-172 [ 3 ] , to create two novel benchmarks, CZSFood-90 and CZSFood-164, tailored for the CZSFR task. Specifically, we replace the original category labels with compositions of cuisines and ingredients to align with the compositional nature of the task. To better evaluate model performance in practical scenarios, we introduce a real-world testing method. This approach comes from the observation that most novel food categories arise from discovering new ingredients or recombining existing ingredients in a new way. The real-world testing therefore evaluates models on unseen cuisine-ingredient compositions, simulating how new dishes emerge in real-world settings. Our proposed framework, SalientFusion , achieves state-of-the-art (SOTA) performance on both benchmarks under closed-world and real-world testing settings. Additionally, SalientFusion demonstrates its generalizability by achieving SOTA results on the MIT-States [ 5 ] , a benchmark widely used in the general CZSL task beyond food recognition. These results highlight the robustness and effectiveness of our method in addressing the challenges of CZSFR. Figure 2: Challenges of CZSFR. (a) shows the “grilled beef” image affected by redundant information and role confusion. (b) and (c) are “stew sparerib” and “stew seafood”, representing different compositions within the same cuisine. Overall, our main contributions can be summarized as follows: • We are the first to explore CZSFR and extensively explore existing CZSL methods on food image datasets. • We introduce the SalientFusion for CZSFR. The proposed SalientFormer and the DebiasAT module solve the problem of the redundant information, role confusion and semantic bias in a single attribute. • We benchmark the CZSFR with two food image datasets and propose the real-world testing method. The proposed SalientFusion achieves the best performance on all food datasets and a widely-used general dataset. 2 Related Work Figure 3: Architecture of SalientFusion . The SalientFormer extracts salient features via segmentation and depth detection and fuses them, while the DebiasAT module aligns text prompts with output fused features to address compositional challenges using CLIP-based encoders. From ZSFL to Compositional Understanding. Traditional Zero-Shot Food Learning (ZSFL) methods, whether generative (e.g., ESE-GAN [ 6 ] ) or attribute-based (e.g., SeeDS [ 23 ] ), typically treat food categories as monolithic entities. This holistic view limits their ability to generalize to novel dishes created by combining known cuisines and ingredients. Our work addresses this by shifting focus to Compositional Zero-Shot Food Recognition (CZSFR), which explicitly models the compositional nature of food. Compositional Zero-Shot Learning (CZSL). Existing CZSL methods primarily fall into two camps. (1) Representation-based approaches like CompCos [ 7 ] and SymNet [ 15 ] aim to learn disentangled representations for primitives (e.g., attributes and objects). While effective at separating concepts, they

Read the complete text