January Food Benchmark (JFB): A Public Benchmark Dataset and Evaluation Suite for Multimodal Food Analysis

Amir Hosseinian, Ashkan Dehghani Zahedani, Umer Mansoor, Noosheen Hashemi, Mark Woodward · 2025

Read January Food Benchmark (JFB): A Public Benchmark Dataset and Evaluation Suite for Multimodal Food Analysis by Amir Hosseinian, Ashkan Dehghani Zahedani, Umer Mansoor, Noosheen Hashemi, Mark Woodward (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-09966.txt.

Introduction

Amir Hosseinian and co-authors release a public test set for AI food analysis. Judging what is in a meal from a photo could aid nutrition tracking. Progress stalls for lack of shared, real world test data. Their benchmark holds one thousand food photos, each checked and labeled by people.

A scoring frame goes with the set, built to weigh models in a rounded, use based way. The team ran broad vision language models on it as baselines. Their own specialist model beat the best general model by a clear margin on the joint score. The authors offer the set and frame as common ground for later work in meal analysis.

Catalog record

CREATORAmir Hosseinian, Ashkan Dehghani Zahedani, Umer Mansoor, Noosheen Hashemi, Mark Woodward
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-09966

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

January Food Benchmark (JFB): A Public Benchmark Dataset and Evaluation Suite for Multimodal Food Analysis Amir Hosseinian, Ashkan Dehghani Zahedani, Umer Mansoor, Noosheen Hashemi, Mark Woodward Abstract Progress in AI for automated nutritional analysis is critically hampered by the lack of standardized evaluation methodologies and high-quality, real-world benchmark datasets. To address this, we introduce three primary contributions. First, we present the January Food Benchmark (JFB), a publicly available collection of 1,000 food images with human-validated annotations. The validation process followed a rigorous protocol to ensure high-quality, reliable ground-truth labels. Second, we detail a comprehensive benchmarking framework, including robust metrics and a novel, application-oriented overall score designed to assess model performance holistically. Third, we provide baseline results from both general-purpose Vision-Language Models (VLMs) and our own specialized model, january/food-vision-v1 . Using this framework, we find that january/food-vision-v1 establishes a strong baseline with an Overall Score of 86.2, a 12.1-point improvement over the strongest general-purpose, oracle-prompt configuration ( GPT-4o (Best) , 74.1). This work provides the research community with a valuable new evaluation dataset and a rigorous framework to guide and benchmark future developments in automated nutritional analysis. Keywords: food recognition, vision-language models, nutritional analysis, benchmark dataset, computer vision 1 Introduction The rising prevalence of diet-related chronic diseases has underscored the urgent need for accessible and accurate dietary tracking tools. Automating food logging from images captured on mobile devices presents a user-friendly solution, yet the underlying AI task is fraught with difficulty [ 1 , 2 ] . Evaluating the accuracy of an AI’s output is a critical but challenging step. This difficulty stems from two core problems: the lack of standardized metrics and, more importantly, the absence of high-quality, publicly available datasets that can serve as a reliable ground truth for the nuanced domain of food analysis. Existing general-purpose Vision-Language Models (VLMs) have demonstrated impressive capabilities across a wide range of tasks [ 3 , 4 ] , though their performance on specialized, fine-grained domains remains an active area of investigation [ 5 ] . The specific domain of food image analysis—which requires identifying a dish, recognizing its constituent ingredients, and estimating nutritional content—presents unique challenges. These include high visual similarity between different dishes, severe ingredient occlusion, and the need to infer non-visual attributes like portion size and cooking methods [ 6 , 7 ] . Without a specialized, high-fidelity dataset, it is difficult to quantify how well these general models perform in this domain or to measure the value of domain-specific fine-tuning. To address these challenges, we present three primary contributions: • A High-Quality Benchmark Dataset (JFB): We introduce the January Food Benchmark (JFB), a collection of 1,000 real-world food images with human validated annotations for meal names, ingredients, and macronutrients. The dataset is released under a CC-BY-4.0 license to encourage reproducibility and downstream research. • An Automated Benchmarking Framework: We propose a comprehensive evaluation methodology with robust metrics for meal identification, ingredient recognition, nutritional estimation, latency, and cost. We also introduce a novel Overall Score , a weighted metric tailored to reflect application-oriented needs for a balanced, high-performing system. • Baseline Performance Scores: We benchmark top general-purpose models, evaluated under both average and best-case zero-shot conditions , against our specialized model, january/food-vision-v1 , to establish a quantitative performance ranking and quantify the gap between generalist and specialist approaches. Our analysis quantifies the significant performance gains from a specialized approach, establishing a strong baseline for future research and development in this challenging domain. 1.1 Comparison to Existing Resources While numerous food datasets exist, JFB is specifically designed to fill a gap in high-fidelity, meal-level analysis from unconstrained, real-world user photos. Table 1 contrasts JFB with other prominent resources, highlighting its unique focus on validated, multi-faceted annotations for complex meals, coupled with an open evaluation suite. Table 1 : Comparison of JFB with existing food datasets and benchmarks. Dataset Meal Name Ingredient List Macronutrients Real-World Photos Human Validation Food-101 [ 8 ] ✓ ✗ ✗ ✗ ✗ Recipe1M+ [ 9 ] ✓ ✓ ✗ ✗ ✗ MEAL [ 10 ] ✓ ✓ ✓ ◐ ◐ JFB (Ours) ✓ ✓ ✓ ✓ ✓ ✓= Fully Provided ◐= Partially Provided ✗ = Not Provided As the table illustrates, large-scale datasets like Food-101 and Recipe1M+ are invaluable for pre-training but are unsuitable for this evaluation task as they lack complete, validated nutritional annotations. While MEAL [ 10 ] is a closer analogue, its data is a mix of real-world and web-scraped images with only partial validation, making it less reliable as a ground-truth benchmark. JFB is unique in providing all requisite components: comprehensive annotations (meal name, ingredients, and nutrition) that are fully human-validated on a dataset composed entirely of real-world mobile photos . This validation was performed according to a strict annotation protocol, ensuring a reliable and consistent ground truth for the complex, multi-faceted task of automated food analysis. 2 Background Automated dietary assessment has emerged to address the well-documented limitations of traditional methods like 24-hour recalls and food diaries [ 11 ] . Image-based analysis, powered by AI, offers a more objective and scalable alternative. The field’s technical evolution has been rapid. Early approaches on datasets like UEC-FOOD256 [ 12 ] and Food-101 [ 8 ] relied on hand-crafted features, which were quickly superseded by deep learning [ 13 ] . The application of Convolutional Neural Networks (CNNs), particularly ResNets [ 14 ] , marked a significant milestone. More recently, Vision Transformers (ViTs) have emerged as a powerful alternative [ 15 ] . Both architectures are often pre-trained on massive datasets and then fine-tuned on specialized data, a practice shown to be critical for achieving high performance in food recognition [ 16 ] . The availability of large-scale food datasets—from item-centric collections like Food2K [ 17 ] to those linking images with recipes like Recipe1M+ [ 9 ] and recent benchmarks for complex meal-level analysis [ 10 ] —has been crucial in driving this progress. Despite architectural advancements, food analysis remains a formidable set of technical challenges. It is a canonical fine-grained visual classification (FGVC) problem [ 6 ] , complicated by high intra-class variance and the deformable nature of most food items [ 18 ] . To address this, some research has focused on multi-task learning frameworks that jointly recognize a dish and its ingredients [ 19 ] . However, ingredient recognition is still hampered by heavy occlusion, and portion size estimation—inferring 3D volume from a 2D image—remains a primary bottleneck for quantitative accuracy, despite progress with dedicated datasets like PFID [ 20 ] and DoFoo [ 21 ] . The current frontier is defined by general-purpose Vision-Language Models (VLMs). Foundational models like CLIP learned rich visual-semantic embeddings from web-scale language supervision [ 3 ] . This evolved into instruction-tuned VLMs, such as LLaVA [ 4 ] and InstructBLIP [ 22 ] , which can follow complex prompts to perform detailed visual analysis. These models offer a paradigm shift from single-task classifiers to versatile, zero-shot analytical tools. However, their power comes with risks; they are prone to factual hallucination [ 23 ] , and their generalist training may lack the nuanced, domain-specific knowledge required for a safety-critical application like nutritional science. Recent work has begun to highlight these safety concerns specifically within

Read the complete text