MM-Food-100K: A 100,000-Sample Multimodal Food Intelligence Dataset with Verifiable Provenance
Yi Dong, Yusuke Muraoka, Scott Shi, Yi Zhang · 2025
Read MM-Food-100K: A 100,000-Sample Multimodal Food Intelligence Dataset with Verifiable Provenance by Yi Dong, Yusuke Muraoka, Scott Shi, Yi Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10429.txt.
Introduction
Yi Dong, Yusuke Muraoka, Scott Shi, and Yi Zhang release a public data set of 100,000 food images with rich labels, in a 2025 paper. The set is a curated open tenth of a larger corpus of 1.2 million images gathered in six weeks from more than 87,000 contributors. Each entry records details such as dish name and region, and each submission links to a contributor wallet in a secure ledger, so its origin can be checked.
The team describes the data schema, the gathering pipeline, and the quality checks that mix community sourcing with AI screening. To prove the set is useful, they fine-tuned large vision-language models on it to predict nutrition from photos. The tuned models beat their out-of-box versions across standard metrics. The rest of the corpus stays reserved for paid access, with revenue shared to contributors.
Catalog record
| CREATOR | Yi Dong, Yusuke Muraoka, Scott Shi, Yi Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-10429
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MM-Food-100K: A 100,000-Sample Multimodal Food Intelligence Dataset with Verifiable Provenance Yi Dong, Yusuke Muraoka, Scott Shi, Yi Zhang Abstract We present MM-Food-100K , a high-quality, 100,000-sample multimodal food intelligence dataset designed for fine-tuning AI models. Our proposed data sourcing workflow combines community contributions with automated quality review by Large Vision-Language Models, resulting in real-world photos with rich, multi-level annotations. We demonstrate the dataset’s utility by showing that models fine-tuned on MM-Food-100K consistently outperform original Large Vision-Language Models on food classification and regression tasks. We also introduce Codatta Protocol, a novel data contribution framework that uses a privacy-preserving blockchain protocol to track data provenance and enable royalty-based rewards for contributors. MM-Food-100K is released for public use, and the Codatta Protocol provides a new paradigm for building and sustaining high-quality, community-sourced datasets. 1 Introduction Human-generated data remains indispensable for training reliable multimodal AI systems. While synthetic data can amplify scale, it tends to recycle model priors, misses edge-case realism, and struggles with grounded facts. As a result, real-world sourced data and advanced, knowledge-rich labeling continues to be in high demand. Today’s human-intelligence pipelines exhibit a dual-market failure . On the demand side, businesses face high costs to obtain domain-rich, verifiable data. On the supply side, knowledge providers are frequently underpaid or misaligned with downstream value, while attribution is rarely preserved, depressing participation and diversity of expertise. In this paper we introduce MM-Food-100K , a 100,000-sample, multimodal (image + textual annotation) dataset released for public research. The dataset samples are diverse, showcasing a variety of dishes and ingredients, as shown in Figure 1 . MM-Food-100K is a curated \sim 10% open subset of an original 1.2-million , quality-accepted corpus collected in six weeks from 87,000+ contributors. Data is sourced via a partnership campaign between Binance Wallet and the Codatta protocol , which combines community sourcing with configurable AI-assisted quality checks. Each submission is linked to an on-chain wallet address to provide verifiable provenance and attribution; a fully on-chain protocol is on the roadmap. Importantly, contributors were invited with zero upfront payment; instead, the model emphasizes attribution and prospective revenue sharing for the retained commercial portion of the corpus, aligning incentives without front-loading cost or risk. Figure 1 : Overview of Food Intelligence Dataset Samples: A collage of diverse dishes and ingredients from multiple cuisines, showcasing the variety in the Food Intelligence dataset-including noodles, dumplings, soups, stir-fries, seafood, meats, breads, fruits, and snacks-captured in varied lighting, perspectives, and serving styles. Contributions: • Dataset (primary) . We release MM-Food-100K , a public multimodal food-intelligence dataset annotated for dish/cuisine, visible ingredients, portion and scale cues and etc., with evidence and provenance for auditability. We demonstrate its utility by fine-tuning large vision–language models for image-based food-related prediction (such as dish recognition), observing consistent improvements over out-of-box baselines. • Framework (secondary) . We describe briefly the Codatta blockchain-based data contribution model—data infrastructure for privacy-preserving transparency, structured task/schema design, and configurable human/AI quality assurance, contributor profile and reputation system, and wallet-linked provenance —that enabled verifiable data acquisition at scale and royalty-based payment business model via data classification. 2 The MM-Food-100K Dataset 2.1 Overview and applications MM-Food-100K is a public, 100,000-sample subset of a larger \sim 1.2M food corpus curated for fine-grained food intelligence . Each record pairs an image with structured annotations that enable recognition (dish/cuisine), ingredient extraction, portion and calorie estimation, cooking-method reasoning, and authenticity checks. The dataset targets practical applications such as image-based nutrition estimation, recipe recommendation, diet logging/health monitoring, and fine-grained vision research. Why another dataset? A scan of publicly available food datasets highlights three recurring gaps: (i) insufficient diversity and scale, (ii) limited annotation, “monolithic” labels with little schema depth (e.g., no portion/nutrition), and (iii) curated imagery that diverges from real-world photos. MM-Food-100K addresses these limitations by combining breadth with multi-level, structured annotations tailored to downstream AI tasks. 2.2 Schema (what a record contains) Each item includes an image link and a JSON metadata block with the following key fields (non-exhaustive): dish_name , food_type (homemade/restaurant/packaged/raw), ingredients , portion_size (ingredient-level estimates), nutritional_profile (kcal/protein/fat/carbs when available), cooking_method , and authenticity indicators ( camera_or_phone_prob , online_download_prob , food_prob ). A visual representation of these annotations, which can be a mix of human and AI labels, is provided in Figure 2 , and the core schema fields are detailed in Table 1 . This design supports multi-task learning (classification, extraction, and regression) without changing file formats. Figure 2 : Selected Samples from the Food Intelligence Dataset: Images with Mixed Human and AI Annotations. Examples of food images with mixed annotations—green cells are human-labeled, red cells are AI-predicted—showing dish details, ingredients, cooking methods, and nutrition. Table 1 : Core schema fields Group Example fields Purpose Identity dish_name, food_type, cuisine tag recognition / taxonomy Ingredients ingredients[] extraction / reasoning Portion portion_size (g/ml per item) portion / calorie models Nutrition nutritional_profile (kcal, P/F/C) nutrition prediction Preparation cooking_method method reasoning Authenticity camera_or_phone_prob, online_download_prob, food_prob data realism filters 2.3 Codatta’s Two-Stage, AI-Augmented Data Sourcing Workflow Creating high-quality, large-scale multimodal datasets from diverse community sources is challenging due to the trade-off between quality assurance and operational costs. To address this, we designed and implemented a novel two-stage, AI-augmented data sourcing workflow. As illustrated in Figure 3 , this pipeline is a core innovation of the Codatta protocol, turning heterogeneous user-generated content into verifiable, high-fidelity records at scale. Our method consists of the following key steps: (1) Submission and Progressive Enrichment : Initially, a Human Data Provider submits an image and corresponding annotations. These annotations follow a structured schema and taxonomy (L1-L5), allowing for progressive enrichment from basic dish names to complex, evidence-backed nutritional and portion cues. (2) Initial Quality Review : Each submission proceeds to an Initial Quality Review stage, where an Automated LVM Review conducts a rapid vetting process. This first-pass check filters out low-quality or irrelevant submissions and provides near-real-time feedback. If a submission is rejected, the data provider can revise and resubmit. (3) Final Quality Review : Submissions that pass the initial review are forwarded to a more thorough Final Quality Review . This stage leverages a more sophisticated, Programmed LVM to perform detailed checks, ensuring the accuracy and integrity of the data. (4) Data Curation and Distribution : Accepted and validated data is then processed for Data Curation and deduplication. It is subsequently split into two distinct sets: a Public Access Subset (10%) for the research community and a Commercial Access Subset (90%) for commercial use. This dual-access distribution model retains wallet-linked provenance, laying the foundation for future royalty-based rewards. This two-stage, AI-augmented workflow achieves high-quality data acquisition at scale and a sustainable balance between cost and data quality. It establishes a transparent process and a reward mechanism that benefits both community contributors and downstream applications. Figure 3 : Data Pipeline: Two-Stage Quality Review and Dual-Access Distribution. A workflow for reviewing food image–annotation pairs with quick initial checks, detailed final reviews, and a split into public and commercial sets to balance openness and provider rewards. 2.4 Descriptive statistics (what the open subset looks like) The dataset card reports 100,000 images with a broad mix of source contexts (‘food_type’), illustrating real-world capture conditions and variety. The distribution of ‘food_type’ is as follows: Homemade (46,555), Restaurant (35,461), Raw vegetables & fruits (9,357), Packaged