OmniFood8K: Single-Image Nutrition Estimation via Hierarchical Frequency-Aligned Fusion
Dongjian Yu, Weiqing Min, Qian Jiang, Xing Lin, Xin Jin, Shuqiang Jiang, [0.5em] School of Software, Yunnan University, China State Key Laboratory of AI SafetyInstitute of Computing Technology, Chinese Academy of Sciences, China · 2026
Read OmniFood8K: Single-Image Nutrition Estimation via Hierarchical Frequency-Aligned Fusion by Dongjian Yu, Weiqing Min, Qian Jiang, Xing Lin, Xin Jin, Shuqiang Jiang, [0.5em] School of Software, Yunnan University, China State Key Laboratory of AI SafetyInstitute of Computing Technology, Chinese Academy of Sciences, China (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12356.txt.
Introduction
Yu and five colleagues introduce OmniFood8K, a food dataset of 8,036 samples with detailed nutrition labels and multi-view images, filling thin coverage of Chinese dishes and freeing methods from depth sensors. They add NutritionSynth-115K, a large synthetic set with varied mixes and exact labels. Their single-image system predicts a depth map, refines it with a scale-shift residual adapter, fuses RGB and depth traits in the frequency domain, and predicts nutrition through a mask-based head. Tests show it beats existing approaches.
Catalog record
| CREATOR | Dongjian Yu, Weiqing Min, Qian Jiang, Xing Lin, Xin Jin, Shuqiang Jiang, [0.5em] School of Software, Yunnan University, China State Key Laboratory of AI SafetyInstitute of Computing Technology, Chinese Academy of Sciences, China |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30820 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.12356
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OmniFood8K: Single-Image Nutrition Estimation via Hierarchical Frequency-Aligned Fusion Dongjian Yu, Weiqing Min, Qian Jiang, Xing Lin, Xin Jin, Shuqiang Jiang, [0.5em] School of Software, Yunnan University, China State Key Laboratory of AI SafetyInstitute of Computing Technology, Chinese Academy of Sciences, China Abstract Accurate estimation of food nutrition plays a vital role in promoting healthy dietary habits and personalized diet management. Most existing food datasets primarily focus on Western cuisines and lack sufficient coverage of Chinese dishes, which restricts accurate nutritional estimation for Chinese meals. Moreover, many state-of-the-art nutrition prediction methods rely on depth sensors, restricting their applicability in daily scenarios. To address these limitations, we introduce OmniFood8K, a comprehensive multimodal dataset comprising 8,036 food samples, each with detailed nutritional annotations and multi-view images. In addition, to enhance models’ capability in nutritional prediction, we construct NutritionSynth-115K, a large-scale synthetic dataset that introduces compositional variations while preserving precise nutritional labels. Moreover, we propose an end-to-end framework for nutritional prediction from a single RGB image. First, we predict a depth map from a single RGB image and design the Scale-Shift Residual Adapter (SSRA) to refine it for global scale consistency and local structural preservation. Second, we propose the Frequency-Aligned Fusion Module (FAFM) to hierarchically align and fuse RGB and depth features in the frequency domain. Finally, we design a Mask-based Prediction Head (MPH) to emphasize key ingredient regions via dynamic channel selection for more accurate prediction. Extensive experiments on multiple datasets demonstrate the superiority of our method over existing approaches. Project homepage: https://yudongjian.github.io/OmniFood8K-food/ 1 Introduction Figure 1 : Representative examples from the OmniFood8K dataset. With the improvement of living standards and the diversification of dietary habits, nutritional assessment of food has become a key research focus in both public health and personal health management 28 ; 57 ; 38 . Accurate estimation of food energy and nutrient intake plays a vital role in preventing chronic diseases such as obesity, diabetes, and cardiovascular disorders 22 ; 45 . However, traditional nutritional assessment methods 59 ; 35 ; 48 often rely on manual recording or expert estimation, which are not only time-consuming and labor-intensive but also prone to subjective bias 20 ; 32 . This makes accurate and consistent evaluation challenging, particularly in real-world daily dietary applications 14 ; 37 . Recent advances in computer vision and multimodal learning have significantly advanced the development of automated food nutrition assessment 42 ; 43 ; 44 ; 69 ; 10 ; 11 . Deep learning techniques have shown great potential in automatically recognizing, segmenting, analyzing food images to estimate food mass, volume, and nutritional composition 58 ; 16 ; 47 . Moreover, leveraging multimodal features such as textual descriptions and sensor data can further enhance estimation accuracy and robustness, facilitating personalized nutrition monitoring and healthcare systems 53 ; 8 . Despite these advances, current methods still encounter two fundamental limitations that constrain their practical usability. (1) Data Limitation. Existing datasets are heavily biased toward Western cuisines, with limited representation of Chinese food. The inherent diversity, complex ingredient composition, and non-standardized preparation of Chinese food make reliable annotation and quantitative nutritional analysis particularly challenging. (2) Algorithmic Limitation. Many advanced approaches rely on depth maps captured by depth cameras for accurate nutrition estimation. Nevertheless, in most real-world contexts, food photographs are taken with conventional RGB cameras rather than depth sensors, thereby restricting the practical adoption of these methods. To address these two limitations, this work makes two key contributions. First, we introduce OmniFood8K, a comprehensive multimodal food nutrition dataset designed for real-world scenarios, as shown in Figure 1 . The dataset covers the entire food preparation process, including raw ingredient images with mass information, structured recipe descriptions, full cooking videos, and multi-view images of the finished dishes, all paired with detailed nutritional annotations, as illustrated in Figure 2 . Unlike existing datasets, OmniFood8K employs a unified, process-oriented design that preserves causal links from ingredient preparation to final presentation and provides comprehensive nutritional annotations. In addition, to further improve the model’s nutritional prediction capability, we construct NutritionSynth-115K, a large-scale synthetic dataset with compositional diversity and precise nutritional annotations. Second, we propose an end-to-end framework that predicts nutritional information directly from a single RGB image. Compared with depth-sensor-based approaches, our method is more generalizable and scalable for real-world dietary assessment. Specifically, we first employ a pre-trained depth estimation model 62 to predict the depth information of food. To correct scale bias and local structural errors, we design a Scale-Shift Residual Adapter (SSRA) for consistent global and local calibration. We then employ a Frequency-Aligned Fusion Module (FAFM) to hierarchically fuse RGB and adapted depth features, aligning multi-modal representations in the frequency domain for enhanced cross-modal learning. Finally, a Mask-based Prediction Head (MPH) leverages dynamic channel selection and region-aware attention to emphasize key ingredient regions, improving nutrition prediction accuracy. Overall, our contributions are summarized as follows: • We introduce OmniFood8K, a comprehensive multimodal food dataset with detailed nutritional annotations, covering ingredients, recipes, cooking videos, and multi-view food images. Additionally, we construct NutritionSynth-115K, a large-scale synthetic dataset featuring diverse food compositions and precise nutritional annotations. • We propose an end-to-end framework for predicting food nutritional information from a single RGB image. It incorporates a Scale-Shift Residual Adapter (SSRA) to refine depth estimation and achieve consistent calibration of both global scale and local geometry. • We further design a Frequency-Aligned Fusion Module (FAFM) to hierarchically fuse RGB and depth features in the frequency domain, and a Mask-based Prediction Head (MPH) that dynamically selects informative channels, thereby enhancing the accuracy of nutritional prediction. • Extensive experiments on multiple datasets demonstrate that our method outperforms state-of-the-art approaches, validating its effectiveness and scalability for food nutrition assessment. 2 Related work Figure 2 : Overview of the OmniFood8K dataset: data collection process and category distribution. 2.1 Food Datasets Recent advances in food computing have led to the release of numerous publicly available datasets 15 ; 1 , which have substantially promoted research across multiple domains, such as food recognition, cross-modal analysis, and nutritional estimation 12 ; 40 ; 4 ; 52 . Representative datasets such as Food-101 5 , VIREO Food-172 6 , ISIA Food-500 30 , and Food2K 31 cover diverse food categories and have been widely adopted for recognition and classification tasks 68 ; 67 ; 36 . Additionally, Recipe1M 41 aligns food images with textual recipes, serving as a fundamental resource for cross-modal retrieval and recipe generation tasks. Nutrition5k 53 includes detailed nutritional annotations and has been instrumental in advancing research on food nutrition assessment. However, its primary focus on Western dishes, with limited coverage of complex cuisines such as Chinese food, constrains its general applicability. FoodSeg103 61 is a dataset designed for food image segmentation, aiming to facilitate fine-grained understanding of food components. MetaFood3D 7 is a multimodal dataset comprising 637 3D food objects across 108 categories, accompanied by detailed nutritional information and diverse visual modalities. The FastFood dataset 37 collects food images and their corresponding nutritional information from official fast-food brand websites, providing valuable data for nutrition analysis and related studies. 2.2 Food Nutrition Assessment Traditional nutrition assessment methods are time-consuming, labor-intensive, and often limited in predictive accuracy 9 ; 54 ; 39 . With advances in computer vision, AI-based approaches for nutrition prediction