Multimodal ML: Quantifying the Improvement of Calorie Estimation Through Image-Text Pairs

Arya Narang · 2025

Read Multimodal ML: Quantifying the Improvement of Calorie Estimation Through Image-Text Pairs by Arya Narang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11705.txt.

Introduction

Arya Narang measures whether adding a dish's name to its photo helps a neural network estimate calories. Nutrition tracking from images alone is error-prone, and a short text label is nearly free to supply, so even a small gain could matter at scale.

Using the Nutrition5k dataset curated by Google, an image-only convolutional network is compared with a multimodal twin that also reads the dish name. Mean absolute error fell by just over one kilocalorie, about a one and a quarter percent improvement. The gain is real but modest, and the study frames the honest question: when does a second input earn its place in a deployed estimator?

Catalog record

CREATORArya Narang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-11705

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multimodal ML: Quantifying the Improvement of Calorie Estimation Through Image-Text Pairs Arya Narang Abstract 1 Introduction Various developed countries have experienced a continuous rise in obesity [1]. This can be burdensome on healthcare and has thus prompted governments to introduce laws and regulations to restaurants and food chains in an attempt to promote healthier eating [1]. One patent rule is the mandatory display of calories in menus, which has not come without an added cost to businesses. For example, Section 4205 of the Affordable Care Act (ACA) in the USA required certain food chains to display caloric information on menu items which was estimated to cost $315 million to comply [2]. These costs were primarily due to nutritional analysis - if one could drastically diminish this overhead, it would greatly lower business spending [2]. This creates the motivation for programming an AI that can accurately estimate the number of calories in a dish simply through an image and text prompt. If successful, business owners could use merely a smartphone camera to rapidly perform these calculations over manually studying the ingredients of each dish. Currently, there exists a myriad of apps that utilize the power of AI to perform calorie estimation [3]. This study investigates inexpensive methods in an attempt to maximize the accuracy of these predictions - if another technique produced extremely precise results at the cost of a lot of time and money, then the benefit would be negated. Another key motivation of this paper stems from a recent university project of mine. I collaborated in a group of 4 to develop an Android app called BiteBack which allows users to discover local food businesses and track multiple loyalty schemes on one app. Businesses are able to display their menus in the app but a feature is needed to allow calories to be visible too. Similar to the primary incentive, we want to minimize the cost for businesses to display calories and instead allow them to produce estimates using alternative, cheaper models. Two techniques are compared in this paper: calorie estimation using an image-only prompt and calorie estimation using a multimodal model with image and text inputs. The objective is to quantify the improvement in predictions when utilizing the latter model over the unimodal one. 2 Related Work 2.1 CaLoRAify Last year, Yao, D et al. combined MiniGPT-4v2, a vision-language model (VLM), with LoRA and RAG to predict the ingredients and calorific value of food items [4]. They curated a dataset of approximately 330,000 image-text pairs and trained their model to a high precision [4]. Despite the improved accuracy, MiniGPT-4v2 requires copious amounts of RAM (up to 23GB) [7] which is not suitable for the BiteBack app. This is because any costs to utilize the AI on a server would have to be forwarded to business owners which counteracts the aim to minimize their expenses. Moreover, the model solely relies on an image of food to estimate calories and does not consider if the dish name or a brief description of the food as added input can enhance precision [4]. Instead, a lightweight, deployable model is trained in this paper to allow users to give both a dish name and image to estimate the calories in their meals rapidly. 2.2 Encoder-Decoder Frameworks Ma et al. produced a paper in 2023 that improves on previous encoder-decoder frameworks developed in the past to estimate the energy in food [5]. Their model consists of segmentation masks that capture the positioning of food in an image, isolating it from the background and refining feature extraction [5]. One similarity between this model and CaloRAify is that it relies only on image data and fails to consider how a short textual input can further reduce the MAE in predictions. 2.3 Large-Scale Recipe Datasets Ruede, R. et al. demonstrated how incorporating both text and image inputs can minimize the error in calorie estimation in their 2020 paper [6]. This research strongly relates to the methods used in this project as it is one of the few papers that quantifies the improvement in calorie prediction through an additional text input [6]. While this was an effective approach, the model they trained relied on lengthy inputs that detailed the ingredients and recipe of each dish [6]. This can be highly inconvenient for business owners who may need to approximate the calorific value of numerous menu items - in this scenario, quick estimations are a priority. Hence this paper aims to quantify the improvement brought by small text inputs over detailed recipes. 3 Methods 3.1 Dataset In this project, the Nutrition5k dataset was used and split into a training and testing dataset [8]. Nutrition5k was created by taking videos of 5,000 dishes (of which 3̃300 are used in this study), with certain frames from each video extracted to produce a dataset containing images [8]. In addition, nutritional information regarding each food item is stored (e.g. calories, fat content, dish name). Thus, the following data is stored for each dish: a video; images selected from the video to capture the dish from different angles; depth photos; nutritional information; name; ingredients [8]. However, it must be noted that the Nutrition5k dataset was developed by visiting food places in Los Angeles that focused on western cuisines [8]. As a result, Asian, African, Middle Eastern and other cuisines were not considered during the curation of this dataset. 3.2 Pre-processing Dataset All images processed for training and testing were retrieved by running a gsutil script on a local machine (which downloaded birds-eye view images of each dish and stored them as PNGs) and any textual data (e.g. dish names, calories) was collected by downloading the relevant CSV files. Nevertheless, Nutrition5k contains a lot of extraneous information that needed to be filtered. Only the following data was relevant to this project: images, dish names and calories. This required use of the pandas library in Python to filter the CSV file and extract details from the relevant columns, storing the data as a Pandas DataFrame . After filtering the CSV files, the images were processed. In order to add further diversity to the set of pictures, a script was run that iterated through each image and randomly inverted it and altered its brightness and contrast within a 10% range (so that the brightness and contrast of each photo would be within 90% and 110% of its original value). Finally, the image paths and textual data were stored as a TensorFlow dataset, ready to train the CNNs. 3.3 Building Image-Only CNN In order to boost the precision of the model whilst ensuring it was lightweight and deployable, the MobileNetV2 CNN was incorporated as the base model [9]. For the regression head, it was essential that non-linear high-level features were captured for calorie estimation which 2 dense layers coupled with ReLU activation and dropout layers helped extract. A dropout layer randomly zeros a specified proportion of outgoing nodes from the previous layer to prevent overfitting [10] and a ReLU activation function ensures any outputs by a node that are negative are mapped to 0 [9] - both of these features prevent the neural network from acting as merely a stacked linear function. Figure 1: Overview of proposed image-only

Read the complete text