SnappyMeal: Design and Longitudinal Evaluation of a Multimodal AI Food Logging Application

Liam Bakar, Zachary Englhardt, Vidya Srinivas, Girish Narayanswamy, Dilini Nissanka, Shwetak Patel, Vikram Iyer · 2025

Read SnappyMeal: Design and Longitudinal Evaluation of a Multimodal AI Food Logging Application by Liam Bakar, Zachary Englhardt, Vidya Srinivas, Girish Narayanswamy, Dilini Nissanka, Shwetak Patel, Vikram Iyer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-03907.txt.

Introduction

Liam Bakar and six colleagues designed SnappyMeal, a multimodal artificial intelligence app for logging food. Interviews with nutrition experts and regular trackers showed paper and app journals are inflexible, weakly followed, and often inaccurate. Flexible logging matters because diet records underpin links between food, medicine, fitness, and health.

SnappyMeal accepts several input types, asks goal-dependent follow-up questions when context is missing, and pulls detail from grocery receipts and nutrition databases. Benchmark tests were joined by a three-week real-world deployment with over five hundred logged items. Users praised the input options and perceived accuracy strongly, suggesting a more context-aware path for dietary tracking tools.

Catalog record

CREATORLiam Bakar, Zachary Englhardt, Vidya Srinivas, Girish Narayanswamy, Dilini Nissanka, Shwetak Patel, Vikram Iyer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-03907

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SnappyMeal: Design and Longitudinal Evaluation of a Multimodal AI Food Logging Application Liam Bakar, Zachary Englhardt, Vidya Srinivas, Girish Narayanswamy, Dilini Nissanka, Shwetak Patel, Vikram Iyer Abstract . Food logging, both self-directed and prescribed, plays a critical role in uncovering correlations between diet, medical, fitness, and health outcomes. Through conversations with nutritional experts and individuals who practice dietary tracking, we find current logging methods, such as handwritten and app-based journaling, are inflexible and result in low adherence and potentially inaccurate nutritional summaries. These findings, corroborated by prior literature, emphasize the urgent need for improved food logging methods. In response, we propose SnappyMeal, an AI-powered dietary tracking system that leverages multimodal inputs to enable users to more flexibly log their food intake. SnappyMeal introduces goal-dependent follow-up questions to intelligently seek missing context from the user and information retrieval from user grocery receipts and nutritional databases to improve accuracy. We evaluate SnappyMeal through publicly available nutrition benchmarks and a multi-user, 3-week, in-the-wild deployment capturing over 500 logged food instances. Users strongly praised the multiple available input methods and reported a strong perceived accuracy. These insights suggest that multimodal AI systems can be leveraged to significantly improve dietary tracking flexibility and context-awareness, laying the groundwork for a new class of intelligent self-tracking applications. Figure 1 . Traditional food logging using handwritten diaries or even mobile applications rely heavily on manual data entry or suffer from poor accuracy estimation techniques. We instead develop a smartphone-based multimodal AI system that combines diverse multimodal context from food and receipt images to natural language text and audio with interactive follow up questions to improve tracking flexibility and contextual awareness. Demonstrating antiquated food logging against novel food logging. Demonstrating antiquated food logging, using scales, calculators, and notebooks, against novel food logging, using mobile devices, multiple modalities, and artificial intelligence. 1. Introduction It is well studied that proper nutrition and eating habits are correlated with healthier living and reduced risk for a number of conditions ( 20 ) . However, what constitutes healthy diet and dietary patterns varies greatly from individual to individual, and is influenced by factors such cultural and demographic background, geographic location, pre-existing conditions, and food intolerance/sensitivities. The definition of healthy diet is further convoluted by factors such as self-image, societal pressures, socioeconomic status, and personal goals. As such there is not a “one-size-fits-all” solution to individual eating habits. To this end, food logging, both self-directed and prescribed, plays a critical role in uncovering correlations between diet, medical, fitness, and health outcomes for individuals ( 12 ; 40 ) . Nutritional tracking tools have emerged as a way to bring more awareness and structure to eating habits. These tools enable the 69% of U.S. adults keeping track of at least one health indicator (weight, diet, exercise regimen, or symptom) ( 15 ) to better monitor intake, set goals, and reflect on patterns using smartphone and wearable based logging. However, unlike fitness trackers (e.g., smartwatches) that can automatically measure health metrics (e.g., step counts, heart rate), modern nutritional tracking tools, both handwritten and app-based, largely depend on manual inputs. To better understand the current landscape of nutrition tracking apps, we conduct a formative study with nine participants spanning both professional dietitians and food journalers (those who log diet). Our results show that usability and convenience, psychological impact, long-term sustainability, and the risk of tracking fatigue (a common phenomenon in which users disengage due to the effort and mental load of constant self-monitoring ( 10 ) ) all contribute to low adherence and tracking accuracy ( 37 ; 16 ) . Notably, users seek intuitive and flexible systems that align better with their personalized goals and lifestyle. Designing a single tracking solution that will seamlessly fit into diverse users’ lifestyles is challenging, as each individual has varying dietary needs and nutrition tracking objectives. A person aiming for muscle gain has different nutritional needs (e.g., higher protein intake) than someone trying to manage cholesterol or lose weight ( 38 ) . Dietary guidelines provided by organizations such as the FDA are not specific or tailored enough to provide nutritional guidance for each individual ( 1 ; 29 ) . While many apps exist for self tracking (MyFitnessPal, LoseIt, etc), our formative study finds they fall short when estimating portion sizes and matching food items ( 7 ; 16 ) . Many tools do not log important context about food preparation or seasonings, and struggle on the challenging problem of estimating specific nutrients. Recent advances in artificial intelligence (AI) open up new opportunities to reimagine how nutrition tracking systems can be designed and experienced. In particular, multimodal foundation models can jointly process text, images, and audio, allowing users to describe meals in natural language, upload photos of food or receipts, or record brief voice notes. These capabilities enable more flexible and personalized methods of logging that may better fit diverse lifestyles, contexts, and abilities. At first glance, such technology appears to offer a straightforward solution to the long-standing challenges of nutritional tracking. However, deploying AI-powered systems reveals that many core issues remain unsolved. Even with multimodal inputs, models often lack the contextual awareness needed to make accurate inferences: photos may be taken at poor angles, ingredients may be occluded, or voice notes may omit crucial details. The challenge, then, is how to fill in this missing context without increasing user burden or disrupting the tracking experience. To explore this opportunity and its associated challenges, we develop SnappyMeal , an AI-powered nutrition tracking application that integrates multimodal inputs, retrieval-augmented context, and adaptive interactivity. SnappyMeal surfaces the tensions at the intersection of AI, HCI, and system design, balancing automation with user flexibility and context-seeking, and highlights that simply “adding AI to the loop” does not automatically solve the difficult problems of nutrition tracking. Instead, meaningful progress depends on understanding what users value, how they wish to engage with AI assistance, and how systems can adapt to individual goals and contexts without imposing additional cognitive or interactional load. We introduce three complementary strategies for context augmentation in AI-powered nutrition tracking: (1) leveraging RAG to retrieve not only structured nutritional data but also visually and semantically similar food images, enhancing contextual understanding, (2) incorporating receipts to provide additional context about what the user actually bought, reducing reliance on potentially incomplete images or descriptions, and (3) selectively generating follow-up questions that directly elicit missing information from users in a goal-directed manner. Together, these strategies aim to improve flexibility and personalization of food logging while mitigating user effort and cognitive load. We summarize our contributions below: • We conduct a formative study identifying both the key gaps in current leading food tracking technologies and opportunities for AI systems to close these gaps. • We develop an end-to-end mobile and cloud system that supports diverse multimodal inputs—images, natural language text, and speech—to capture meal context, and introduce the first AI system that interactively queries users to fill in missing details. • We investigate these AI-powered features through a real-world, three-week longitudinal study yielding over 500 food logs revealing both progress and key challenges in using AI to create flexible, context-awareness nutrition tracking systems. 2. Related Work 2.1. Nutrition Tracking For Health Dietary tracking can help individuals with

Read the complete text