Mitigating Cross-Modal Distraction and Ensuring Geometric Feasibility via Affordance-Guided, Self-Consistent MLLMs for Food Preparation Task Planning
Yu-Hong Shen Chuan-Yu Wu Yi-Ru Yang Yen-Ling Tai Yi-Ting Chen · 2025
Read Mitigating Cross-Modal Distraction and Ensuring Geometric Feasibility via Affordance-Guided, Self-Consistent MLLMs for Food Preparation Task Planning by Yu-Hong Shen Chuan-Yu Wu Yi-Ru Yang Yen-Ling Tai Yi-Ting Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13055.txt.
Introduction
Five authors test large multimodal models as planners for robot food prep tasks in 2025. Good plans must judge amounts, reach, place, and crash risks at once, yet past tests split those skills apart. Their new QuARC test joins all four in one kitchen scene.
Two flaws showed up in current models, drift across image and text inputs, and plans that break shape rules. Fixes join self-checking thought chains with a tool that flags what can be grasped or reached. The method hit a 76.7 percent pass rate against 36.7 percent for a strong rival, with no added fine-tuning.
Catalog record
| CREATOR | Yu-Hong Shen Chuan-Yu Wu Yi-Ru Yang Yen-Ling Tai Yi-Ting Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13055
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mitigating Cross-Modal Distraction and Ensuring Geometric Feasibility via Affordance-Guided, Self-Consistent MLLMs for Food Preparation Task Planning Yu-Hong Shen Chuan-Yu Wu Yi-Ru Yang Yen-Ling Tai Yi-Ting Chen Abstract We study Multimodal Large Language Models (MLLMs) with in-context learning for food preparation task planning. In this context, we identify two key challenges: cross-modal distraction and geometric feasibility. Cross-modal distraction occurs when the inclusion of visual input degrades the reasoning performance of a MLLM. Geometric feasibility refers to the ability of MLLMs to ensure that the selected skills are physically executable in the environment. To address these issues, we adapt Chain of Thought (CoT) with Self-Consistency to mitigate reasoning loss from cross-modal distractions and use affordance predictor as skill preconditions to guide MLLM on geometric feasibility. We construct a dataset to evaluate the ability of MLLMs on quantity estimation, reachability analysis, relative positioning and collision avoidance. We conducted a detailed evaluation to identify issues among different baselines and analyze the reasons for improvement, providing insights into each approach. Our method reaches a success rate of 76.7% on the entire dataset, showing a substantial improvement over the CoT baseline at 36.7%. I INTRODUCTION Imagine walking into a high-end tofu pudding shop where a service robot is tasked with preparing customized dessert orders. A table in front of the robot is set with bowls of tofu pudding and various toppings. Customers interact with the robot using natural language, specifying their preferences, such as ‘Place some mung beans onto my tofu pudding.’ To fulfill these requests, the robot must accurately interpret human instructions, ground the specified food items, and perform the corresponding manipulation skills to complete tofu pudding preparation. Interpreting human instructions and converting them into the corresponding action sequences remains a challenge in robotics. Task planning [ 3 , 4 , 5 , 6 , 7 , 8 ] decomposes a given goal specification into subtasks based on the available robot skill set. To handle goal specifications expressed in natural language, Large Language Models (LLMs) have been investigated for their ability to generate structured plans from textual input. More recently, Multimodal Large Language Models (MLLMs) have extended LLMs by incorporating additional modalities, such as visual inputs, enabling them to perceive and interpret the world. In this work, we study MLLMs with in-context learning for food preparation task planning. Due to the difficulties in recovering from spillage in real-world settings and the lack of datasets in simulation that resemble our food preparation scenario, we construct our dataset in a simulated environment to evaluate the abilities of MLLMs. Specifically, we choose IsaacGym [ 9 ] as our simulator due to its high-fidelity in rendering a wide range of food items. We identify cross-modal distraction and geometric feasibility as the main challenges to applying MLLMs for food preparation task planning. Reasoning across different modalities has been an active area of research in the community [ 10 , 11 ] . However, challenges such as visual hallucination and illusion highlight that incorporating image inputs can sometimes produce unintended or misleading results [ 12 , 13 ] . Our experiments reveal that for tasks not requiring visual input, adding images to the same prompt distracts the MLLM, degrading its reasoning performance. We term this phenomenon cross-modal distraction. For example, in tasks that require only semantic reasoning, incorporating an image observation causes MLLMs to generate infeasible skill sequences, overlook previous actions, or scoop twice despite being instructed to scoop only once. Additionally, we find that MLLMs struggle with reasoning about geometric feasibility, which involves ensuring that the planned sequence of robot actions is physically executable within the spatial constraints of the environment. For example, in Figure , MLLM fails to recognize that there is a bowl in front of the dumbwaiter door and attempts to open the dumbwaiter directly. A natural question arises: how can we address the limitations of MLLMs in food preparation task planning within an in-context learning setting? Our approach uses Chain of Thought with self-consistency to counter performance drops from cross-modal distractions and leverages skill affordance to provide geometric feasibility information for MLLM. We study task planning under closed-loop setting, where the robot iteratively selects actions using the new image input. Chain of Thought prompting [ 20 ] helps MLLM to reason through problems step by step, leading to better responses. Self-Consistency Verification in MLLM checks whether the model generates reliable outputs across multiple runs, enabling it to detect anomalies and maintain stable performance. Skill affordance refers to the likelihood that a skill can be successfully executed given a specific environment state and acts as the action precondition for executing certain skills. We encode crucial information for identifying geometric feasibility as preconditions, enabling verification of whether the decisions made by MLLM are feasible. Our contributions are as follows: 1. We construct a dataset designed for task planning in food preparation, categorizing different scenarios to evaluate various robotic capabilities. 2. We adapt CoT with Self-Consistency to closed-loop task planning settings to combat reasoning loss caused by cross-modal distractions 3. We build up action preconditions with predicates to enable MLLM to take geometric feasibility into consideration. II RELATED WORK II-A Large Models in Robotics As Large Language Models (LLMs) continue to advance, several studies [ 4 , 5 , 6 , 7 ] demonstrate how LLMs can be leveraged to follow natural language instructions. However, it is crucial to bridge the information gap between textual input and environmental observations when using LLMs. For tasks that require context-specific judgments, perception models are needed to extract relevant information to supply the LLM as decision-making criteria. Since the reference information for natural language instructions can vary greatly across instances, exhaustively listing all possible references is both cumbersome and time-consuming. MLLMs can process image input and extract relevant information through prompting. They are commonly used as success detectors or precondition checkers, or scene descriptors [ 6 , 16 , 17 , 18 , 19 ] . Most of them are used in the form of visual question answering. Most of them are used in the form of visual question answering, whereas we aim for MLLMs to autonomously extract and implicitly utilize this information for planning accurate action sequences. II-B Reasoning Ability of Large Models There are many studies that explore ways to enhance the reasoning ability of large models. We focus on two categories, Chain of Thought (CoT) and Self-Consistency. CoT [ 20 ] guides LLMs to break down problems into step-by-step reasoning for better results. Several variants exist [ 21 , 22 ] , with zero-shot CoT [ 23 ] being the most relevant to our work. The main difference is that we combine the two-stage pipeline into a single prompt and we prompt the model to describe the goal the robot needs to achieve in the current iteration instead of using the magic prompt ”Let’s think step by step.” Self-Consistency methods employ a ”sample and select” approach, where large models generate several possible answers and determine the final output with majority vote [ 24 , 25 ] . We extend this concept to closed-loop task planning in robotics, using the output of previous iterations as samples to assess the LLM’s consistency at the current iteration. \includegraphics [width=]img/pipeline_example.png Fig. 2: Overview of