Break Out the Silverware: Semantic Understanding of Stored Household Items Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus CCS: Computing methodologies Knowledge representation and reasoning CCS: Compu

Michaela Levi Richter, Reuth Mirsky, Oren Glickman · 2025

Read Break Out the Silverware: Semantic Understanding of Stored Household Items Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus CCS: Computing methodologies Knowledge representation and reasoning CCS: Compu by Michaela Levi Richter, Reuth Mirsky, Oren Glickman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23739.txt.

Introduction

Michaela Levi-Richter, Reuth Mirsky, and Oren Glickman set a homely test for household robots: someone says bring me a plate. The robot must guess where plates are kept, usually hidden in a drawer or cabinet. Robots can see and grasp, but they lack the everyday commonsense that tells humans where things belong. The authors formalize this as the Stored Household Item Challenge.

Their benchmark combines a small real-world set of one hundred item photographs with human-annotated answers from participants' kitchens, and a larger development set of thousands of annotated public kitchen images. Their system, NOAM, turns the visual scene into a language description of the space and its visible containers, then asks a large language model to infer the hidden item's likely home. NOAM beat vision-language baselines by a wide margin and approached human accuracy.

Catalog record

CREATORMichaela Levi Richter, Reuth Mirsky, Oren Glickman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-23739

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Break Out the Silverware: Semantic Understanding of Stored Household Items Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus CCS: Computing methodologies Knowledge representation and reasoning CCS: Computing methodologies Computer vision CCS: Computing methodologies Artificial intelligence Michaela Levi Richter, Reuth Mirsky, Oren Glickman Abstract . “Bring me a plate.” For domestic service robots, this simple command reveals a complex challenge: inferring where everyday items are stored, often out of sight in drawers, cabinets, or closets. Despite advances in vision and manipulation, robots still lack the commonsense reasoning needed to complete this task. We introduce the Stored Household Item Challenge , a benchmark task for evaluating service robots’ cognitive capabilities: given a household scene and a queried item, predict its most likely storage location. Our benchmark includes two datasets: (1) a real-world evaluation set of 100 item-image pairs with human-annotated ground truth from participants’ kitchens, and (2) a development set of 6,500 item-image pairs annotated with storage polygons over public kitchen images. These datasets support realistic modeling of household organization and enable comparative evaluation across agent architectures. To begin tackling this challenge, we introduce NOAM (Non-visible Object Allocation Model) , a hybrid agent pipeline that combines structured scene understanding with large language model inference. NOAM converts visual input into natural language descriptions of spatial context and visible containers, then prompts a language model (e.g., GPT-4) to infer the most likely hidden storage location. This integrated vision-language agent exhibits emergent commonsense reasoning and is designed for modular deployment within broader robotic systems. We evaluate NOAM against baselines including random selection, vision-language pipelines (Grounding-DINO + SAM), leading multimodal models (e.g., Gemini, GPT-4o, Kosmos-2, LLaMA, Qwen), and human performance. NOAM significantly improves prediction accuracy and approaches human-level results, highlighting best practices for deploying cognitively capable agents in domestic environments. C. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.) 1. Introduction Service robots are increasingly capable of navigating and manipulating objects in household environments. Yet, they remain limited in a fundamental way: commonsense reasoning about where items are typically stored when those items are not visible. For humans, such inference is effortless. When asked to fetch a mug, we do not search the refrigerator or under the sink; instead, we rely on deeply shared expectations about household organization. Translating this intuition into machine reasoning remains an unsolved problem. We introduce the Stored Household Item Challenge , a new benchmark designed to evaluate a specific form of commonsense reasoning: predicting the likely storage location of household items that are not visible. Given an indoor scene and a queried item, the task is to infer which storage container (such as a drawer, cabinet, or closet) is most likely to contain the item, despite it being hidden from view. Unlike traditional object detection, which focuses on visible entities, this task requires reasoning about concealed items based on indirect visual cues and prior semantic knowledge. We hypothesize that household storage follows consistent patterns (for example, mugs are usually stored in upper cabinets rather than drawers) that can be learned and generalized. Figure 1. Kitchen scene shown to Gemini, with its predicted spoon storage location highlighted in the bounding box. Kitchen scene shown to Gemini, with its predicted spoon storage location highlighted in the bounding box. Figure 2. Kitchen scene shown to GPT, with its predicted spoon storage location highlighted in the bounding box. Kitchen scene shown to GPT, with its predicted spoon storage location highlighted in the bounding box. Current benchmarks overlook this reasoning dimension. Vision-Language Models (VLMs) and object detectors perform well on visible items but fail when asked to infer the invisible. For instance, we asked Gemini and GPT to identify where a spoon is most likely stored, using the following prompt: In the given image, which drawer or cabinet is the most likely place to store a spoon? Return the image with the exact drawer or cabinet clearly marked. Do not modify any other part of the image. As shown in Figures 1 and 2 , both models struggled to select a plausible container. To our knowledge, no prior structured benchmark exists for this ability. Existing datasets lack labels for expected-but-not-visible items, and current models are not trained to reason about functional storage norms or household organization. This gap highlights both a scientific challenge and a practical barrier for embodied agents operating in real homes. To support this task, we construct two datasets: (1) a 6,500-pair development set based on real kitchen scenes (SUN dataset), with container annotations by humans; and (2) a 100-example evaluation set collected from real homes. These datasets enable scalable training and robust testing across varied environments. We benchmark random baselines, VLMs, Multimodal Large Language Models (MLLMs), and human predictions (Figure 3 , top and middle left). We also introduce NOAM (Non-visible Object Allocation Model), a novel language-based pipeline that reframes the task as structured text-based reasoning (Figure 3 , bottom left). By describing scenes and candidate storage options in natural language and querying LLMs, NOAM significantly outperforms VLMs, though still below human performance. This suggests that language-driven inference offers a promising path for equipping agents with commonsense reasoning skills. Rather than chasing marginal detection gains, our aim is to formalize a reasoning capability absent in today’s models: context-aware inference about what is unseen but likely. The Stored Household Item Challenge captures this missing skill that is crucial for robots in cluttered, unfamiliar spaces. As LLMs evolve, we expect progress, but the challenge will remain a meaningful test of real-world understanding. To summarize, the contributions of this work are threefold: 1. New challenge: the Stored Household Item Challenge. 2. An annotation pipeline and dataset of item-image pairs grounded in real household scenes. 3. NOAM (Non-visible Object Allocation Model), a language-based model that reframes visual inference as a structured textual reasoning task. We compare it against baselines, including random choice, VLM, MLLMs and humans. Figure 3. Depiction of the Stored Household Item Challenge (right). We compare the performance of various models on this task (left), including grounded vision-language models, multimodal LLMs, and our model, NOAM. Depiction of the Stored Household Item Challenge (right). We compare the performance of various models on this task (left), including grounded vision-language models, multimodal LLMs, and our model, NOAM. 2. Related Work Our research lies at the intersection of three areas: domestic service robots, scene understanding, and object detection. While our work introduces a new approach to reasoning about hidden items, it builds on established methods across these domains, particularly recent advances in vision, vision-language, and MLLMs to enhance semantic understanding in complex domestic environments. Domestic Service Robots Research on cleaning robots has expanded beyond vacuuming to include service robots capable of tidying, organizing groceries, washing dishes, and setting tables. Prior work focuses on learning processes tailored to specific homes ( 14 ; 20 ; 37 ) and on user-instructed behaviors ( 16 ; 19 ; 28 ; 35 ; 41 ; 42 ; 46 ) . Some approaches use GNNs to learn user preferences from observations ( 18 ) or exploit object hierarchies and collaborative filtering ( 2 ) . However, most pick-and-place tasks involve visible surfaces like shelves or tables ( 15 ;

Read the complete text