Identifying the Unknown: Prompt-Free Open Vocabulary Anomaly Recognition for Robot-Object Interaction

Philipp Allgeuer, Jan-Gerrit Habekost, Stefan Wermter · 2026

Read Identifying the Unknown: Prompt-Free Open Vocabulary Anomaly Recognition for Robot-Object Interaction by Philipp Allgeuer, Jan-Gerrit Habekost, Stefan Wermter (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26829.txt.

Introduction

Philipp Allgeuer, Jan-Gerrit Habekost, and Stefan Wermter present AnomNOVIC, a system that lets robots recognise objects they have never been told about. Open-vocabulary detectors usually need a text prompt or a candidate list. Continuous robot deployment cannot stop to write prompts. Their two-stage framework works in a known workspace without any prompt.

An anomaly detector, trained as a masked autoencoder, first proposes boxes around anything object-like. A fast open-vocabulary classifier then names the contents of each box. No candidate class list is needed. Tests used a tabletop setup with the NICOL humanoid robot. The system beat strong open-vocabulary baselines by wide margins. On a wild test set with forty-eight objects, prompt-free detection and naming reached high accuracy.

Catalog record

CREATORPhilipp Allgeuer, Jan-Gerrit Habekost, Stefan Wermter
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.26829

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Identifying the Unknown: Prompt-Free Open Vocabulary Anomaly Recognition for Robot-Object Interaction Philipp Allgeuer, Jan-Gerrit Habekost, Stefan Wermter Abstract Robots operating in real-world environments must in general be able to recognize previously unseen objects. As robotic systems move toward open-world autonomy, there is a growing, yet largely unmet, need for open vocabulary object detectors that are prompt-free and efficient enough for continuous deployment. We present AnomNOVIC, a two-stage known-workspace framework that combines a masked autoencoder (MAE) trained for anomaly detection, with NOVIC, a powerful real-time prompt-free open vocabulary image classifier. The MAE produces generic object-agnostic bounding boxes, allowing NOVIC to classify salient image regions without requiring a predefined candidate class list. We evaluate AnomNOVIC against strong open vocabulary baselines in a tabletop robot-object environment featuring the NICOL humanoid robot, reaching 47.1%​ AP / 57.5%​ AP 50 for prompt-free recognition, and 59.0%​ AP / 72.5%​ AP 50 if class candidates are provided. Across additional datasets, including an in-the-wild test set with 48 unique objects, AnomNOVIC reaches up to 82.6% prompt-free detection and classification accuracy. These results significantly surpass all tested open vocabulary baselines, including YOLO-World-v2, OWLv2, and YOLOE. 1 Introduction Robotic agents driven by Large Language Models (LLMs) are transforming human-robot interaction, enabling natural dialogue, reasoning, and task planning. These high-level capabilities, however, often remain bottlenecked by a robot’s real-time visual perception. Open vocabulary object detectors are a promising approach to this challenge, as they enable zero-shot discovery and identification of previously unseen objects encountered at runtime, especially in unstructured or dynamic environments. Existing open vocabulary object detectors, however, come at varying levels of true ‘openness’. Many state-of-the-art detectors, including ViLD [ 8 ] , YOLO-World [ 5 ] , and OWLv2 [ 12 ] , require a list of candidate object classes at inference time (‘prompted’ models), which is a strong constraint that severely limits their practical utility. If a robot can only detect objects it has explicitly been instructed to look for, then it cannot actually recognize any novel objects in practice, undermining the core supposed advantage of open vocabulary detection. Prompted models also tend to fail ungracefully in the presence of novel objects, frequently producing false positives. Open vocabulary models that do not require any list of candidate classes are referred to as prompt-free [ 18 ] . A second common limitation of state-of-the-art open vocabulary detectors is that many are trained or fine-tuned on datasets of limited concept coverage. Even if a model is technically ‘open vocabulary’, it will still fail to detect concepts that are insufficiently represented in its training data. Thus, if a training dataset contains only a few thousand unique object concepts, any resulting model will in practice have a rather limited active vocabulary at inference time. This issue is exacerbated by the need for substantial examples per concept to facilitate effective learning and generalization. As a reference, the RAM model [ 22 ] constructed its vocabulary by parsing and expanding upon 14M training captions, ultimately arriving at only 4585 tags—including many non-object terms, and only after manually adding many missing terms ( \approx 1K) from known classification datasets. If we compare this to the open vocabulary prompted YOLO-World [ 5 ] and prompt-free YOLOE [ 18 ] models, both trained on under 1.6M images, we can see that the ‘openness’ they can achieve is strongly limited. Among current open vocabulary models, our previous work NOVIC [ 1 ] is rather unique in being capable of real-time prompt-free image classification over a near-unconstrained taxonomy of object nouns ( \approx 43K). However, while NOVIC excels at assigning fine-grained free-form classifications to images as a whole, it can neither classify nor localize multiple individual objects within an image, which is crucial for real applications. In this work we present AnomNOVIC, an integration of NOVIC’s prompt-free classification capabilities into an open vocabulary object detection pipeline, and apply it to the tabletop scenario of the NICOL robot (see Fig.​ 1 ). A key challenge is how to identify suitable regions of interest (RoIs) without introducing class or vocabulary biases. AnomNOVIC addresses this by training a transformer-based anomaly detection masked autoencoder (MAE) [ 9 ] to model the robot’s object-free operational workspace as a normative baseline, and flagging reconstruction anomalies at inference time as object region proposals. Additional anomaly mask and table mask output channels further enhance the quality of these proposals. The final obtained RoIs are cropped from the input images and passed to NOVIC, which assigns accurate fine-grained free-form labels. By decoupling object localization from classification, AnomNOVIC leverages the axiomatic notion of “anything that does not belong” to discover objects of any shape, texture, or novelty in an unbiased manner. 2 Related Work Figure 1: Left: The LLM-powered NICOL is used in a human-robot-object interaction scenario requiring open vocabulary object detection. Right: A two-stage anomaly recognition model, AnomNOVIC, unites an anomaly detection masked autoencoder (MAE) with a strong prompt-free open vocabulary image classification model (NOVIC). 2.1 Open Vocabulary Object Detection Open vocabulary object detectors can be broadly divided into prompted and prompt-free approaches. ViLD [ 8 ] was among the first to use pretrained vision-language models for open detection, passing prompt-templated class names through a CLIP text encoder [ 14 ] to form classification weights for region proposals from a distilled class-agnostic backbone. Other prompted detectors such as RegionCLIP [ 23 ] , DetCLIPv3 [ 19 ] , and Grounding DINO [ 11 ] extend this idea by aligning regional features with text embeddings and/or unifying detection and phrase grounding via multimodal pretraining. For fast inference, YOLO-World [ 5 ] adapts the efficient single-stage anchor-free YOLOv8 backbone and adds a reparameterizable vision-language path aggregation network. Models like OVR-CNN [ 20 ] and OWL-ViT [ 13 ] fine-tune detection heads on datasets of restricted diversity and rely on contrastive image-text pretraining for implicit generalization. OWLv2 [ 12 ] scales this up by self-training on web-scale image-text data pseudo-labeled by OWL-ViT, but remains non-real-time and still requires explicit class prompts at inference. The self-training mainly consolidates concepts OWL-ViT can already recognize, rather than effectively learning genuinely novel categories. While these methods can to some degree recognize arbitrary classes specified at inference time, they still require a user-supplied class vocabulary, and their performance depends heavily on the scope of the underlying training data. YOLOE [ 18 ] seeks to alleviate the first of these constraints by supporting a “prompt-free” mode. This prompt-free mode frames open vocabulary classification as a retrieval task based on a fixed mid-sized reference vocabulary. A special prompt embedding selects generic class-agnostic object proposals, and the corresponding object embeddings are matched against the complete built-in vocabulary for category retrieval. This fixed vocabulary is exactly the limited 4585 RAM tags [ 22 ] discussed in Sec.​ 1 . The use of this limited fixed vocabulary, combined with training on datasets of limited concept coverage, imposes an upper bound on YOLOE’s true ‘openness’. NOVIC [ 1 ] is a recent model that achieves prompt-free open vocabulary image classification by leveraging the knowledge of a pretrained CLIP model. It has dictionary-level coverage of the English language, with its classification decoder trained exclusively on content-controllable synthetic text datasets. The model, however, cannot localize objects or classify multiple of them per image. NOVIC

Read the complete text