YOLOE: Real-Time Seeing Anything
Ao Wang, Lihao Liu, Hui Chen, Zijia Lin, Jungong Han, Guiguang Ding · 2025
Read YOLOE: Real-Time Seeing Anything by Ao Wang, Lihao Liu, Hui Chen, Zijia Lin, Jungong Han, Guiguang Ding (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07465.txt.
Introduction
A detector that sees anything, in real time, from words, pictures, or no prompt at all is the pitch of YOLOE, a 2025 arXiv paper by Ao Wang and co-authors. Classic detectors only know their training labels. Open ended rivals usually pay for freedom with speed or messy deployment.
Three tricks keep this one fast. Text prompts are aligned through a small add on network that vanishes at run time. Picture prompts split into meaning and activation branches. A built in large vocabulary handles the prompt free case without calling a language model. On standard tests the system showed strong zero shot skill at a third of the usual training cost.
Catalog record
| CREATOR | Ao Wang, Lihao Liu, Hui Chen, Zijia Lin, Jungong Han, Guiguang Ding |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-07465
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
YOLOE: Real-Time Seeing Anything Ao Wang, Lihao Liu, Hui Chen, Zijia Lin, Jungong Han, Guiguang Ding Abstract Object detection and segmentation are widely employed in computer vision applications, yet conventional models like YOLO series, while efficient and accurate, are limited by predefined categories, hindering adaptability in open scenarios. Recent open-set methods leverage text prompts, visual cues, or prompt-free paradigm to overcome this, but often compromise between performance and efficiency due to high computational demands or deployment complexity. In this work, we introduce YOLOE, which integrates detection and segmentation across diverse open prompt mechanisms within a single highly efficient model, achieving real-time seeing anything. For text prompts, we propose Re-parameterizable Region-Text Alignment (RepRTA) strategy. It refines pretrained textual embeddings via a re-parameterizable lightweight auxiliary network and enhances visual-textual alignment with zero inference and transferring overhead. For visual prompts, we present Semantic-Activated Visual Prompt Encoder (SAVPE). It employs decoupled semantic and activation branches to bring improved visual embedding and accuracy with minimal complexity. For prompt-free scenario, we introduce Lazy Region-Prompt Contrast (LRPC) strategy. It utilizes a built-in large vocabulary and specialized embedding to identify all objects, avoiding costly language model dependency. Extensive experiments show YOLOE’s exceptional zero-shot performance and transferability with high inference efficiency and low training cost. Notably, on LVIS, with 3 × \times less training cost and 1.4 × \times inference speedup, YOLOE-v8-S surpasses YOLO-Worldv2-S by 3.5 AP. When transferring to COCO, YOLOE-v8-L achieves 0.6 AP b and 0.4 AP m gains over closed-set YOLOv8-L with nearly 4 × \times less training time. Code and models are available at https://github.com/THU-MIG/yoloe . 1 Introduction Object detection and segmentation are foundational tasks in computer vision [ 48 , 15 ] , with widespread applications spanning autonomous driving [ 2 ] , medical analyses [ 55 ] , and robotics [ 8 ] , etc . Traditional approaches like YOLO series [ 47 , 1 , 21 , 3 ] , have leveraged convolutional neural networks to achieve real-time remarkable performance. However, their dependence on predefined object categories constrains flexibility in practical open scenarios. Such scenarios increasingly demand models capable of detecting and segmenting arbitrary objects guided by diverse prompt mechanisms, such as texts, visual cues, or without prompt. Figure 1 : Comparison of performance, training cost, and inference efficiency between YOLOE (Ours) and advanced YOLO-Worldv2 in terms of open text prompts. LVIS AP is evaluated on minival set and FPS w/ TensorRT and w/ CoreML is measured on T4 GPU and iPhone 12, respectively. The results highlight our superiority. Given this, recent efforts have shifted towards enabling models to generalize for open prompts [ 49 , 20 , 80 , 5 ] . They target single prompt type, e.g . , GLIP [ 32 ] , or multiple prompt types in a unified way, e.g . , DINO-X [ 49 ] . Specifically, with region-level vision-language pretraining [ 32 , 37 , 65 ] , text prompts are usually processed by text encoder to serve as contrastive objectives for region features [ 49 , 20 ] , achieving recognition for arbitrary categories, e.g . , YOLO-World [ 5 ] . For visual prompts, they are often encoded as class embeddings tied to specified regions for identifying similar objects, by the interaction with image features or language-aligned visual encoder [ 19 , 30 , 49 , 5 ] , e.g . , T-Rex2 [ 20 ] . In prompt-free scenario, existing methods typically integrate language models, finding all objects and generating the corresponding category names conditioned on region features sequentially [ 49 , 62 ] , e.g . , GenerateU [ 33 ] . Despite notable advancements, a single model that supports diverse open prompts for arbitrary objects with high efficiency and accuracy is still lacking. For example, DINO-X [ 49 ] features a unified architecture, which, however, incurs resource-intensive training and inference overhead. Additionally, individual designs for different prompts in separate works exhibit suboptimal trade-offs between performance and efficiency, making it difficult to directly combine them into one model. For example, text-prompted approaches often incur substantial computational overhead when incorporating large vocabularies, due to complexity of cross-modality fusion [ 5 , 37 , 32 , 49 ] . Visual-prompted methods usually compromise deployability on edge devices owing to the transformer-heavy design or reliance on additional visual encoder [ 20 , 30 , 67 ] . Prompt-free ways, meanwhile, depend on large language models, introducing considerable memory and latency costs [ 33 , 49 ] . In light of these, in this paper, we introduce YOLOE(ye), a highly efficient , unified , and open object detection and segmentation model, like human eye, under different prompt mechanisms, like texts, visual inputs, and prompt-free paradigm. We begin with YOLO models with widely proven efficacy. For text prompts, we propose a Re-parameterizable Region-Text Alignment (RepRTA) strategy, which employs a lightweight auxiliary network to improve pretrained textual embeddings for better visual-semantic alignment. During training, pre-cached textual embeddings require only the auxiliary network to process text prompts, incurring low additional cost compared with closed-set training. At inference and transferring, auxiliary network is seamlessly re-parameterized into the classification head, yielding an architecture identical to YOLOs with zero overhead. For visual prompts, we design a Semantic-Activated Visual Prompt Encoder (SAVPE). By formalizing regions of interest as masks, SAVPE fuses them with multi-scale features from PAN to produce grouped prompt-aware weights in low dimension in an activation branch and extract prompt-agnostic semantic features in a semantic branch. Prompt embeddings are derived through aggregation of them, resulting in favorable performance with minimal complexity. For prompt-free scenario, we introduce Lazy Region-Prompt Contrast (LRPC) strategy. Without relying on costly language models, LRPC leverages a specialized prompt embedding to find all objects and a built-in large vocabulary for category retrieval. By matching only anchor points with identified objects against the vocabulary, LRPC ensures high performance with low overhead. Thanks to them, YOLOE excels in detection and segmentation across diverse open prompt mechanisms within one model, enjoying high inference efficiency and low training cost. Notably, as shown in Fig. 1 , under 3 \times less training cost, YOLOE-v8-S significantly outperforms YOLO-Worldv2-S [ 5 ] by 3.5 AP on LVIS [ 14 ] , with 1.4 \times and 1.3 \times inference speedups on T4 and iPhone 12, respectively. In visual-prompted and prompt-free settings, YOLOE-v8-L outperforms T-Rex2 by 3.3 AP r and GenerateU by 0.4 AP with 2 \times less training data and 6.3 \times fewer parameters, respectively. For transferring to COCO [ 34 ] , YOLOE-v8-M / L outperforms YOLOv8-M / L by 0.4 / 0.6 AP b and 0.4 / 0.4 AP m with nearly 4 \times less training time. We hope that YOLOE can establish a strong baseline and inspire further advancements in real-time open prompt-driven vision tasks. 2 Related Work Traditional detection and segmentation. Traditional approaches for object detection and segmentation primarily operate under closed-set paradigms. Early two-stage frameworks [ 12 , 48 , 15 , 4 ] , exemplified by Faster R-CNN [ 48 ] , introduce region proposal networks (RPNs) followed by region-of-interest (ROI) classification and regression. Meanwhile, single-stage detectors [ 38 , 35 , 56 , 72 , 10 ] prioritizes speed through grid-based predictions within a single network. The YOLO