RF-DETR: Neural Architecture Search for Real-Time Detection Transformers

Isaac Robinson, Peter Robicheaux, Matvei Popov, Deva Ramanan, Neehar Peri · 2025

Read RF-DETR: Neural Architecture Search for Real-Time Detection Transformers by Isaac Robinson, Peter Robicheaux, Matvei Popov, Deva Ramanan, Neehar Peri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-09554.txt.

Introduction

Isaac Robinson and four co-authors search automatically for object detectors that are both fast and accurate on a user's own dataset. Big open-vocabulary detectors often stumble on domains unlike their pretraining data. Their RF-DETR approach skips fine-tuning a heavy vision-language model. It fine-tunes one base network, then explores thousands of architectures sharing its weights.

Weight-sharing neural architecture search traces the accuracy-latency trade-off curve without retraining each candidate. The nano version reaches about 48 average precision on COCO in their tests. It beats its nearest real-time rival by more than five points at similar speed. The largest version tops 60 average precision on COCO, a first for real-time detectors, they report. It also beats a grounding model on Roboflow100-VL while running twenty times faster.

Catalog record

CREATORIsaac Robinson, Peter Robicheaux, Matvei Popov, Deva Ramanan, Neehar Peri
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-09554

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RF-DETR: Neural Architecture Search for Real-Time Detection Transformers Isaac Robinson, Peter Robicheaux, Matvei Popov, Deva Ramanan, Neehar Peri Abstract Open-vocabulary detectors achieve impressive performance on COCO, but often fail to generalize to real-world datasets with out-of-distribution classes not typically found in their pre-training. Rather than simply fine-tuning a heavy-weight vision-language model (VLM) for new domains, we introduce RF-DETR, a light-weight specialist detection transformer that discovers accuracy-latency Pareto curves for any target dataset with weight-sharing neural architecture search (NAS). Our approach fine-tunes a pre-trained base network on a target dataset and evaluates thousands of network configurations with different accuracy-latency tradeoffs without re-training . Further, we revisit the “tunable knobs” for NAS to improve the transferability of DETRs to diverse target domains. Notably, RF-DETR significantly improves on prior state-of-the-art real-time methods on COCO and Roboflow100-VL. RF-DETR (nano) achieves 48.0 AP on COCO, beating D-FINE (nano) by 5.3 AP at similar latency, and RF-DETR (2x-large) outperforms GroundingDINO (tiny) by 1.2 AP on Roboflow100-VL while running 20 × 20\times as fast. To the best of our knowledge, RF-DETR (2x-large) is the first real-time detector to surpass 60 AP on COCO. Our code is available on GitHub . 1 Introduction Object detection is a fundamental problem in computer vision that has matured in recent years ( 12 ; 22 ; 35 ) . Open-vocabulary detectors like GroundingDINO ( 24 ) and YOLO-World ( 10 ) achieve remarkable zero-shot performance on common categories like car , truck , and pedestrian . However, state-of-the-art vision-language models (VLMs) still struggle to generalize to out-of-distribution classes, tasks and imaging modalities not typically found in their pre-training ( 36 ) . Fine-tuning VLMs on a target dataset significantly improves in-domain performance at the cost of runtime efficiency (due to heavy-weight text encoders) and open-vocabulary generalization. In contrast, specialist (i.e., closed-vocabulary) object detectors like D-FINE ( 30 ) and RT-DETR ( 52 ) achieve real-time inference, but underperform fined-tuned VLMs like GroundingDINO. In this paper, we modernize specialist detectors by combining internet-scale pre-training with real-time architectures to achieve state-of-the-art performance and fast inference. Are Specialist Detectors Over-Optimized for COCO? Sustained progress in object detection can be largely attributed to standardized benchmarks like PASCAL VOC ( 11 ) and COCO ( 22 ) . However, we find that recent specialist detectors implicitly overfit to COCO at the cost of real-world performance using bespoke model architectures, learning rate schedulers, and augmentation schedulers. Notably, state-of-the-art object detectors like YOLOv8 ( 17 ) generalize poorly to real-world datasets with significantly different data distributions from COCO (e.g., number of objects per image, number of classes, and dataset size). To address these limitations, we present RF-DETR, a scheduler-free approach that leverages internet-scale pre-training to generalize to real-world data distributions. To better specialize our model for diverse hardware platforms and dataset characteristics, we revisit neural architecture search (NAS) in the context of end-to-end object detection and segmentation. Rethinking Neural Architecture Search (NAS) for DETRs. NAS discovers accuracy-latency tradeoffs by exploring architectural variants within a pre-defined search space. NAS has been previously studied in the context of image classification ( 43 ; 4 ) and for model sub-components like detector backbones 44 and FPNs 14 . Unlike prior work, we explore end-to-end weight-sharing NAS for object detection and segmentation. Our key insight, inspired by OFA ( 4 ) , is that we can vary model inputs like image resolution, and architectural components like patch size during training. Further, weight-sharing NAS allows us to modify inference configurations like the number of decoder layers and query tokens to specialize our strong base model without fine-tuning . We evaluate all model configurations with grid search on a validation set. Importantly, our approach does not evaluate the search space until the base model has been fully-trained on the target dataset. As a result, all possible sub-nets (i.e., model configurations within the search space) achieve strong performance without further fine-tuning, significantly reducing the computational cost of optimizing for new hardware. Interestingly, we find that sub-nets not explicitly seen during training still achieve high performance, suggesting that RF-DETR can generalize to unseen architectures (cf. H ). Extending RF-DETR for segmentation is also relatively straightforward and only requires adding a lightweight instance segmentation head. We denote this model as RF-DETR-Seg. Notably, this allows us to also leverage end-to-end weight-sharing NAS to discover Pareto optimal architectures for real-time instance segmentation. Standardizing Latency Evaluation. We evaluate our approach on COCO ( 22 ) and Roboflow100-VL (RF100-VL) ( 36 ) and achieve state-of-the-art performance among real-time detectors. RF-DETR (nano) outperforms D-FINE (nano) by 5% AP on COCO at comparable run-times, and RF-DETR (2x-lage) beats GroundingDINO (tiny) on RF100-VL at a fraction of the runtime. RF-DETR-Seg (nano) outperforms YOLOv11-Seg (x-large) on COCO while running 4 \times as fast. However, comparing RF-DETR’s latency with prior work remains challenging because reported latency evaluation varies significantly between papers. Notably, each new model re-benchmarks the latency of prior work for fair comparison on their hardware. For example, D-FINE’s reported latency evaluation of LW-DETR ( 8 ) is 25% faster than originally reported. We identify that this lack of reproducibility can be primarily attributed to GPU power throttling during inference. We find that buffering between forward passes limits power over-draw and standardizes latency evaluation (cf. Table 1 ). Contributions. We present three major contributions. First, we introduce RF-DETR, a family of scheduler-free NAS-based detection and segmentation models that outperform prior state-of-the-art on RF100-VL ( 36 ) and real-time methods with latencies \leq 40 ms on COCO ( 22 ) (cf. Fig. 1 ). To the best of our knowledge, RF-DETR is the first real-time detector to exceed 60 mAP on COCO. Next, we explore the “tunable-knobs” for weight-sharing NAS to improve accuracy-latency tradeoffs for end-to-end object detection (cf. Fig. 3 ). Notably, our use of a weight-sharing NAS allows us to leverage large-scale pre-training and effectively transfer to small datasets (cf. Tab. 4 ). Lastly, we revisit current benchmarking protocols for measuring latency and propose a simple standardized procedure to improve reproducibility. Figure 1: Accuracy-Latency Pareto Curve. We plot the Pareto accuracy-latency frontier for real-time detectors on the COCO detection val-set (top left, bottom left), COCO segmentation val-set (top right), and RF100-VL test-set (bottom right). Since RF100-VL contains 100 distinct datasets, we select target latencies for the N, S, M, L, XL, 2XL configurations, search for RF-DETR models with latencies within 10% of the target and report their average performance after fine-tuning to convergence. Importantly, all points along RF-DETR’s continuous Pareto curves for COCO are derived from a single training run. 2 Related Works Neural Architecture Search (NAS) automatically identifies families of model architectures with different accuracy-latency tradeoffs ( 55 ; 56 ; 33 ; 3 ) . Early NAS approaches ( 55 ; 33 ) focused primarily on maximizing accuracy, with little consideration for efficiency. As a result, discovered architectures (e.g., NASNet and AmoebaNet) were often computationally expensive. More recent hardware-aware NAS methods ( 5 ; 42 ; 47 ) address this limitation by incorporating hardware feedback directly into the search process. However, these methods must repeat the search and training process for each new hardware platform. In contrast, OFA ( 4 ) proposes a weight-sharing NAS that decouples training and search by simultaneously

Read the complete text