RT-DETRv2 Explained in 8 Illustrations

Ethan Qi Yang CHUA, Jen Hong TAN Data Science and Artificial Intelligence Lab Singapore General Hospital · 2025

Read RT-DETRv2 Explained in 8 Illustrations by Ethan Qi Yang CHUA, Jen Hong TAN Data Science and Artificial Intelligence Lab Singapore General Hospital (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01241.txt.

Introduction

Object detection models are notoriously hard to picture, and RT-DETRv2, a real-time detector, is no exception. Ethan Qi Yang Chua and Jen Hong Tan set out to explain its design through eight carefully drawn illustrations. Existing diagrams, they argue, do little to show how the parts actually work together.

The article moves from the whole pipeline down to key pieces, including the encoder, the decoder, and multi-scale deformable attention. It visualizes how tensors flow through the system and unpacks the logic behind each module. Chua and Tan aim to give researchers and practitioners a clear mental model of the detector under the hood. The piece is a teaching guide rather than a new model, built to make a complex design genuinely understandable.

Catalog record

CREATOREthan Qi Yang CHUA, Jen Hong TAN Data Science and Artificial Intelligence Lab Singapore General Hospital
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01241

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RT-DETRv2 Explained in 8 Illustrations Ethan Qi Yang CHUA, Jen Hong TAN Data Science and Artificial Intelligence Lab Singapore General Hospital Abstract ABSTRACT Object detection architectures are notoriously difficult to understand, often more so than large language models. While RT-DETRv2 [ 1 ] represents an important advance in real-time detection, most existing diagrams do little to clarify how its components actually work and fit together. In this article, we explain the architecture of RT-DETRv2 through a series of eight carefully designed illustrations, moving from the overall pipeline down to critical components such as the encoder, decoder, and multi-scale deformable attention. Our goal is to make the existing one genuinely understandable. By visualizing the flow of tensors and unpacking the logic behind each module, we hope to provide researchers and practitioners with a clearer mental model of how RT-DETRv2 works under the hood. 1. INTRODUCTION Object detection plays a critical role in computer vision, with models such as YOLO [ 2 ] achieving efficiency by framing detection as a single-pass regression task over dense anchor boxes, followed by heuristic post-processing like non-maximum suppression (NMS). While fast, these approaches often struggled with spatial context and multi-scale objects. The introduction of DETR (DEtection TRansformer) [ 3 ] marked a shift: object detection was reformulated as a set-prediction problem, where global attention and bipartite matching directly produce object queries. Although elegant, DETR converges slowly and has difficulty with small objects due to its reliance on global attention. RT-DETRv2 [ 4 ] was designed to overcome these limitations. It incorporates deformable attention, hybrid encoding, denoising, query-based decoding, and other refinements to achieve real-time performance without anchor heuristics. Despite these advances, the architecture of RT-DETRv2 remains complex, and existing diagrams in the literature rarely make the tensor flow or design choices easy to grasp. In this article, we aim to demystify RT-DETRv2. Using eight carefully designed illustrations, we progressively break down the model—starting from the overall pipeline, through the encoder and decoder, and down to intricate components like multi-scale deformable attention. Our goal is not to introduce new methods, but to make an important model accessible by visualizing how its parts truly fit together. 2. METHOD The RT-DETRv2 architecture can be divided into five main components: the CNN backbone, hybrid encoder, denoising task, query selection, and decoder. In this article, ResNet50 [ 5 ] is the CNN backbone, and we follow the default configuration described in the original paper/code in Transformers library, using input images of size (640,640) , 300 object queries, and 100 denoising queries (though the actual number of denoising bounding boxes, denoted by d from Figure to 4 to 8 , varies across batches). We use B to denote batch size. Figure 1 illustrates the overall tensor flow in RT-DETRv2 during inference, where the denoising task is disabled. Figure 1 : The structure of an RT-DETRv2 model with denoising task off. 2.1 CNN Backbone The input image first passes through a ResNet-based embedding layer followed by several ResNet stages [ 5 ] , producing three multi-scale feature maps of sizes (B,256,80,80) , (B,256,40,40) , and (B,256,20,20) . These feature maps capture information at different spatial resolutions: the 80\times 80 map preserves fine-grained spatial detail but carries relatively shallow semantic context, whereas the 20\times 20 map provides coarse spatial resolution with richer semantic representation. The intermediate 40\times 40 map strikes a balance between the two, supporting effective multi-scale feature integration in later stages of the model. 2.2 Hybrid Encoder The hybrid encoder [ 6 ] consists of an encoder and a series of fusion pathways (Figure 2 ). The encoder performs self-attention [ 7 ] only on the feature map with lowest resolution and highest semantic richness to understand relationships and identify object features globally. The fusion pathways perform top-down and bottom-up feature aggregation [ 8 ] through the use of upsampling, downsampling, concatenation, and Cross Stage Partial Networks [ 9 ] to ensure that high resolution features also contain deep semantic meaning. Figure 2 : The structure of the hybrid encoder. The hybrid encoder is split into two distinct components: the encoder which performs normal self-attention, and the fusion pathways as seen by the Top-Down Feature Pyramid and Bottom-Up Path Aggregation Networks. The Top-Down Feature Pyramid and Bottom-Up Aggregation Network are fusion pathways which aim to produce feature maps which are high in semantic richness and spatial resolution. For the Top-Down Feature Pyramid, the (B,256,20,20) tensor is returned by the encoder and upsampled to size (40,40) . It is concatenated with the (B,256,40,40) tensor from ResNet Stage 2 (Figure 1 ) resulting in 512 feature maps (256+256) . This is passed through a Cross Stage Partial (CSP) Network (Figure 3 ) to output a tensor of shape (B,256,40,40) . The process repeats, where the tensor is upsampled and concatenated with the (B,256,80,80) feature maps from ResNet Stage 3 to result again in 512 feature maps. It is passed through another CSP Net to output a tensor of shape (B,256,80,80) . The Bottom-Up Path Aggregation Network operates similar to the Top-Down Feature Pyramid, just that it downsamples the tensors instead. The (B,256,80,80) tensor is then downsampled to size (40,40) and concatenated with the earlier (B,256,40,40) CSP Net output to produce a tensor with 512 maps. The tensor goes through the same process to output shape (B,256,40,40) . These steps are repeated to output another tensor of shape (B,256,20,20) . The resultant tensors of size (80,80) , (40,40) and (20,20) each have 256 channels. These fused representations are the semantically and spatially rich outputs of the hybrid encoder for the rest of the model. Figure 3 : Left is the structure of the encoder, performing rather straightforward self-attention on the input tensor. Right is structure of the Cross Stage Partial Network which utilises RepVGG blocks [ 10 ] to allow for faster convergence and fewer vanishing gradients in deep neural networks. 2.3 Query Selection The purpose of query selection is to reduce the dense set of candidate features into a smaller set of informative queries for the decoder. Once the (B,256,80,80) , (B,256,40,40) , and (B,256,20,20) feature maps have been returned by the hybrid encoder, they are concatenated and reshaped to a (B,8400,256) tensor where 8400 represent the spatial locations across all scales and 256 represent the number of channels for each spatial location. A valid mask of shape (1,8400,1) is then applied through element-wise multiplication to the tensor in order to omit any invalid spatial locations. The tensor is then passed through a linear layer to map feature embeddings to classification logits, resulting in shape (B,8400,256) . Afterwards, the tensor is passed through a classification branch and a localisation branch. The classification branch passes the tensor through a linear layer resulting in (B,8400,80) whereby the prediction ranges between 80 object classes. The (B,8400,80) tensor is then passed through a TopK algorithm to gather the top 300 queries which likely contain important information to be input into the decoder. The localisation branch is passed through a linear layer resulting in (B,8400,4) where the last dimension represents the coordinates (x_{1},y_{1},x_{2},y_{2}) . Anchors are added to the tensor at the localisation branch before the TopK queries are filtered out. Query selection ensures that only the most relevant

Read the complete text