Improved Vision-to-Chart Buoy Association with Learned World-to-Image Projection

Borja Carrillo-Perez · 2026

Read Improved Vision-to-Chart Buoy Association with Learned World-to-Image Projection by Borja Carrillo-Perez (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22942.txt.

Introduction

Carrillo-Perez improves vision-to-chart buoy association for the MaCVi 2026 challenge with a small change to a DETR-based fusion baseline. The baseline forced the transformer to learn the world-to-image projection implicitly from distance and bearing queries. Instead, a dedicated QueryMLP network predicts each buoy's waterline contact point in the picture from chart data and IMU orientation. The added spatial prior lifted the entry to second place, with an overall score of 0.7386.

Catalog record

CREATORBorja Carrillo-Perez
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE11956
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.22942

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improved Vision-to-Chart Buoy Association with Learned World-to-Image Projection Borja Carrillo-Perez Abstract This report presents a lightweight modification to the DETR-based fusion transformer baseline for the MaCVi 2026 Vision-to-Chart data association challenge. The challenge baseline decoder receives per-buoy queries encoding world-space distance and bearing, forcing the transformer to implicitly learn the complex geometric projection from world coordinates to image pixels. Instead, in this work, an additional dedicated MLP (QueryMLP) is trained to explicitly predict the buoy’s waterline contact point in the image from chart measurements and IMU orientation data. The predicted pixel coordinates are appended to the query vector of the baseline decoder, providing a direct spatial prior per buoy and reducing the geometric reasoning burden on the transformer decoder. On the challenge leaderboard the presented approach achieves an Overall score of 0.7386 0.7386 ( F 1 = 0.8055 F_{1}=0.8055 , mIoU = 0.6718 \mathrm{mIoU}=0.6718 ) on the held-out test set, placing second among all submissions. Code is available at https://github.com/bcarrpe/macvi26-visionmap-querymlp . 1 Introduction The MaCVi 2026 Vision-to-Chart challenge 1 1 1 https://macvi.org/workshop/cvpr/challenges/vision_map requires detecting buoys in maritime camera images and matching each detection to its corresponding entry in a nautical chart. Each chart entry provides the buoy’s GPS-derived distance and bearing relative to the vessel, while an Inertial Measurement Unit (IMU) file records pitch, roll, heading, and position at capture time. The evaluation metric is the mean of F_{1} -score and \mathrm{mIoU} (Overall). The challenge baseline [ 4 ] modifies DETR [ 1 ] by replacing its fixed learned object queries with a variable number of chart-derived embeddings. Per-buoy features (normalized distance and bearing) are passed through a small multilayer perceptron (MLP) to produce 256 -dimensional decoder inputs. The transformer decoder then cross-attends these embeddings against ResNet-50 image features to predict visibility and bounding box location for each chart entry. Because the query encodes only world-space measurements, the decoder must implicitly learn the geometric projection from world coordinates to image pixels; this mapping depends on the vessel’s instantaneous pitch, roll, and heading. The presented approach addresses this directly, inspired by prior work of Carrillo-Perez et al. in maritime ship recognition and georeferencing [ 2 , 3 ] . They demonstrate that a learned mapping can bridge world coordinates and pixel coordinates from paired training data alone, without explicit camera calibration. QueryMLP is trained to learn this world-to-image projection using distance, inverse of distance, bearing, and IMU orientation as inputs. Its output serves as a spatial prior that tells the decoder approximately where each buoy should appear in the image, thereby reducing what the decoder must discover from scratch to appearance-based refinement. A further insight from Carrillo-Perez et al. [ 2 , 3 ] motivates the choice of prediction target: the waterline contact point of a maritime object is the most geometrically meaningful image-space location, since it corresponds directly to the object’s position on the water plane. For buoys, this point is approximated by the bottom-center edge of the bounding box [c_{x},\;c_{y}+h/2] , used as the MLP output. 2 Method 2.1 QueryMLP: Learned World-to-Image Projection QueryMLP is trained independently of the main DETR pipeline to predict the normalized buoy waterline contact point from six features derived from the chart query and IMU files. Unlike the baseline, which forms decoder queries from normalized distance and bearing alone, QueryMLP also uses the available IMU orientation measurements to learn the world-to-image projection: • Normalized distance (d/1000) : scale-invariant range prior. • Inverse distance (clipped to [0,10] ): nonlinear depth cue that responds sharply to nearby buoys. • Normalized bearing (\beta/180) : horizontal image position prior. • Pitch (normalized by 10^{\circ} ): shifts the expected vertical position with vessel pitch. • Roll (normalized by 10^{\circ} ): accounts for lateral horizon tilt. • Heading (normalized by 180^{\circ} ): global bearing context. The normalization constants reflect the physical operating ranges of each quantity. Distance is divided by 1000 m, the challenge’s maximum range threshold, mapping the full operating range to [0,1] . Inverse distance is clipped at 10 : without clipping, a buoy at 1 m would yield \mathrm{inv\_dist}=1000 , dominating the network input; clipping at 10 corresponds to distances of 100 m or closer. This preserves the useful nonlinear variation at realistic ranges while preventing extreme values from destabilizing training. Bearing ranges from -180^{\circ} to +180^{\circ} , so dividing by 180 is standard unit normalization for this angular quantity. Pitch and roll are divided by 10^{\circ} , which reflects the small vessel motion range observed in the dataset, mapping the actual data range to approximately [-1,1] . Heading similarly ranges over \pm 180^{\circ} and is normalized by 180 for the same reason. QueryMLP is a four-layer network with architecture 6\rightarrow 128\rightarrow 128\rightarrow 128\rightarrow 2 . Each hidden layer is followed by BatchNorm1d, ReLU, and Dropout (0.2) , with a Sigmoid output layer. The ground-truth target is the normalized horizontal center and bottom edge (c_{x},c_{y}+h/2) , corresponding to the buoy’s waterline contact point and extracted directly from the existing challenge annotations. It is trained on paired (query, IMU, label) examples from the training set using AdamW (\mathrm{lr}=10^{-3} , weight decay =10^{-4}) with batch size 256 , cosine annealing, and SmoothL1 loss, for up to 1000 epochs with early stopping (patience =60 ), converging at epoch 585 . QueryMLP is evaluated using the Euclidean pixel distance between its predicted point and the ground-truth target (c_{x},\;c_{y}+h/2) . The model achieves on the validation set a median error of 18.4 px (mean =27.1 px, 90 th percentile =59.4 px). Figure 1 illustrates the QueryMLP architecture in comparison with the baseline query construction. Figure 1: Query construction pipeline. (a) Baseline: chart distance and bearing are fed directly into the embedding MLP. (b) Ours: a frozen QueryMLP takes six features (distance, bearing, and three IMU orientation angles) and predicts the buoy waterline contact point [c_{x},c_{y}\!+\!h/2] in image coordinates. These pixel coordinates are concatenated with the normalized distance and bearing to form a 4 -dimensional query vector, widening the embedding MLP input from 2 to 4 . All other architecture components are unchanged from [ 4 ] . 2.2 Integration into the DETR Pipeline Once trained, QueryMLP is held fixed. During DETR training and inference the dataset loader runs it on every chart query to produce predicted pixel coordinates [c_{x},\;c_{y}+h/2] . They are then concatenated with the normalized distance and bearing to form a four-dimensional query vector q=[\mathrm{dist}_{\text{norm}},\mathrm{bearing}_{\text{norm}},c_{x},\;c_{y}+h/2]. This augmented query is then passed through the baseline decoder MLP, whose input dimension is widened from 2 to 4 , to produce the 256 -dimensional decoder embedding used by DETR. QueryMLP is kept fixed during DETR training and inference. All other aspects of the architecture remain identical to the baseline: the ResNet-50 backbone, the six-layer transformer encoder and decoder, and the BCE + L1 + GIoU loss. 3 Results The ATON dataset [ 4 ] 2 2 2 https://drive.google.com/drive/folders/1OXSok1Aux0rfygNQHFIe8goB695DeN3f contains 5{,}189 samples ( 4{,}285 train / 904 val) at 960\times 540 px. The method development and model selection were performed on the public train/validation split of the dataset. The final leaderboard results were obtained from the evaluation by the challenge organizers on a private held-out test set. Not all images contain visible buoys; empty label files indicate buoy-free frames. Performance is evaluated by \mathrm{Overall}=(F_{1}+\mathrm{mIoU})/2

Read the complete text