Keypoint Semantic Integration for Improved Feature Matching in Outdoor Agricultural Environments

Rajitha de Silva, Jonathan Cox, Marija Popović, Cesar Cadena, Cyrill Stachniss, Riccardo Polvara · 2025

Read Keypoint Semantic Integration for Improved Feature Matching in Outdoor Agricultural Environments by Rajitha de Silva, Jonathan Cox, Marija Popović, Cesar Cadena, Cyrill Stachniss, Riccardo Polvara (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08843.txt.

Introduction

Rajitha de Silva and five co-authors improve feature matching for farm robots in two thousand twenty-five. Their test ground is vineyards, where rows of vines look much alike. Like looks make image points hard to tell apart. That hurts pose checks and place finding for the robot.

The team adds semantic facts to each key point in the image. Points in regions with meaning get richer codes. That makes twins in looks easier to split apart. Tests span two tasks, pose checks and place finding. Match accuracy rose across all point and code types tried. Gains held over months of hard field work.

Catalog record

CREATORRajitha de Silva, Jonathan Cox, Marija Popović, Cesar Cadena, Cyrill Stachniss, Riccardo Polvara
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08843

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Keypoint Semantic Integration for Improved Feature Matching in Outdoor Agricultural Environments Rajitha de Silva, Jonathan Cox, Marija Popović, Cesar Cadena, Cyrill Stachniss, Riccardo Polvara Abstract Robust robot navigation in outdoor environments requires accurate perception systems capable of handling visual challenges such as repetitive structures and changing appearances. Visual feature matching is crucial to vision-based pipelines but remains particularly challenging in natural outdoor settings due to perceptual aliasing. We address this issue in vineyards, where repetitive vine trunks and other natural elements generate ambiguous descriptors that hinder reliable feature matching. We hypothesise that semantic information tied to keypoint positions can alleviate perceptual aliasing by enhancing keypoint descriptor distinctiveness. To this end, we introduce a keypoint semantic integration technique that improves the descriptors in semantically meaningful regions within the image, enabling more accurate differentiation even among visually similar local features. We validate this approach in two vineyard perception tasks: (i) relative pose estimation and (ii) visual localisation. Across all tested keypoint types and descriptors, our method improves matching accuracy by 12.6%, demonstrating its effectiveness over multiple months in challenging vineyard conditions. I Introduction Modern agricultural systems require robust robotic solutions to automate labour-intensive tasks and enhance operational efficiency. Among these, vision-based perception gaining traction due to its affordability. For autonomous mobile robots operating in outdoor agricultural environments, perception systems must accurately detect and track visual features for tasks such as relative pose estimation and visual localisation. However, vineyards present repetitive structures—vine trunks, poles, water pipes—that complicate feature correspondence between consecutive frames. Both traditional and learned keypoint descriptors focus on local regions, lacking broader semantic context, which can lead to false-positive matches for spatially distant but visually similar keypoints. This challenge, known as perceptual aliasing, remains a major hurdle towards dependable vision-based navigation in outdoor settings. Fig. 1: KSI enhances the keypoint descriptors with corresponding semantic instances to minimise aliasing during keypoint matching in vineyards. Keypoint matches on semantic masks are visualised here across March and June. In this paper, we address this challenge by exploiting the semantic and morphological properties of repetitive regions to reduce descriptor ambiguity and improve matching accuracy. Our approach enriches keypoint descriptors with semantic context to increase their distinctiveness, focusing on vineyard environments where repetitive structures are widely recognised yet rarely supplemented with semantic cues for descriptor differentiation [ 1 ] . Vine trunks, for instance, appear visually similar, producing nearly identical descriptors; however, morphological variations (Fig. 1 ) can be encoded with semantic embeddings to refine keypoint representations. This approach supports accurate keypoints matching across multiple months, even under significant seasonal changes. Typical feature matching pipelines rely on heuristic filtering [ 1 ] or incorporate context at the matching stage [ 2 ] , but they still struggle with uniform descriptors and repetitive spatial layouts [ 3 , 4 ] . We propose keypoint semantic integration (KSI), a framework for enhancing the keypoint descriptors with semantic mask instances on which they are located. An autoencoder [ 5 ] compresses the semantic mask into a feature vector, generating a semantic embedding that enhances descriptors’ distinctiveness. Consequently, keypoints on semantically meaningful regions gain enriched descriptors, while those in the background retain their original descriptors. Our contributions are are: (i) a semantic integration method, KSI 1 1 1 To support reproducibility and future research, our implementation and dataset will be available at: [Link will be available at acceptance] , that mitigates perceptual aliasing in visually repetitive agricultural environments; (ii) an extensive ablation and evaluation study demonstrating the method’s robustness under seasonal changes in vineyards; and (iii) the Semantic Bacchus Long Term (SemanticBLT) dataset with panoptic segmentation annotations in vineyards. II Related Work Visual feature extraction and matching are essential first steps in a traditional visual simultaneous localisation and mapping (SLAM) pipeline. Classic feature extractors, such as Scale Invariant Feature Transform (SIFT) [ 6 ] and Oriented FAST and Rotated BRIEF (ORB) [ 7 ] , remain popular for their rotation and scale invariance and computational efficiency, while deep learning-based feature extractors like SuperPoint [ 8 ] , D2-Net [ 9 ] , and Reliable and Repeatable Detector Descriptor (R2D2) [ 10 ] ) prioritise robustness, repeatability, and reliability. However, neither traditional nor deep learning–based descriptors inherently capture the semantic context of a keypoint, instead relying on local neighbourhood information of the keypoint. Variants such as Semantic SuperPoint [ 11 ] combine semantic information with descriptor learning, but must be retrained for each new semantic class, hindering generalisation capabilities. Other approaches integrate semantic cues at the matching stage or via heuristic filtering [ 1 ] , adapt descriptors online based on the current environment [ 12 ] , or exploit sequence information to address repetitive scenes in agricultural environments [ 13 ] . Keypoint matching in agricultural environments poses significant challenges due to the scarcity of unique landmarks, repetitive crop appearances, lightning changes, and uneven terrain [ 3 , 4 ] . Seasonal changes and foliage growth [ 14 ] exacerbate these issues, leading to perceptual aliasing and matching errors. Classic matching algorithms, such as nearest neighbour search, struggle to address these issues effectively. Ways to address this include exploiting geometric structures of keypoints [ 3 ] or exploiting background knowledge of the agricultural domain [ 13 , 15 ] . Learning-based matches e.g., SuperGlue [ 2 ] , LightGlue [ 16 ] harness graph neural networks (GNNs) or transformers to incorporate local context for more robust matching. Tanco et al. [ 17 ] highlight the limitations of traditional detection algorithms through an agriculture-specific feature descriptor and matching method. However, vineyard-specific solutions remain limited [ 18 ] . Papadimitriou et al. [ 1 ] address this issue by restricting keypoint extraction to semantically significant regions, such as vine trunks, which retain a relatively consistent appearance over time and remain visible despite foliage growth. Their approach, however, simply masks these regions rather than integrating explicit semantic context into the descriptors. The use of semantic cues in keypoint detection and matching has been explored in a variety of contexts, including urban environments [ 19 ] , remote sensing [ 20 ] , robotic manipulation [ 21 ] . and agriculture [ 17 ] , where semantic information is learned from different object classes. Yet most existing semantic keypoint extractors rely on a multi-task backbone [ 22 ] with separate heads for semantic labels and keypoint descriptors, limiting them to the classes encountered during training. Tanco et al. [ 17 ] observe that, while their approach is trained with a semantic decoding head, its performance in inferring semantic labels at test time remains suboptimal — suggesting opportunities to better understand the impact of this multi-task setup on keypoint descriptors compared to standard descriptor-based methods. These limitations accentuate the need for a modular approach that extends easily to new semantic classes, improving generalisation in specialised contexts such as agriculture. Fig. 2: Overview of proposed KSI framework: Input semantic instances \mathcal{S_{I}} and the set of keypoints \mathcal{K_{I}} are extracted from each image in the input image pair \mathcal{I} . The descriptors of keypoints on a semantic instance are combined with the corresponding semantic embedding to generate semantically enriched keypoint descriptors while the background descriptors are left unaltered. The semantically enriched descriptors and the background descriptors are

Read the complete text