Template-based Object Detection Using a Foundation Model

Valentin Braeutigam, Matthias Stock, Bernhard Egger · 2026

Read Template-based Object Detection Using a Foundation Model by Valentin Braeutigam, Matthias Stock, Bernhard Egger (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19773.txt.

Introduction

Braeutigam, Stock, and Egger detect objects from templates without any model training. Most detectors learn from data and need a training set. Their target uses have low variation but strict cost limits. One use is automatic testing of graphical interfaces. The method takes segments from a segmentation foundation model. A simple feature-based classifier then labels those segments. Tests targeted icons in navigation maps for car software testing. The results came close to trained detectors such as YOLO.

Catalog record

CREATORValentin Braeutigam, Matthias Stock, Bernhard Egger
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45814
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19773

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Template-based Object Detection Using a Foundation Model Valentin Braeutigam, Matthias Stock, Bernhard Egger Abstract Most currently used object detection methods are learning-based, and can detect objects under varying appearances. Those models require training and a training dataset. We focus on use cases with less data variation, but the requirement of being free of generation of training data and training. Such a setup is for example desired in automatic testing of graphical interfaces during software development, especially for continuous integration testing. In our approach, we use segments from segmentation foundation models and combine them with a simple feature-based classification method. This saves time and cost when changing the object to be searched or its design, as nothing has to be retrained and no dataset has to be created. We evaluate our method on the task of detecting and classifying icons in navigation maps, which is used to simplify and automate the testing of user interfaces in automotive industry. Our methods achieve results almost on par with learning-based object detection methods like YOLO, without the need for training. Figure 1 : Overview of Our Workflow. We use a SAM model to segment all objects in the input image and use each segment as an object proposal. The object proposals can either be generated from the input images or the images with removed text which are generated as shown in Fig. 4 . To reduce the number of icon proposals we first compute the correlation of the color histograms with each histogram of the templates and reject candidates without matching templates. For the remaining templates we compare the metric values (LPIPS or CLIP) between the candidate and the template crops. If there are values below a certain threshold the template with the best metric value is predicted as match. 1 Introduction In the field of object detection, template matching was considered state-of-the-art for a long time. This has changed for most applications with the use of strong features and machine learning. One advantage of learning-based methods is that they can be very robust to appearance changes, but this is not required for every application. We focus on one problem field where these methods are applied, namely in automated UI testing in the automotive context, especially for detecting icons in an automotive navigation map. In this use case, there is only a restricted variety of appearance changes, as the icons are changing in size, but are not rendered in perspective, and have no illumination. Nevertheless, there are frequent changes in the design or icons to search, while most of the tests are still implemented by manually created test cases using pixel comparisons of the renderings to template images created before. This is very time-consuming during creation of test cases and it requires adaptation of the test cases every time the design changes, which again takes much effort. One approach to be more data invariant is the use of template matching, which seems suitable to detect UI elements as there is not much variation compared to objects rendered in perspective. As they are orthographic this leads to the simplification that they are not rendered from different perspectives, which reduces the number of required templates. However, in the case of icons the problem of different scales remains, which causes problems when applying standard template matching. Additionally, there is the challenge of similar or partly equal icons, e.g. when having the icons for a parking lot and a parking garage. A third problem can be other objects that partially cover the searched objects. In our example, this is the case with text, e.g. city names or street names, that is rendered in front of the icons. One solution to deal with all of these problems, which is already in use for automated testing [ 1 ] are learning-based methods. There are various learning-based object detection methods, that have shown a good accuracy [ 29 , 7 , 15 , 36 ] , and some methods also while predicting results in real time, e.g. the YOLO models [ 29 ] . With these methods the presence can be checked automatically, based on classes specified by the training data, and the models can be trained to perform on different scales of searched objects. However, in this scenario changing designs remain a problem, as the methods have to be newly trained to adapt to the new unseen objects. This is especially a problem, when using continuous integration, as the test pipeline would have to be updated, re-trained and checked every time the design changes. Another disadvantage is the requirement of a training dataset, which has to be created and newly generated or adapted with each design change. Additionally, the dataset has to cover the possible positions of the icons on the different possible backgrounds and icon sizes. Especially during a design phase where the appearance of tested items often changes, this can be time consuming and is therefore not suitable for integrated testing. We present a new method, which is a modern intersection between template matching and object detection methods that combines advantages of both. Although it is not restricted to the depicted use case, we evaluate it for this one. The method uses a foundation model for segmentation, concretely Segment Anything 2.1 (SAM2.1) [ 27 ] , respectively SAM3 [ 3 ] , in combination with a template-based classification. For the second part features generated by publicly available pre-trained networks from commonly used methods like CLIP [ 26 ] or LPIPS [ 35 ] are used. With this approach, we circumvent the need to re-train and collect datasets for the training while still being scale invariant and robust to partial occlusions, which combines the advantages of both approaches. In addition, we extend our approach by segmenting text in the input images that potentially covers the searched elements. Afterwards, we apply inpainting to the image masks to remove the text in the image and to improve classification results. Our code is publicly available at https://github.com/valentinBraeutigam/Template-Based-Object-Detection . Our contributions are as follows: • we present a method to detect and classify objects in an image with no need for learning/adapting to new objects • instead of a dataset we need only one template per object, each of them can be easily replaced or modified • our method is scale-invariant and robust to partial occlusion by text removal • we provide source code of the proposed template-matching framework 2 Related Work Template Matching. There are several template matching techniques capable of detecting a given template in an image [ 2 , 38 , 13 , 12 , 32 ] . These algorithms are usually based on a sliding-window approach [ 2 ] which means that they iterate over the whole image and compare the template at each window position with the input image by a metric. Commonly used metrics are squared difference, cross-correlation, and correlation coefficient. These techniques have the problem that they are often not size invariant, which is usually solved by iterating over the image with different scaling factors of the template image, e.g. in [ 38 ] . But this can be very tedious as, depending on the size changes that shall be covered, the process has to be

Read the complete text