Semantic Segmentation of Transparent and Opaque Drinking Glasses with the Help of Zero-shot Learning
Annalena Blänsdorf, Tristan Wirth, Arne Rak, Thomas Pöllabauer, Volker Knauthe, Arjan Kuijper · 2025
Read Semantic Segmentation of Transparent and Opaque Drinking Glasses with the Help of Zero-shot Learning by Annalena Blänsdorf, Tristan Wirth, Arne Rak, Thomas Pöllabauer, Volker Knauthe, Arjan Kuijper (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15004.txt.
Introduction
Annalena Blänsdorf and co-authors teach a model to outline drinking glasses, including glasses it has never seen, in images. Transparent things nearly vanish against their backgrounds, which defeats ordinary segmentation. Their 2025 model, TransCaGNet, swaps a transparent-object backbone into a zero-shot segmentation design, so new glass types can be named by text rather than by examples.
Training ran on a synthetic data set spanning many lighting and scene conditions, and testing added a real-world set the team captured. Against a leading rival, their model scored better on overlap and accuracy overall, while the rival kept an edge on unseen classes. Letting the model assign several plausible labels to one glass, paired with a general segmenter, lifted scores sharply, by up to about fourteen points. Trained on hard synthetic scenes, the models did even better on real photos.
Catalog record
| CREATOR | Annalena Blänsdorf, Tristan Wirth, Arne Rak, Thomas Pöllabauer, Volker Knauthe, Arjan Kuijper |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15004
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Semantic Segmentation of Transparent and Opaque Drinking Glasses with the Help of Zero-shot Learning Annalena Blänsdorf, Tristan Wirth, Arne Rak, Thomas Pöllabauer, Volker Knauthe, Arjan Kuijper Abstract Segmenting transparent structures in images is challenging since they are difficult to distinguish from the background. Common examples are drinking glasses, which are a ubiquitous part of our lives and appear in many different shapes and sizes. In this work we propose TransCaGNet , a modified version of the zero-shot model CaGNet . We exchange the segmentation backbone with the architecture of Trans4Trans to be capable of segmenting transparent objects. Since some glasses are rarely captured, we use zero-shot learning to be able to create semantic segmentations of glass categories not given during training. We propose a novel synthetic dataset covering a diverse set of different environmental conditions. Additionally we capture a real-world evaluation dataset since most applications take place in the real world. Comparing our model with ZegClip we are able to show that TransCaGNet produces better mean IoU and accuracy values while ZegClip outperforms it mostly for unseen classes. To improve the segmentation results, we combine the semantic segmentation of the models with the segmentation results of SAM 2 . Our evaluation emphasizes that distinguishing between different classes is challenging for the models due to similarity, points of view, or coverings. Taking this behavior into account, we assign glasses multiple possible categories. The modification leads to an improvement up to 13.68 % 13.68\% for the mean IoU and up to 17.88 % 17.88\% for the mean accuracy values on the synthetic dataset. Using our difficult synthetic dataset for training, the models produce even better results on the real-world dataset. The mean IoU is improved up to 5.55 % 5.55\% and the mean accuracy up to 5.72 % 5.72\% on the real-world dataset. 1 Introduction Drinking glasses are common items in our households and a ubiquitous part of our lives. Since we use them every day it is important for autonomous agents to distinguish between them. Furthermore, visual aids for visually impaired people might be improved by classifying and segmenting glasses correctly. But even for the same beverage, the glasses differ in shape and size, which make them hard to differentiate. Furthermore the segmentation of transparent glasses is difficult since their appearance can change depending on backgrounds [ 32 , 45 ] , viewing angles [ 50 , 7 ] , lighting conditions [ 50 , 7 ] , and distortions [ 24 ] . This segmentation task is even more challenging if some glass categories are not given during training, for instance since a type of glass is rarely used or unusually shaped. Such classes are referred to as unseen classes. Using zero-shot learning, the models are able to provide semantic segmentations of even those during evaluation [ 20 , 19 ] . In this work, two machine learning models generate semantic segmentations of transparent and opaque glasses from IKEA 1 1 1 https://www.ikea.com/ . During semantic segmentation, a class label is assigned to all image pixels [ 22 ] . We modify the zero-shot learning model CaGNet [ 17 , 18 ] by exchanging the segmentation backbone with the architecture of Trans4Trans [ 51 , 52 ] , a segmentation network specified for transparent objects. Our resulting model TransCaGNet should be able to provide a semantic segmentation of transparent glasses. We compare TransCaGNet with the existing model ZegClip [ 56 ] based on CLIP [ 36 ] using different test scenarios. For the evaluation, we create a challenging synthetic novel dataset with high variance using BlenderProc2 [ 9 ] and 3D models of IKEA glasses [ 27 ] to train and evaluate the models. Our evaluation shows that TransCaGNet produces better results for the known classes, but ZegClip outperforms it for the unseen classes. Since most applications are used under real-world conditions, we also captured a real-world evaluation dataset. We are able to show that the models trained with synthetic data produce even better results on the real dataset than on the synthetic one during evaluation. The mean IoU is improved up to 5.55\% and the mean accuracy up to 5.72\% on the real-world dataset. To improve the segmentation results and reduce the fragmentation inside objects, we combine the original semantic segmentation masks created by TransCaGNet and ZegClip with the segmentation results of SAM 2 [ 37 ] . The mean IoU values are enhanced by up to 6.3\% and the mean accuracy values up to 10.53\% . We highlight during our evaluation that the similarity between some classes influences the result of the models significantly. Especially if two glasses are close in appearance, if the view is partially obscured, or through the use of some glasses for multiple purposes, it is difficult for the models to distinguish between the classes. To overcome these challenges, we reorganize the classes by assigning 3D models to multiple possible classes. It leads to an further improvement of up to 8.05\% for the mean IoU-values and up to 7.68\% for the mean accuracy. In summary, this paper offers the following contributions: • We provide a novel dataset consisting of 7,000 synthetic and 225 real images of drinking glasses with semantic segmentation labels, • we propose TransCaGNet , a zero-shot learning model based on CaGNet which segmentation backbone we replace with the architecture of Trans4Trans , and • we propose a policy incorporating SAM 2 in addition to a merging-strategy for similar classes, that increases the IoU by up to 13.68\% and the mean accuracy by up to 17.88\% . 2 Related Work In this chapter, we provide an overview of existing datasets with transparent objects. Furthermore, we discuss the segmentation of transparent objects and some possible applications and describe existing zero-shot learning models for non-transparent objects. Datasets with transparent objects There are multiple datasets with transparent objects for different tasks. The dataset from TransCut [ 48 ] captures real-world light-field images of different transparent objects for segmentation. Synthetic and real images with ground-truth object masks, attenuation masks, and refractive flow fields are provided by the dataset used for TOM-Net [ 5 ] for learning the refractive flow. ClearGrasp [ 38 ] is used for 3D geometry estimation containing synthetic and real-world RGBD images, semantic segmentation of all transparent objects, poses, and scene’s surface normals. This dataset is re-used by Wirth et al. [ 43 ] and extended with additional real-world images. ProLit [ 55 ] provides a light-field image dataset for transparent object recognition, segmentation, and 6D pose estimation. TOD [ 29 ] consists of stereo images with corresponding depth and 3D keypoints for 3D pose estimation. The GDD dataset [ 32 ] is focused on detecting glass surfaces containing real-world images with corresponding segmentation masks. Vector-LabPic [ 11 ] provides real-world images of vessels for semantic and instance segmentation tasks. For semantic segmentation, Xie et al. [ 45 ] create the real-world dataset Trans10K , whose images are taken from the internet and labeled by hand. Their annotations are extended through Trans10Kv2 [ 46 ] by splitting the classes of Trans10K into new, more subdivided classes. Furthermore, the stereo dataset StereOBJ-1M [ 30 ] is used for 6D pose estimation of transparent objects.