Distillation of Diffusion Features for Semantic Correspondence
Frank Fundel, Johannes Schusterbauer, Vincent Tao Hu, Björn Ommer · 2024
Read Distillation of Diffusion Features for Semantic Correspondence by Frank Fundel, Johannes Schusterbauer, Vincent Tao Hu, Björn Ommer (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03512.txt.
Introduction
Frank Fundel and three co-authors present a 2024 paper on matching parts across images, a task called semantic correspondence. Knowing that a point on one cat matches a point on another supports editing, tracking, and 3D work. Features from large image generators do this well, but top methods stack several huge models and run slowly. The team instead distills two big vision models into one smaller student.
Distillation copies the teachers' skills into the compact model, which kept high accuracy at much lower cost. Adding 3D data during training lifted performance further, with no hand-labeled matches needed. The student beat current state-of-the-art methods on their tests while using far less computing. Tests included video correspondence, a demanding real-world case. Code and model weights are public. The authors pitch the approach as practical for everyday vision tools.
Catalog record
| CREATOR | Frank Fundel, Johannes Schusterbauer, Vincent Tao Hu, Björn Ommer |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03512
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Distillation of Diffusion Features for Semantic Correspondence Frank Fundel, Johannes Schusterbauer, Vincent Tao Hu, Björn Ommer Abstract Semantic correspondence, the task of determining relationships between different parts of images, underpins various applications including 3D reconstruction, image-to-image translation, object tracking, and visual place recognition. Recent studies have begun to explore representations learned in large generative image models for semantic correspondence, demonstrating promising results. Building on this progress, current state-of-the-art methods rely on combining multiple large models, resulting in high computational demands and reduced efficiency. In this work, we address this challenge by proposing a more computationally efficient approach. We propose a novel knowledge distillation technique to overcome the problem of reduced efficiency. We show how to use two large vision foundation models and distill the capabilities of these complementary models into one smaller model that maintains high accuracy at reduced computational cost. Furthermore, we demonstrate that by incorporating 3D data, we are able to further improve performance, without the need for human-annotated correspondences. Overall, our empirical results demonstrate that our distilled model with 3D data augmentation achieves performance superior to current state-of-the-art methods while significantly reducing computational load and enhancing practicality for real-world applications, such as semantic video correspondence. Our code and weights are publicly available on our project page. Figure 1 : Our method achieves better performance and throughput with less parameters on the SPair-71k dataset. The circle size represents number of parameters. For more details, see Tab. 1 . 1 Introduction Semantic correspondence involves finding relationships between different regions across two images, enabling a wide range of applications such as 3D reconstruction [ 47 ] , image-to-image translation [ 91 ] , object tracking [ 22 ] , video segmentation [ 95 ] , pose estimation [ 85 , 37 ] , explainable AI [ 87 ] and visual place recognition [ 42 ] . Traditional approaches often rely on handcrafted features like SIFT [ 58 ] and SURF [ 5 ] , which, despite their robustness, have limitations in capturing semantic meaning. With the advent of deep learning, Convolutional Neural Networks [ 101 , 31 , 73 , 67 ] and Vision Transformers [ 43 , 45 , 11 , 40 , 12 ] have revolutionized the field, providing powerful methods to extract semantically rich features from images in various ways. Since finding semantic correspondences is challenging due to the need for extensive world knowledge, the transition to large-scale supervised deep learning was hindered by the limited availability of ground-truth annotations. To address this issue, recent efforts have focused on weakly-supervised [ 94 ] and self-supervised methods to extract per-pixel descriptors, most notably DINO [ 1 , 29 ] . Diffusion Models (DMs) have shown great capabilities in generating high-quality images, due to their large-scale training paradigm which has also led them to learn rich world representations, useful for downstream tasks such as semantic correspondence [ 32 , 90 ] . Since a previous work “A Tale of Two Features” [ 102 ] demonstrated that those two models are complementary in their features, current state-of-the-art approaches [ 103 , 63 ] all rely on combining two large vision foundation models. This comes with several limitations. First, diffusion models often need to be run multiple times to incorporate different timesteps, leading to increased computational complexity and longer inference times. Second, these combined models are typically very large networks with a substantial number of parameters, which require significant computational resources and memory. Third, managing these combined models introduces numerous additional hyperparameters, such as timestep(s), layer(s) and prompt, and mixing weights, complicating further training. This becomes particularly challenging when calculating correspondences in videos or in other real-time applications such as semantic video correspondence. Additionally, the increased number of parameters reduces computational efficiency, posing challenges for server-less applications. Distillation [ 34 ] is a well-known method broadly applied in various tasks like classification [ 55 ] , segmentation [ 60 ] , self-supervised learning [ 25 ] and generative models’ sampling acceleration [ 78 ] . This naturally raises a question: can we also apply distillation on top of the ”Tale of Two Features” [ 102 ] , and ideally squeeze the parameter burden further to boost inference speed in the task of semantic correspondence? We propose to leverage knowledge distillation for semantic correspondence to reduce the computational load of existing off-the-shelf methods, without retraining a full model. In summary, we make the following contributions: • We propose a parameter-efficient approach to distill the semantic correspondence capabilities of two complementary large vision foundation models into a more compact and efficient model. • We additionally propose a novel fine-tuning protocol to further boost the model’s performance by incorporating 3D data augmentation. This approach not only achieves new state-of-the-art results but also allows the model to improve without the need for labeled data. • We validate the effectiveness of our method on three canonical benchmark datasets, demonstrating state-of-the-art performance on semantic correspondence with significantly reduced inference time. 2 Related Work Visual Correspondence. Visual correspondence refers to the identification of meaningful associations between different points or regions in images. Traditional methods for geometric correspondence, such as SIFT [ 58 , 59 ] and SURF [ 5 ] , involve extracting hand-crafted visual descriptors and matching them across images. Later, it was discovered, that semantic correspondence also emerges in deep Convolutional Neural Network (CNN) features [ 61 , 101 , 105 , 23 ] trained on other tasks such as classification. Subsequently, specifically designed architectures and algorithms were proposed to tackle the task of semantic correspondence using deep features of CNNs [ 14 , 44 , 56 , 48 , 75 , 74 , 51 , 49 , 43 , 45 ] and recently also transformers [ 11 , 40 , 12 ] . However, finding semantic correspondences is a much more challenging task that requires vast knowledge about the world. Thus, the transition to large-scale supervised deep learning has been impeded by the limited availability of ground-truth correspondence annotations. As a solution to this issue, recent efforts have emerged in the form of weakly-supervised [ 94 ] and self-supervised [ 38 ] methods to generate per-pixel (dense) descriptors, most notably DINO [ 1 , 29 ] . DINO is a large Vision Transformer (ViT) [ 18 ] trained using a self-distillation approach, where the model learns to predict its own output under different augmentations, effectively leveraging large amounts of unlabeled data to generate high-quality dense descriptors. Generative models need to possess extensive knowledge about the world to synthesize new data accurately. Therefore, the internal representations of large generative models, such as Generative Generative Adversarial Networks (GANs) [ 24 ] , can be effectively utilized for identifying visual correspondences within specific image categories [ 68 ] . Similarly, internal representations of Diffusion Models (DMs) recently demonstrated their potential in various downstream tasks. Diffusion Model Representations. Diffusion Models have significantly advanced the field of image generation, achieving state-of-the-art performance [ 17 , 76 , 82 ] and their applications extend beyond image creation to various vision tasks such as image segmentation [ 2 , 4 , 10 , 39 , 88 , 96 , 89 ] , object detection [ 9 ] , image classification [ 50