Beyond the Encoder: Joint Encoder-Decoder Contrastive Pre-Training Improves Dense Prediction
Sébastien Quetin, Tapotosh Ghosh, Farhad Maleki · 2025
Read Beyond the Encoder: Joint Encoder-Decoder Contrastive Pre-Training Improves Dense Prediction by Sébastien Quetin, Tapotosh Ghosh, Farhad Maleki (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17526.txt.
Introduction
Sébastien Quetin, Tapotosh Ghosh, and Farhad Maleki present a 2025 paper on teaching vision models both to encode and decode images before any labeled training. Most self-supervised methods pre-train only the encoder, leaving the decoder to learn later from labels. For dense tasks like outlining objects, that split wastes potential. The team proposes DeCon, a framework that pre-trains both parts together with a combined contrastive loss.
Pre-trained on standard image sets, DeCon set top marks on most tasks tested. It improved object detection and instance segmentation on a major benchmark over its base framework. Semantic segmentation also gained on two city and scene data sets. The gains held across backbones and in low-data and out-of-domain settings. The authors argue joint pre-training yields richer features for dense prediction. Their code is public for others to build on.
Catalog record
| CREATOR | Sébastien Quetin, Tapotosh Ghosh, Farhad Maleki |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-17526
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond the Encoder: Joint Encoder-Decoder Contrastive Pre-Training Improves Dense Prediction Sébastien Quetin, Tapotosh Ghosh, Farhad Maleki Abstract Contrastive learning methods in self-supervised settings have primarily focused on pre-training encoders, while decoders are typically introduced and trained separately for downstream dense prediction tasks. However, this conventional approach overlooks the potential benefits of jointly pre-training both encoder and decoder. In this paper, we propose DeCon , an efficient encoder-decoder self-supervised learning (SSL) framework that supports joint contrastive pre-training. We first extend existing SSL architectures to accommodate diverse decoders and their corresponding contrastive losses. Then, we introduce a weighted encoder-decoder contrastive loss with non-competing objectives to enable the joint pre-training of encoder-decoder architectures. By adapting a contrastive SSL framework for dense prediction, DeCon establishes consistent state-of-the-art performance on most of the evaluated tasks when pre-trained on Imagenet-1K, COCO and COCO+. Notably, when pre-training a ResNet-50 encoder on COCO dataset, DeCon improves COCO object detection and instance segmentation compared to the baseline framework by +0.37 AP and +0.32 AP, respectively, and boosts semantic segmentation by +1.42 mIoU on Pascal VOC and by +0.50 mIoU on Cityscapes. These improvements generalize across recent backbones, decoders, datasets, and dense tasks beyond segmentation and object detection, and persist in out-of-domain scenarios, including limited-data settings, demonstrating that joint pre-training significantly enhances representation quality for dense prediction. Code is available at https://github.com/sebquetin/DeCon.git . * * footnotetext: Equal contributions. Co-first authors are ordered in alphabetical order. Corresponding author: T. Ghosh ([email protected]) 1 Introduction The growing demand for efficient Deep Learning (DL) methods stems from their ability to address complex tasks. While traditional approaches like supervised learning can train high-performing models, they depend on large volumes of high-quality annotated data. Obtaining such annotations is often labor-intensive, expensive, and sometimes impractical, even when abundant data is available. A common workaround is to use pre-trained models, typically encoders trained from scratch on large-scale datasets, to boost performance on downstream tasks with limited labels. Typically, an encoder is pre-trained to capture a representation of the input data, which can then be fine-tuned for downstream tasks such as classification or segmentation. During the downstream fine-tuning phase, randomly initialized layers are typically added on top of the encoder, and the complete architecture is retrained using a supervised learning approach on an annotated dataset. ImageNet [ 20 ] is a widely utilized labeled dataset for such pre-training. However, models pre-trained on ImageNet classification task often transfer poorly to downstream dense prediction tasks, such as detection and segmentation [ 76 , 60 , 52 ] . Creating large-scale annotated datasets for every task is impractical. Self-Supervised Learning (SSL) is a promising alternative, enabling models to learn from the large-scale unannotated data to enhance performance on various downstream tasks. These pre-trained models can reach downstream performance close to a fully supervised approach while using significantly fewer labeled examples. Notable approaches such as SimCLR [ 10 ] , VicReg [ 3 ] , and MoCo [ 28 ] showed promising results by leveraging contrastive learning for downstream classification. However, most SSL frameworks primarily target classification-based downstream tasks. Some efforts, such as PixCon [ 49 ] and DenseCL [ 60 ] , focus on dense prediction tasks like segmentation or object detection. However, these approaches only pre-train encoders for downstream use and rely on local losses adapted from classification-oriented SSL methods without jointly training the decoder. In this work, we propose a Decoder-aware contrastive learning (DeCon) approach, an encoder-decoder framework that jointly pre-trains encoder and decoder in an SSL manner. DeCon enhances the representation power of the encoder, and prepares it more effectively for integration with decoders in downstream dense prediction tasks. The contributions of this paper are as follows: • We propose DeCon-SL , a single-level joint encoder-decoder adaptation of contrastive dense SSL frameworks. We show that jointly pre-training the encoder and decoder enhances the representation power of the encoder and improves downstream dense prediction performance, even when transferring only the pre-trained encoder. • We further extend this adaptation by introducing DeCon-ML , which builds on DeCon-SL with a multi-level decoder loss function and an encoder-level channel dropout to promote a comprehensive encoder feature utilization. • DeCon achieves new state-of-the-art (SOTA) results in COCO object detection and instance segmentation, and Pascal VOC, Cityscapes and ADE20K semantic segmentation when pre-trained on ImageNet-1K and COCO+. • We show that DeCon consistently improves performance across different backbones, a range of dense downstream tasks beyond object detection, semantic and instance segmentation as well as in various out-of-domain scenarios. • We show that DeCon can achieve gains without increasing parameter count and with comparable GPU cost relative to the original framework, e.g. SlotCon [ 63 ] . 2 Related Work Self-Supervised pre-training: In early SSL works, deep learning models were trained on pretext tasks where supervision was derived directly from the data itself. This enabled the models to learn meaningful representations without relying on manual annotations. These tasks included predicting missing parts of an image [ 50 ] , context prediction [ 21 ] , solving jigsaw puzzles [ 46 ] , colorizing grayscale images [ 73 ] , and predicting rotations [ 25 ] . These methods aimed to create surrogate objectives that encouraged models to develop feature representations transferable to downstream tasks. However, they often produced task-specific representations with limited generalizability. Recent SSL methods prioritize generalizability and move away from hand-crafted pretext tasks. These approaches can be broadly categorized into generative and contrastive methods. Generative approaches learn representations by modeling the underlying data distribution. This is typically achieved through reconstruction-based methods, where the model predicts corrupted parts of the input [ 27 , 1 , 22 , 71 ] , or through adversarial methods, where a generator model learns to synthesize realistic samples to challenge a discriminator, thereby encouraging the encoder to extract informative semantic features [ 65 , 35 ] . Contrastive methods, which are the focus of this research, learn data representations by maximizing the similarity between features extracted from two augmented views of the same image. To avoid model collapse—a phenomenon where the model maps all inputs to the same representation—strategies such as negative sampling [ 28 , 11 ] , stop-gradient mechanisms [ 13 ] , variance regularization [ 3 , 4 ] , momentum encoders [ 28 ] , and other architectural and optimization techniques have been employed [ 6 ] . Dense pre-training: Although most of the aforementioned frameworks can be used for downstream segmentation or detection, they are not designed specifically for pixel-level—i.e., dense prediction downstream tasks such as instance and semantic segmentation. Consequently, developing SSL approaches that generalize well to such downstream tasks remains an active area of research [ 63 , 60 , 59 , 4 , 47 ] . The main idea behind these approaches is to shift from a global image similarity to a local similarity. They aim to enforce local similarity based on the location of pixels in an image [ 70 , 59 , 47 ] , local pixel features [ 60 ] , or both [ 38 , 4 , 49 ] . Local similarity can also be enforced in a region-based manner [ 24 , 30 , 68 , 63 ] . These methods outperform those relying solely on