DC-ViT: Modulating Spatial and Channel Interactions for Multi-Channel Images

Umar Marikkar, Syed Sameed Husain, Muhammad Awais, Sara Atito · 2026

Read DC-ViT: Modulating Spatial and Channel Interactions for Multi-Channel Images by Umar Marikkar, Syed Sameed Husain, Muhammad Awais, Sara Atito (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14281.txt.

Introduction

Marikkar and co-authors present DC-ViT, a vision transformer built for multi-channel images with mixed channel sets. Stains, sensors, and settings change channel counts, which breaks fixed-channel encoders. Joint attention over all channels can wash out channel-specific detail. DC-ViT splits token updates into two paths: one for space within a channel, one for mixing across channels. A decoupled aggregation step then learns how much each channel counts for the task at hand.

Catalog record

CREATORUmar Marikkar, Syed Sameed Husain, Muhammad Awais, Sara Atito
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46357
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.14281

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DC-ViT: Modulating Spatial and Channel Interactions for Multi-Channel Images Umar Marikkar, Syed Sameed Husain, Muhammad Awais, Sara Atito Abstract Training and evaluation in multi-channel imaging (MCI) remains challenging due to heterogeneous channel configurations arising from varying staining protocols, sensor types, and acquisition settings. This heterogeneity limits the applicability of fixed-channel encoders commonly used in general computer vision. Recent Multi-Channel Vision Transformers (MC-ViTs) address this by enabling flexible channel inputs, typically by jointly encoding patch tokens from all channels within a unified attention space. However, unrestricted token interactions across channels can lead to feature dilution, reducing the ability to preserve channel-specific semantics that are critical in MCI data. To address this, we propose Decoupled Vision Transformer (DC-ViT), which explicitly regulates information sharing using Decoupled Self-Attention (DSA), which decomposes token updates into two complementary pathways: spatial updates that model intra-channel structure, and channel-wise updates that adaptively integrate cross-channel information. This decoupling mitigates informational collapse while allowing selective inter-channel interaction. To further exploit these enhanced channel-specific representations, we introduce Decoupled Aggregation (DAG), which allows the model to learn task-specific channel importances. Extensive experiments across three MCI benchmarks demonstrate consistent improvements over existing MC-ViT approaches. 1 Introduction Training and evaluating vision encoders on Multi-Channel Imaging (MCI) data is fundamentally challenging due to variability in channel configurations across datasets. Unlike natural images, where three RGB channels are standardised, multi-channel images may contain an arbitrary number of channels whose semantics depend on the data acquisition process. This variability arises in diverse settings such as multi-channel microscopy, where different staining protocols produce distinct biological markers, and remote sensing, where sensor modalities and fusion strategies determine spectral composition. For instance, HPA [ 23 ] contains four channels, JUMP-CP [ 6 ] contains five, WTC-11 [ 24 ] contains three, and So2Sat [ 27 ] includes 18 channels over multiple spectral bands. This heterogeneity makes it difficult to directly apply fixed-channel encoder designs that are standard in natural image modelling. To accommodate variable channel configurations, Multi-Channel Vision Transformers (MC-ViTs) have emerged as the dominant architecture to train on multi-channel datasets [ 3 , 2 , 22 , 17 ] . These models are designed to be channel-adaptive, enabling them to process images with a flexible number of input channels within a unified framework. Following the Vision Transformer (ViT) paradigm [ 13 ] , each channel is independently partitioned into respective patch tokens, after which all tokens are concatenated into a single sequence and processed jointly by shared self-attention layers. This long sequence formulation allows information to be exchanged freely across spatial locations and channels throughout the encoder architecture. Under this long-sequence formulation, prior work explores mechanisms to mitigate feature homogenisation across channels. This is achieved either by introducing channel-specific tokens [ 3 , 2 ] , or token diversification losses [ 22 ] , or a combination of both, and demonstrates that this increased inter-channel diversity generally improves downstream performance. However, despite these modifications, the underlying attention mechanism remains fully shared. Consequently, spatial and channel-wise interactions are entangled, limiting explicit control over how cross-channel information is incorporated. This unstructured token mixing dilutes channel-specific information within the encoder, thereby limiting the representation capability of MC-ViTs as MCI channels often encode distinct concepts. Yet, complete isolation is also undesirable as certain channels benefit from complementary contextual signals provided by others [ 14 , 19 ] . The objective is thus not to remove cross-channel interaction, but to regulate it. Motivated by this, we propose Decoupled Vision Transformer (DC-ViT), which explicitly regulates inter-channel interactions within the encoder, while maintaining semantic diversity across channels in the learnt representations. DC-ViT is built upon two components, Decoupled Self-Attention (DSA) and Decoupled Aggregation (DAG), which together regulate inter-channel interaction during encoding and representation aggregation. Figure 1 : DC-ViT consistently improves existing MC-ViTs across MCI benchmarks. y-axis is the average scrore over CHAMMI, JUMP-CP and So2Sat downstream tasks. Figure 2 : Schematic of DC-ViT. Channels build strong low-level representations independently, and are then introduced to cross-channel interactions. DSA(sp) and DSA(sp+ch) denote DC-ViT blocks ( Eq. 4 ) with l\notin\mathcal{M} and l\in\mathcal{M} in Eq. 5 , respectively. To regulate inter-channel interactions, we introduce Decoupled Self-Attention (DSA), which factorises token updates into spatial and a channel-wise components. The spatial branch aggregates information across spatial locations within the same channel to reinforce channel-specific features, while the channel-wise branch adaptively integrates information across channels. This separation enables structured cross-channel interaction and reduces the complexity of fully joint attention. To further leverage channel-specific semantics in forming final representations, we propose Decoupled Aggregation (DAG), a hierarchical aggregation scheme that first produces channel-specific embeddings and then aggregates them to form the final representation. This design explicitly models channel contributions to downstream tasks, aligning the representation structure with the heterogeneous nature of MCI datasets and their task formulations. Together, DC-ViT provides structured regulation of cross-channel information exchange, mitigating feature dilution while retaining beneficial context sharing. As shown in Fig. 2 , across standard MCI benchmarks, DC-ViT consistently outperforms prior MC-ViT baselines. 2 Background and Related Work Our work is motivated by the observation that channels in MCI datasets carry semantically distinct information which are not effectively encoded in existing MC-ViT architectures. Therefore, we first review evidence of channel heterogeneity across MCI datasets, and then review existing MC-ViT architectures. Channel Heterogeneity in MCI datasets. The semantic structure of channels in MCI datasets differs fundamentally from that of RGB images, where channels carry largely redundant information. For example, in immunofluorescent (IF) microscopy, individual channels capture a specific aspect of cellular organisation or phenotype. Some channels mainly provide stable structural context, while others encode signals that vary with the biological condition of interest [ 19 ] . In the HPA dataset, the protein of interest is imaged together with reference markers for the Nucleus, Microtubules, and Endoplasmic Reticulum, which support the interpretation of protein localisation [ 23 ] . In JUMP-CP, DNA and ER channels largely convey positional information, whereas RNA, AGP, and Mitochondrial channels reflect morphology changes induced by perturbations [ 6 , 10 ] . In WTC-11, the nucleus channel contains the primary biological signal, with membrane and protein channels providing additional context [ 24 ] . CM4AI-Bridge2AI adopts a similar staining scheme, combining target proteins with structural markers such as DAPI, anti-tubulin, and ER [ 9 ] . Similarly, semantic variation between channels is observed in satellite-based multi-channel imaging. In the So2Sat LCZ42 dataset, channels contain visible, near-infrared, and short-wave infrared bands, each encoding different physical properties of the Earth’s surface [ 27 ] . Some bands primarily capture stable geometric or reflectance information, while others are more sensitive to surface materials or human activity. As in IF microscopy, these spectral channels are not interchangeable, and modelling approaches must account for their distinct roles. Multi-Channel Vision Transformers. Vision encoders designed for multi-channel imaging (MCI) data have been explored across application domains including climate modelling [ 16 , 20 ] , cell microscopy [ 8 , 14 , 12 ] , and satellite imagery [ 27 ] . Recent work in this area largely falls under the class of multi-channel Vision Transformers (MC-ViTs), which extend standard Vision Transformers (ViTs) [ 13 ] to accommodate MCI data. Unlike vanilla ViTs, which embed spatial patches from all channels

Read the complete text