Mixing Configurations for Downstream Prediction

Juntang Wang Hao Wu Runkun Guo Yihan Wang Dongmian Zou Shixin Xu · 2025

Read Mixing Configurations for Downstream Prediction by Juntang Wang Hao Wu Runkun Guo Yihan Wang Dongmian Zou Shixin Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19248.txt.

Introduction

Juntang Wang and four co-authors present Configuration-Mixed Prediction. It uses clustering features at many resolutions at once. Most methods force one global choice of resolution. Recent work shows only finitely many stable splits arise. Each stable split is called a configuration.

Their module is called MixConfig. It pulls configurations from any frozen embedding. For each sample, it learns how to weight them. The weights use context, cluster assignment, and stability scores. Tests span tabular, molecular, vision, and text data. The mixed features help prediction. The base model needs no retraining.

Catalog record

CREATORJuntang Wang Hao Wu Runkun Guo Yihan Wang Dongmian Zou Shixin Xu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19248

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mixing Configurations for Downstream Prediction Juntang Wang Hao Wu Runkun Guo Yihan Wang Dongmian Zou Shixin Xu Abstract Humans possess an innate ability to group objects by similarity—a cognitive mechanism that clustering algorithms aim to emulate. Recent advances in community detection have enabled the discovery of configurations —valid hierarchical clusterings across multiple resolution scales—without requiring labeled data. In this paper, we formally characterize these configurations and identify similar emergent structures in register tokens within Vision Transformers. Unlike register tokens, configurations exhibit lower redundancy and eliminate the need for ad hoc selection. They can be learned through unsupervised or self-supervised methods, yet their selection or composition remains specific to the downstream task and input. Building on these insights, we introduce GraMixC, a plug-and-play module that extracts configurations, aligns them using our novel Reverse Merge/Split (RMS) technique, and fuses them via attention heads before forwarding them to any downstream predictor. On the DSNI 16S rRNA cultivation-media prediction task, GraMixC improves the R 2 from 0.6 to 0.9 on various methods, setting a new state-of-the-art. We further validate GraMixC across standard tabular benchmarks, where it consistently outperforms single-resolution and static-feature baselines. 1 Introduction Figure 1: Illustration of CIFAR10 configurations. Each column represents a configuration—clustering at a specific resolution. Learning general-purpose features that enhance downstream tasks has been a long-standing goal in machine learning. One prominent example is clustering ( i.e. , community detection) in unsupervised learning, which groups entities into clusters of similar objects while separating dissimilar ones, without using labels. [ 1 , 2 , 3 ] . Interestingly, this paradigm demonstrates remarkable similarities to human-like behaviors. Decades of cognitive science studies show that even infants have the ability to group objects by similarity [ 4 , 5 ] . In particular, they often organize them at different abstraction levels [ 6 , 7 ] . Inspired by this, recent advances in community detection have extended clustering to the discovery of configurations —hierarchical clusterings that span multiple resolution scales [ 8 ] . For example, as illustrated in the lineage diagram of Fig. 1 , in the CIFAR10 dataset [ 9 ] , coarse configurations may separate vehicles from animals, while finer configurations distinguish between birds, cats, and dogs. These multi-resolution representations reveal rich hierarchical structures that could provide stronger priors or inductive biases for deep models. However, despite their potential, such configurations remain largely underexplored in deep learning, especially in challenging domains where labels are sparse. One such domain is 16S ribosomal RNA (rRNA) gene sequencing, a widely used tool in microbiome studies for identifying and classifying bacteria. Analyzing 16S rRNA data has consistently confronted significant challenges in downstream prediction tasks within label-scarce environments. Previous works in 16S rRNA representation learning have demonstrated substantial benefits for bacterial taxonomic profiling and microbial community analysis [ 10 , 11 , 12 ] . Notably, [ 13 ] showed that full-length sequencing combined with appropriate clustering of intragenomic sequence variation can provide more accurate representation of bacterial species in microbiome datasets. These findings underscore the importance of learning clustered representations without relying on labels. Recent methodologies typically transform clustering results into pseudo-labels to enhance downstream prediction performance. For instance, DeepCluster [ 14 ] iteratively clusters CNN-extracted visual features and leverages these cluster assignments to guide network parameter updates. Graph-based methods such as [ 15 ] employ structural clustering to overcome limitations of traditional contrastive learning approaches that depend on positive and negative sample pairs. Their method captures structural relationships among nodes in heterogeneous information networks, establishing a self-supervised pre-training framework that learns robust network representations from unlabeled data. Nevertheless, aforementioned approaches predominantly focus on a single configuration type, overlooking the potential benefits of mixing configurations across multiple resolution scales. In this paper, we introduce GraMixC, a plug-and-play module that extracts, aligns and mixes graph-based configurations for downstream prediction. The main contributions of the paper are as follows: • We identify three key characteristics of clustering configurations through systematic experimental analysis, providing a novel perspective on enhancing downstream prediction via mixing configurations. • We propose GraMixC, a plug-and-play module based on mixed configurations. We apply it to a novel 16S rRNA cultivation-media prediction task, setting a new state-of-the-art. • We further conduct extensive experiments on multiple standard tabular benchmarks to validate GraMixC’s effectiveness, where it consistently outperforms single-resolution and static-feature baselines. The remainder of this paper is organized as follows. Section 2 analyzes behavioral patterns of configurations. Section 3 details our proposed GraMixC. Section 4 evaluates GraMixC’s performance through extensive experiments. Finally, Section 5 concludes the paper. 2 Preliminary results We first present preliminary experimental results on configurations using CIFAR10. Specifically, we compare patterns of configurations with those of the learnable “register” tokens in a recent vision transformer DINOv2-reg [ 16 ] . Fig. 2 shows the attention maps from our configurations and their register tokens. Moreover, Fig. 3 shows qualitative behaviors of our configurations and their quantitative advantages over registers in terms of feature importance and neighborhood similarity. From these results, we identify three key properties: Configurations emerge via unsupervised or self-supervised learning. We define Near ground truth (GT) balls as balls selected with the highest clustering scores, marked yellow in Fig. 2(a) . As shown in Fig. 2(b) , the attention map, acquired by feeding configurations as tokens to attention heads for linear probing, yields high norm regions substantially overlap with GT balls. On another hand, DINOv2-reg exhibits similar attention map patterns in selected registers (see Fig. 2(c) ), which might be related to registers activating different areas in Fig. 2(d) , similar to slot attention [ 17 , 18 , 19 , 16 ] . Thus, based on the similar attention map behavior, register token can be considered as a latent configuration. Configurations are selected and mixed based on input and task. Configuration selection and mixing refers to learning which resolution scales to focus on for a given downstream task. We visualize this via attention maps over configuration tokens, where high-norm regions indicate the selected scales. In Fig. 2(b) , attention norms vary across rows, showing that each input sample triggers different resolution scales. Without any change to the configurations, we merge the original labels into coarser classes ( Fig. 3(a) ) and plot the new attention map ( Fig. 3(b) ). The attention shifts to align with the coarser GT, whereas DINOv2-reg register tokens remain unchanged unless re-trained. These observations confirm that configuration selection and mixing are input- and task-dependent. Configurations are more informative and less redundant than register tokens. Register tokens can help extract configurations, similar to object detection [ 20 , 21 ] , but selecting a fixed number by feature importance is arbitrary and non-rigorous (see Fig. 3(c) ). Furthermore, register tokens exhibit high redundancy—cosine similarity between their embeddings and their 2 neighbors embeddings is heavily skewed toward 1—whereas configurations yield information less redundant (see Fig. 3(d) ). (a) Configurations (b) Cfg. attention map (c) Reg. attention map (d) Reg. attention imgs Figure 2: Comparison of attention maps obtained from configurations and registers, rows for samples. (a) : Lineage diagram for configurations, near GT balls are marked yellow. (b) : Attention map of configuration tokens in an attention-based linear probing. (c) : Attention map of DINOv2-reg register tokens,

Read the complete text