Abundance-Aware Set Transformer for Microbiome Sample Embedding Conference: ; ; DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies CCS: Applied computing
Hyunwoo Yoo, Gail Rosen · 2025
Read Abundance-Aware Set Transformer for Microbiome Sample Embedding Conference: ; ; DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies CCS: Applied computing by Hyunwoo Yoo, Gail Rosen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11075.txt.
Introduction
Hyunwoo Yoo and Gail Rosen published a 2025 paper on representing a microbiome sample so a computer can learn from it. Each sample is a community: thousands of microbial sequences in varying abundance. Jobs, predicting a trait or telling environments apart, need it compressed into one fingerprint. The common shortcut averages the sequences' embeddings, treating a rare microbe and a dominant one as equals.
Their method weights by that share without changing the model's design. Sequence embeddings are copied in line with their share, then a set transformer attends over the pile and pulls them together. Abundant members thus speak louder, as they do in life. On microbiome tasks, the approach beat averaging and the unweighted model. The lesson is general: when summing up a community, count its members. That share is information, and embeddings should not throw it away.
Catalog record
| CREATOR | Hyunwoo Yoo, Gail Rosen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-11075
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Abundance-Aware Set Transformer for Microbiome Sample Embedding Conference: ; ; DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies CCS: Applied computing Hyunwoo Yoo, Gail Rosen Abstract . Microbiome sample representation to input into LLMs is essential for downstream tasks such as phenotype prediction and environmental classification. While prior studies have explored embedding-based representations of each microbiome sample, most rely on simple averaging over sequence embeddings, often overlooking the biological importance of taxa abundance. In this work, we propose an abundance-aware variant of the Set Transformer to construct fixed-size sample-level embeddings by weighting sequence embeddings according to their relative abundance. Without modifying the model architecture, we replicate embedding vectors proportional to their abundance and apply self-attention-based aggregation. Our method outperforms average pooling and unweighted Set Transformers on real-world microbiome classification tasks, achieving perfect performance in some cases. These results demonstrate the utility of abundance-aware aggregation for robust and biologically informed microbiome representation. To the best of our knowledge, this is one of the first approaches to integrate sequence-level abundance into Transformer-based sample embeddings. 1. Introduction Microbiome samples contain thousands of short DNA sequences derived from diverse microbial species in an environmental or host-associated context ( Mendler et al. 2024 ) . A central challenge in microbiome analysis is to transform such variable-length, unordered sets of sequences into fixed-size, informative representations that can support downstream tasks such as phenotype prediction, environmental classification, or disease detection. These sample-level embeddings are not only critical for training classifiers such as deep neural networks or random forests, but also serve as the basis for similarity comparisons, clustering, and visualization. Recent approaches typically employ pretrained DNA language models such as DNABERT ( Zhou et al. 2023 ) or the Nucleotide Transformer ( Dalla-Torre et al. 2023 ) to compute embeddings for individual sequences. These sequence embeddings are then aggregated—usually via mean or max pooling—to obtain a single vector per sample ( Zaheer et al. 2017 ) . While simple and computationally efficient, these aggregation strategies fail to account for key biological signals such as the abundance of each 16S rRNA and/or amplicon sequence variant (ASV), which reflects the relative prevalence of microbial species in the sample ( Oh and Zhang 2020 ; Mandal et al. 2015 ) . Abundance is not merely a statistical measure; it often carries meaningful biological information. For instance, the abundance of certain taxa may indicate infection status, environmental perturbation, or ecological balance. In disease-related microbiome studies, rare pathogenic organisms may increase in abundance over time, while in ecological monitoring, the presence and concentration of specific indicator taxa can signal pollution levels or habitat changes. In Zhao et al. ( Zhao et al. 2021 ) , 3 types of sample classification are considered: 1) reads are classified to sample attributes, and the overall sample attribute is the final phenotype classification; 2) the sample-level embedding method forms a sample-level vector representation by averaging all read-level embeddings in a query sample; and 3) for the Pseudo OTU method, as described by Woloszynek et al. ( Woloszynek et al. 2019 ) , reads are embedded as vectors and are clustered into Pseudo OTUs (groupings of related read vectors). Then, each query sample’s reads were assigned to those Pseudo OTUs based on distance. The latter method (c) ended up having the best performance, since it took into account the abundance of taxa the best. Simple averaging and the majority vote obscured taxa-abundance, which are important signals, especially when low-abundance but high-importance sequences are diluted by a large number of irrelevant ones. To address this limitation, we propose an abundance-aware Set Transformer architecture for constructing sample embeddings. The Set Transformer ( Lee et al. 2019 ) is a permutation-invariant attention-based model that has shown promise in learning representations from unordered sets. Our proposed method integrates abundance information through two strategies: (1) repetition-based weighting , where each sequence embedding is repeated in proportion to its abundance, and (2) soft attention weighting , where abundance is incorporated directly into the pooling weights. These methods require no architectural changes and can be easily implemented with existing attention-based modules. Figure 1. Overview of the abundance-aware set transformer architecture. (a) End-to-end predictive pipeline in which input DNA sequences are first encoded into contextual embeddings by a DNA language model, then aggregated by an abundance-aware set transformer to produce a fixed-size, weighted representation, and finally passed to a downstream classifier that outputs class probabilities (e.g. pathogen or not). The classifier (a fully-connected neural network or random forest) is trained on a labeled downstream dataset. (b) Abundance-aware Set Transformer architecture, where a DNA language model encoder (stack of multi-head attention, feed-forward networks, and layer normalization) first encodes each sequence into a contextual representation, followed by a Set Transformer comprising ISAB, PMA, and SAB blocks to aggregate sequence embeddings into a fixed-size set representation, and an abundance encoding mechanism that weights the set elements according to their relative sequence abundance before pooling. Specifically, ISAB (Induced Set Attention Block) efficiently reduces computational complexity by introducing a small number of inducing points to capture set-wise interactions, PMA (Pooling by Multihead Attention) uses learnable seed vectors to produce a fixed-size embedding that summarizes the entire set, and SAB (Set Attention Block) refines these representations by modeling self-attention among the pooled set embeddings to capture higher-order relationships. Unlike traditional models that treat DNA sequences independently, our approach captures both the content and structure of the full microbiome sample by leveraging abundance during the aggregation step. This allows the model to focus on biologically meaningful signals, improving both performance and interpretability. We evaluate our method across three real-world microbiome classification tasks involving both clinical and environmental samples. Our results show that abundance-aware Set Transformers consistently outperform standard mean pooling and vanilla Set Transformers. In some cases, our model even achieves perfect classification accuracy. These findings underscore the importance of integrating abundance as a first-class signal in set-based representations for microbiome analysis. Our contributions are summarized as follows: • We propose an abundance-aware Set Transformer to generate biologically meaningful sample-level embeddings from microbiome sequencing data. • We compare four aggregation strategies—mean pooling, soft weighting, Set Transformer, and abundance-aware Set Transformer—across three distinct classification tasks. • We demonstrate that incorporating abundance significantly improves model performance, especially in challenging cross-study settings, without requiring modifications to the base Transformer architecture. 2. Related Work 2.1. Pretrained DNA language models. Large-scale language models trained on genomic sequences have significantly advanced sequence-level representation learning. DNABERT ( Ji et al. 2021 ; Zhou et al. 2023 ) and Nucleotide Transformer ( Dalla-Torre et al. 2023 ; Mendoza-Revilla et al. 2024 ) employ transformer based architectures ( Vaswani et al. 2017 ) to embed k-mer tokenized DNA sequences, and have demonstrated strong performance in tasks such as promoter prediction, splice site detection, and enhancer identification. However, these models operate at the level of individual sequences and do not directly address the challenge of aggregating thousands of such embeddings into a coherent sample-level representation. Table 1. FCNN classification and Random Forest results on Study 14245. Embedding Method Classifier Accuracy Macro Precision Macro Recall Macro F1 Set Transformer FCNN 0.5417 0.2708 0.5000 0.3514 Average FCNN 0.5417 0.2708 0.5000 0.3514 Weighted Average FCNN 0.5417 0.2708 0.5000 0.3514 Weighted Set Transformer FCNN 0.5417 0.2708 0.5000