Mamba-SSM with LLM Reasoning for Feature Selection: Faithfulness-Aware Biomarker Discovery

Pushpa Kumar Balan, Aijing Feng · 2026

Read Mamba-SSM with LLM Reasoning for Feature Selection: Faithfulness-Aware Biomarker Discovery by Pushpa Kumar Balan, Aijing Feng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14334.txt.

Introduction

Balan and Feng test whether LLM chain-of-thought reasoning can filter tissue-composition confounders from saliency gene lists. A Mamba model on TCGA-BRCA data yields a top-50 saliency set, and DeepSeek-R1 checks candidates to produce a final 17-gene set. The raw 50-gene set trails a 5,000-gene baseline, while the LLM-filtered set beats it with 294 times fewer features. A faithfulness audit validates 6 of 17 selected genes as BRCA biomarkers, while 10 of 16 known input genes were missed. That gap shows targeted confounder removal can lift performance without comprehensive recall.

Catalog record

CREATORPushpa Kumar Balan, Aijing Feng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18390
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14334

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mamba-SSM with LLM Reasoning for Feature Selection: Faithfulness-Aware Biomarker Discovery Pushpa Kumar Balan, Aijing Feng Abstract Gradient saliency from deep sequence models surfaces candidate biomarkers efficiently, but the resulting gene lists can be contaminated by tissue-composition confounders that degrade downstream classifiers. We study whether LLM chain-of-thought (CoT) reasoning can filter these confounders, and whether reasoning quality is associated with downstream performance. We train a Mamba SSM on TCGA-BRCA RNA-seq and extract the top-50 genes by gradient saliency; DeepSeek-R1 evaluates every candidate with structured CoT to produce a final 17-gene set. On the held-out test split, the raw 50-gene saliency set (no LLM) performs worse than a 5,000-gene variance baseline (AUC 0.832 vs. 0.903), while the LLM-filtered set surpasses it (AUC 0.927 ), using 294 × \times fewer features. A faithfulness audit (COSMIC CGC, OncoKB, PAM50) shows that 6 of 17 selected genes (35.3%) are validated BRCA biomarkers, while 10 of 16 known BRCA genes present in the input were missed—including FOXA1. This divergence between downstream performance and reasoning faithfulness suggests selective faithfulness in this setting: targeted confounder removal can improve predictive performance without comprehensive recall. The scope of this claim and its limitations are examined in Sec. B . Code: https://github.com/pushpakumarbalan/feature-selection 1 Introduction High-dimensional RNA-seq data ( > 20,000 genes per sample) presents a severe feature-selection problem: most genes are irrelevant to the phenotype of interest, and many that appear predictive are confounders (immune infiltration, tumour purity, batch effects) rather than disease drivers ( Pudjihartono et al. 2022 ) . Standard gradient-based saliency from neural models ranks genes by their gradient magnitude, but this signal reflects what the model learned to use , not biological causality. The top-50 saliency genes from a well-trained Mamba SSM on TCGA-BRCA include muscle-specific genes (MB, UTRN), general immune markers (HLA-DRB1, ITGAL), unannotated lncRNAs, and antisense RNAs with no documented breast cancer role, all of which carry saliency simply because they co-vary with tumour samples at the RNA-seq level. This creates a natural role for LLM reasoning: the model’s encoded biomedical knowledge can in principle distinguish disease drivers (e.g., the ER-signalling gene XBP1, the EMT regulator ZEB1) from these confounders without additional data. But does the LLM’s stated reasoning actually reflect accurate biological knowledge? And is downstream performance a reliable proxy for reasoning faithfulness? We make three contributions: 1. We demonstrate that raw saliency-based feature selection hurts performance relative to a variance baseline (AUC - 0.071), while LLM-filtered selection helps (AUC + 0.024), establishing that LLM reasoning is empirically important in this pipeline. 2. We conduct a faithfulness audit comparing the LLM’s selected gene set against curated BRCA ground-truth databases, revealing a recall of 0.375 on known BRCA genes while still achieving superior downstream AUC, a finding we term selective faithfulness : targeted removal of known non-BRCA genes is sufficient for performance gains, even without comprehensive recall of all true positives. 3. We identify a concrete failure mode: FOXA1, the most important luminal breast cancer pioneer transcription factor and a canonical PAM50 gene, was present in the input but rejected by the LLM— illustrating that LLM biomedical reasoning can be confidently wrong on well-established facts. 2 Related Work LLMs for feature selection: LLM-Select ( Jeong et al. 2025 ) showed that zero-shot LLM feature selection using only feature names can match LASSO on tabular data in some settings. LLM-Lasso ( Zhang et al. 2025 ) integrates domain knowledge through LLM-guided regularisation, penalising literature-supported features less. FreeForm ( Lee et al. 2025 ) demonstrated that LLM ensembling improves variant selection in low-data genomic regimes. Our work differs by studying faithfulness , we audit whether the LLM’s stated biological rationale is correct, not just whether the output improves a metric. Faithfulness of LLM reasoning: A growing body of work questions whether CoT explanations reflect the model’s actual reasoning process ( Turpin et al. 2023 ; Lanham et al. 2023 ) . Most such studies use synthetic tasks with verifiable ground truth. We provide a real-world biological instance: a domain where ground truth (validated cancer driver genes) exists but is large, overlapping, and context-dependent, making faithfulness harder to assess and more practically important. SSMs for genomics: Mamba ( Gu & Dao 2024 ) scales linearly in sequence length, making it tractable for 20,000-dimensional gene expression vectors without attention’s quadratic cost. We use the official mamba-ssm implementation with gradient saliency ( Simonyan et al. 2014 ) to extract a biologically plausible candidate pool before the LLM reasoning step. 3 Methodology 3.1 System Pipeline Figure 1: End-to-end neuro-symbolic pipeline. High-dimensional TCGA-BRCA RNA-seq is processed by Mamba SSM to produce gradient saliency scores (top-50 candidate genes), then filtered by structured DeepSeek-R1 CoT reasoning to form a 17-gene BRCA-specific subset used in downstream comparison. 3.2 Data We use TCGA-BRCA RNA-seq data: 1,095 tumour and 113 matched normal samples, \sim 20,000 protein-coding and non-coding genes, quantified as TPM. Labels are binary: Tumour (1) / Normal (0). 3.3 Phase 1 — Mamba SSM Training Input features are log 2 (TPM + 1)-normalised and filtered to the top-5,000 highest-variance genes. We train an OfficialMambaClassifier : a linear embedding layer projects each gene’s scalar expression value into a d_{\text{model}}=128 dimensional space; a single Mamba block ( d_{\text{state}}=16 , d_{\text{conv}}=4 , expand =2 ) models long-range dependencies across the gene sequence; adaptive average pooling collapses the sequence dimension; a final linear layer with sigmoid produces the tumour probability. Training uses AdamW ( \text{lr}=10^{-4} , 15 epochs, batch size 8) with class-weighted BCELoss ( w_{\text{normal}}=N_{\text{tumour}}/N_{\text{normal}} ) to handle the 8.8:1 tumour/normal imbalance. 3.4 Phase 2 — Gradient Saliency Extraction For each tumour sample, we enable gradients on the input, perform a forward pass, compute the loss, and backpropagate. The gene importance score is s_{j}=\frac{1}{|T|}\sum_{i\in T}\left|\frac{\partial\mathcal{L}}{\partial x_{ij}}\right| , averaged over all tumour samples T . The top-50 genes by s_{j} form the candidate pool \mathcal{G}_{50} . 3.5 Phase 3 — Structured CoT Reasoning \mathcal{G}_{50} is passed to DeepSeek-R1 (7B, local via Ollama, temperature =0.3 ) with a structured prompt that (a) provides saliency scores, (b) explicitly states high saliency does not imply BRCA specificity, (c) requires evaluation of every gene via five rejection criteria (R1–R5) and three keep criteria (K1–K3), and (d) forbids rank-order selection. A post-hoc audit detects rank-copy solutions and hallucinated gene names. The selected subset contains 17 genes and is used as the B3 input set in downstream comparison. 3.6 Phase 4 — Faithfulness Audit The 17 selected genes are cross-referenced against a 101-gene ground-truth set (COSMIC CGC Tier-1, OncoKB BRCA annotations, PAM50 ( Parker et al. 2009 ) , and established pathway genes). A known non-BRCA set (muscle, neuronal, unannotated genes) provides true-negative ground truth. For each of the 50 input genes we record the ground-truth class (validated / known non-BRCA / unknown) and the LLM selection outcome (selected / not selected), yielding selection-level precision, recall, and missed-gene analysis. 4 Results 4.1 Downstream Classification Benchmark Each gene set is evaluated by training an identical Mamba classifier (same architecture, same random seed, same 80/20 stratified split) on the respective feature subset. Results are in Table 1 . Table 1: Classification performance on the held-out test set (20% of 1,231 samples). All conditions use the same Mamba architecture and training procedure; only the input gene set changes.

Read the complete text