Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathology

Franciskus Xaverius Erick, Johanna Paula Müller, Bernhard Kainz · 2026

Read Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathology by Franciskus Xaverius Erick, Johanna Paula Müller, Bernhard Kainz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18419.txt.

Introduction

Franciskus Xaverius Erick, Johanna Paula Müller, and Bernhard Kainz present GAUC, a 2026 training-free method for picking demo examples. The target is vision-language models in histopathology, where fine-tuning is costly and example choice swings results. GAUC works in the pretrained multimodal embedding space, balancing three goals: distributional fidelity, robustness to prompt paraphrases, and a penalty on shaky outputs. On CRC-100K and MHIST it matches the strongest baselines on accuracy while lifting calibration and prompt robustness.

Catalog record

CREATORFranciskus Xaverius Erick, Johanna Paula Müller, Bernhard Kainz
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30510
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18419

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathology Franciskus Xaverius Erick, Johanna Paula Müller, Bernhard Kainz Abstract Vision-language models (VLMs) can couple visual perception with open-ended clinical reasoning, making them attractive for computational histopathology. However, fine-tuning billions of parameters on scarce, expert-annotated pathology data is prohibitive, while in-context learning (ICL), which conditions the VLM on demonstrative image-text pairs without parameter updates, suffers from high sensitivity to which examples are selected and how the query is phrased, producing unreliable diagnostics. Existing selection strategies rely on query-dependent nearest-neighbour retrieval that ignores global data structure, require costly parameter updates, or disregard the joint vision-text embedding geometry of VLMs. We propose GAUC, a training-free coreset selection method operating directly in the pre-trained multimodal embedding space. GAUC jointly optimises three objectives: (1) a Maximum Mean Discrepancy term enforcing distributional fidelity between coreset and full dataset, (2) an Effective Mutual Information Difference regulariser bounding performance degradation under prompt paraphrases by exploiting the VLM’s joint vision-text alignment, and (3) a predictive-variance penalty suppressing overconfident, unstable outputs. On CRC-100K and MHIST across multiple open-source VLM architectures, GAUC consistently improves accuracy, calibration, and prompt robustness over recent ICL selection methods and dataset-distillation baselines, all without a single gradient update. Code is available at Github Repository 1 Introduction Histopathological examination remains the diagnostic gold standard for most solid tumours, requiring trained pathologists to visually assess tissue morphology at high magnification, a process that is time-consuming, subjective, and bottlenecked by a global shortage of specialist expertise [ 5 ] . With colorectal cancer alone accounting for over 1.9 million new cases and 935,000 deaths annually [ 10 ] , scalable tools that support high-throughput tissue classification are needed. The widespread adoption of automated sample preparation pipelines and whole-slide image (WSI) scanners has further intensified this demand by dramatically accelerating the rate at which digitised histology data is generated, placing mounting workload pressure on pathologists [ 5 ] . Pre-trained vision-language models (VLMs) offer a compelling path forward: by coupling a vision encoder with a large language backbone, they interpret complex medical imagery while producing human-readable diagnostic rationales that mirror clinician reasoning [ 16 , 14 ] . Unlike conventional supervised classifiers, VLMs can simultaneously process visual evidence and contextual clinical descriptions, enabling more holistic and interpretable tissue analysis [ 9 ] . However, deploying VLMs clinically is hindered by the intractability of fine-tuning billions of parameters on scarce, privacy-sensitive data [ 14 ] and by persistent overconfident hallucinations that are unacceptable in safety-critical settings [ 22 , 4 ] . In-context learning (ICL) sidesteps the first obstacle by conditioning the model on demonstrative image-label pairs placed directly in the prompt, requiring no parameter updates [ 3 ] . This capability, originally observed in large language models, transfers naturally to VLMs [ 1 , 2 , 15 ] . Yet ICL is notoriously sensitive to both the choice of demonstrations and the phrasing of the textual query [ 19 , 29 , 17 ] . Current remedies either insert learnable shift modules that require additional training [ 12 , 17 ] , or rely on nearest-neighbour retrieval that captures local query similarity but ignores the global distributional structure of the dataset [ 9 ] . Neither accounts for the joint visual-textual embedding geometry of the VLM, nor explicitly controls for prompt-induced instability or predictive overconfidence, shortcomings that are especially damaging in histopathology where class imbalance obscures minority morphologies and minor prompt reformulations can flip a diagnosis. We propose a principled, entirely training-free coreset selection framework for visual ICL that jointly optimises representativeness, prompt robustness, and predictive calibration. Our contributions are: 1. A geometry-aware objective based on Maximum Mean Discrepancy (MMD) that selects real, clinically traceable demonstrations whose embedding distribution preserves the global structure of the full dataset, preventing collapse into dominant morphological clusters. 2. An Effective Mutual Information Difference (EMID)-derived mutual-information regulariser that quantifies response discrepancies under paraphrased queries, steering selection toward in-context sets robust to textual variation across both visual and textual modality embeddings. 3. A variance regularisation term that penalises predictive uncertainty, encouraging demonstrations that minimise output entropy and suppress overconfident hallucinations. 4. Consistent improvements over nearest-neighbour and random baselines on two challenging histopathology benchmarks, CRC-100K (8-class tissue subtyping) and MHIST (binary polyp classification), with fully training-free deployment compatible with any off-the-shelf VLM. Related Work. Recent works in ICL [ 18 ] have formalised demonstrations as latent shift vectors that steer query-token representations, and extended this view to multimodal models through lightweight query-dependent shift modules [ 12 ] and prompt-configuration strategies [ 17 ] . These methods are effective but require learnable parameters or task-specific inference modifications. In computational pathology, k -Nearest Neighbour ( k NN) retrieval of demonstration images has matched or exceeded specialised fine-tuned networks [ 9 ] , yet k NN selects solely by local query proximity. Two broader strategies exist for constructing representative subsets: dataset distillation synthesises artificial samples by matching gradient trajectories, feature distributions, or diffusion-based latent mappings [ 28 , 8 , 7 ] but is computationally expensive and produces clinically untraceable images, whereas coreset selection retrieves real samples and can enforce distributional fidelity through statistical distances such as Maximum Mean Discrepancy [ 25 ] . Curriculum-based adaptation on biomedical figure-caption corpora effectively aligns VLM semantics with visual features [ 16 ] , yet clinical VLMs remain prone to overconfident hallucinations and prompt-induced output instability [ 22 , 4 , 14 ] . The Effective Mutual Information Difference (EMID) upper-bounds the performance degradation from multimodal alignment shifts via Jensen-Shannon divergence in the joint latent space [ 21 ] . The quality of any geometry-aware selection further depends on the feature space; self-supervised Vision Transformers pre-trained on histopathology data through masked image modelling [ 11 ] provide the semantically rich embeddings required for distributional metrics to reflect genuine morphological variation. 2 Method Figure 1 : The GAUC optimization pipeline. GAUC involves 1) MMD matching between the embedded distribution of the full dataset and the selected coreset examples, 2) effective mutual information upper bound regularisation between outputs with original text prompts and paraphrased prompts, and 3) variance uncertainty minimisation of selected coreset examples. Let W_{\theta} denote a pre-trained VLM with frozen parameters \theta . Given a full labelled dataset F=\{(x_{i},y_{i})\}_{i=1}^{|F|} of histopathology patches and their class labels, we seek a compact coreset \mathcal{D}=\{(x_{j},y_{j})\}_{j=1}^{|\mathcal{D}|} with |\mathcal{D}|\ll|F| that, when placed in the prompt as in-context demonstrations, maximises diagnostic accuracy while remaining robust to prompt variation and predictive overconfidence. We optimise \mathcal{D} by minimising a composite objective over three complementary terms (Fig. 1 ), each operating entirely in the pre-trained embedding space of W_{\theta} without any gradient-based parameter updates. Geometry-aware coreset selection via MMD. We require the embedding distribution of \mathcal{D} to faithfully mirror that of F . Let x,x^{\prime} denote independent samples from F and u,u^{\prime} from \mathcal{D} , mapped through the vision encoder of W_{\theta} . We measure distributional discrepancy with the Maximum Mean Discrepancy in a reproducing kernel Hilbert space [ 23 ] : \mathrm{MMD}^{2}(F,\mathcal{D})=\mathbb{E}_{x,x^{\prime}}\!\big[k(x,x^{\prime})\big]+\mathbb{E}_{u,u^{\prime}}\!\big[k(u,u^{\prime})\big]-2\,\mathbb{E}_{x,u}\!\big[k(x,u)\big], (1) where k(x,u)=\exp\!\bigl(-\|x-u\|^{2}/2\sigma^{2}\bigr) is the RBF kernel. Minimising \mathrm{MMD}^{2} enforces a geometric constraint that prevents the coreset from collapsing into locally dominant clusters and ensures global representativeness across all tissue morphologies. Prompt-robustness regularisation via EMID. Minor textual rephrasing of

Read the complete text