Right Regions, Wrong Labels: Semantic Label Flips in Segmentation under Correlation Shift
Akshit Achara Yovin Yathathugoda Nicholas Byrne Michela Antonelli Esther Puyol Anton Alexander Hammers Andrew P. King School of Biomedical Engineering & Imaging Sciences, King’s College London, UK {akshit.achara, yovin.yahathugoda, nicholas.byrne, michela.antonelli, esther.puyol_anton, alexander.hammers, andrew.king}@kcl.ac.uk · 2026
Read Right Regions, Wrong Labels: Semantic Label Flips in Segmentation under Correlation Shift by Akshit Achara Yovin Yathathugoda Nicholas Byrne Michela Antonelli Esther Puyol Anton Alexander Hammers Andrew P. King School of Biomedical Engineering & Imaging Sciences, King’s College London, UK {akshit.achara, yovin.yahathugoda, nicholas.byrne, michela.antonelli, esther.puyol_anton, alexander.hammers, andrew.king}@kcl.ac.uk (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13326.txt.
Introduction
Achara and six colleagues show that segmentation models can outline objects correctly yet attach the wrong label when training ties category to scene. They define a Flip diagnostic that counts foreground pixels that stay foreground but take the wrong foreground identity. Stronger training correlation makes common and rare test conditions diverge further and raises such swaps. They also offer an entropy-based flip-risk score, computed without ground truth, that flags vulnerable cases at inference time.
Catalog record
| CREATOR | Akshit Achara Yovin Yathathugoda Nicholas Byrne Michela Antonelli Esther Puyol Anton Alexander Hammers Andrew P. King School of Biomedical Engineering & Imaging Sciences, King’s College London, UK {akshit.achara, yovin.yahathugoda, nicholas.byrne, michela.antonelli, esther.puyol_anton, alexander.hammers, andrew.king}@kcl.ac.uk |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46087 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.13326
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Right Regions, Wrong Labels: Semantic Label Flips in Segmentation under Correlation Shift Akshit Achara Yovin Yathathugoda Nicholas Byrne Michela Antonelli Esther Puyol Anton Alexander Hammers Andrew P. King School of Biomedical Engineering & Imaging Sciences, King’s College London, UK {akshit.achara, yovin.yahathugoda, nicholas.byrne, michela.antonelli, esther.puyol_anton, alexander.hammers, andrew.king}@kcl.ac.uk Abstract The robustness of machine learning models can be compromised by spurious correlations between non-causal features in the input data and target labels. A common way to test for such correlations is to train on data where the label is strongly tied to some non-causal cue, then evaluate on examples where that tie no longer holds. This idea is well established for classification tasks, but for semantic segmentation the specific failure modes are not well understood. We show that a model may achieve reasonable overlap while assigning the wrong semantic label, swapping one plausible foreground class for another, even when object boundaries are largely correct. We focus on this semantic label-flip behaviour and quantify it with a simple diagnostic (Flip) that counts how often ground truth foreground pixels are assigned the wrong foreground identity while remaining predicted as foreground. In a setting where category and scene are correlated during training, increasing the correlation consistently widens the gap between common and rare test conditions and increases these within-object label swaps on counterfactual groups. Overall, our results motivate assessing segmentation robustness under distribution shift beyond overlap by decomposing foreground errors into correct pixels, flipped-identity pixels, and missed-to-background pixels. We also propose an entropy-based, ground truth label-free ‘flip-risk’ score, which is computed from foreground identity uncertainty, and show that it can flag flip-prone cases at inference time. Code is available at https://github.com/acharaakshit/label-flips . 1 Introduction Modern vision models can achieve high accuracy while leaning on ‘spurious’ cues that are predictive of the target label in the training distribution but not causally tied to the task, a behaviour often described within a classification context as shortcut learning ( 4 ) . The spurious cues can be spatially structured, i.e. background context, imaging artefacts, and characteristic textures may correlate with target classes and provide an easier signal than object-centric evidence. When such correlations change at test time, performance can drop even if in-distribution results look strong. Most spurious-correlation benchmarks and evaluation protocols were developed for classification, where a label is intentionally coupled with a nuisance attribute during training and then evaluated on balanced or counterfactual splits that break the association ( 24 ) . Segmentation work has provided clear evidence that dense predictors can also exploit spurious correlations, for example those introduced by acquisition and preprocessing pipelines such as padding and cropping conventions or annotation markers ( 16 ) . However, segmentation is often assessed with overlap-based metrics such as mIoU or Dice, which largely summarise geometric agreement. This emphasis can miss a failure mode specific to multi-class foreground segmentation: models may achieve reasonable foreground overlap yet assign the wrong semantic identity within that region. Conventional metrics will highlight this as a failure but obscure its true nature. In this paper, we first demonstrate the existence of this phenomenon, which we refer to as semantic label flips in segmentation. A label flip occurs when the prediction remains plausible as a foreground mask but swaps between foreground identities in response to a spurious cue. This differs from purely geometric failure modes where the mask collapses, drifts, or fragments. To isolate label flips under controlled distribution shift, we adapt the standard correlation-shift protocol to dense prediction: we vary the association strength between a binary label Y and a binary attribute \mathcal{A} during training, and evaluate on balanced splits that include all (Y,\mathcal{A}) combinations. This separates aligned conditions that dominate training from counterfactual conditions that break the correlation, enabling a direct test of whether semantic identity remains stable when cues change. We conduct an analysis on two different datasets. First, we adapt Waterbirds ( 24 ) to semantic segmentation, where background context provides a canonical shortcut that can affect semantic decisions without necessarily destroying localisation. We refer to this adapted dataset as Waterbirds-seg . Second, we study a real-mask setting built from confusable categories in Coco ( 17 ) , where object identity (cat vs. dog) is correlated with coarse scene context (indoor vs. outdoor) derived from contextual labels. This is similar to contextual bias studied in natural-image shift benchmarks, where background context becomes spuriously tied to category (e.g., MetaShift-style evaluations) ( 14 ; 15 ) . We refer to this dataset as COCO-CD . Across both datasets, increasing the training-time correlation consistently widens the performance gap between aligned and counterfactual groups and markedly increases within-foreground identity swaps, even when class-agnostic foreground overlap remains comparatively stable. Together, these results motivate assessing segmentation robustness under distribution shift beyond overlap by separating semantic flips from foreground deletion errors. Furthermore, we show that an entropy-based, ground truth label-free ‘flip-risk’ score derived from foreground identity uncertainty can highlight flip-prone predictions for lightweight model monitoring. Contributions. We make three contributions. First, we identify a segmentation failure mode under correlation shift in which models preserve plausible foreground extent but swap semantic identity between confusable foreground classes. Second, we adapt the standard correlation-shift protocol to dense prediction and instantiate it on two different datasets: a segmentation adaptation of Waterbirds and a real-mask COCO setting built from confusable categories with coarse indoor/outdoor context. Third, we complement group-wise overlap with diagnostics that separate semantic from geometric errors: for test-time evaluation with ground truth labels, we introduce Flip and a foreground-conditioned (correct/flip/miss) decomposition, and for deployment-time monitoring without ground truth labels, we propose an entropy-based flip-risk score from foreground identity uncertainty that stratifies flip-prone predictions and concentrates flips in a high-risk tail. 2 Related Work 2.1 Shortcut learning and correlation shift. Shortcut learning refers to models relying on predictive but non-causal cues that correlate with the target in the training distribution, but which can lead to brittle behaviour when those correlations change at test time ( 4 ) . In image classification, this is commonly studied through correlation shift: training data is constructed so that the label is strongly associated with a spurious attribute, while evaluation uses balanced or counterfactual splits that break this association ( 24 ) . Surveys and benchmarks have refined these protocols across vision tasks ( 32 ) . We use the same lens for dense prediction, but focus on a segmentation-specific consequence of cue shift: predictions can remain geometrically plausible while the semantic identity within the foreground becomes unstable. 2.2 Spurious correlations and context dependence in semantic segmentation. Reliance on spurious correlations in segmentation has been documented when shortcut signals arise from the acquisition or annotation pipeline. For example, in medical image segmentation, 16 identified sources such as padding and preprocessing artefacts, cropping conventions, and annotation markers, and showed that segmentation models can exploit these signals despite their lack of anatomical meaning. More broadly, robustness work in dense prediction often studies failures under domain or style shift and reports degradation using overlap metrics such as Dice or mIoU; recent surveys summarise this landscape for semantic segmentation ( 21 ; 25 ) . Overlap metrics such as Dice or mIoU capture how a model performs on segmentation, but they can blur the distinction between geometric failure and semantic