CTSCAN: Evaluation Leakage in Chest CT Segmentation and a Reproducible Patient-Disjoint Benchmark
Anton Ivchenko · 2026
Read CTSCAN: Evaluation Leakage in Chest CT Segmentation and a Reproducible Patient-Disjoint Benchmark by Anton Ivchenko (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15561.txt.
Introduction
Ivchenko exposes evaluation leakage in chest CT segmentation, where mixing slices from one study across train and test inflates reported scores. His CTSCAN benchmark enforces patient-disjoint splits across 89 cases from three sources. Slice-mixed scoring reaches 0.6665 foreground Dice against 0.2066 under case-disjoint scoring, a 69 percent relative drop that shows how much published performance rests on patient reuse.
Catalog record
| CREATOR | Anton Ivchenko |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17759 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.15561
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CTSCAN: Evaluation Leakage in Chest CT Segmentation and a Reproducible Patient-Disjoint Benchmark Anton Ivchenko Abstract Abstract. Reported chest CT segmentation performance can be strongly inflated when train and test partitions mix slices from the same study. We present CTSCAN, a reproducible multi-source chest CT benchmark and research stack designed to measure what survives under patient-disjoint evaluation. The current four-class artifact aggregates 89 cases from PleThora, MedSeg SIRM, and LongCIU, and we show that the original slice-PNG workflow induces near-complete case reuse across train, validation, and test. Using the playground environment and an Apple-GPU validation environment, we run a multi-seed protocol sweep with the same FPN plus EfficientNet-B0 control configuration under slice-mixed and case-disjoint evaluation. Across 3 seeds and 12 epochs per seed, the slice-mixed protocol reaches 0.6665 \pm 0.0067 foreground Dice and 0.5031 \pm 0.0070 foreground IoU, whereas the case-disjoint protocol reaches 0.2066 \pm 0.0509 Dice and 0.1181 \pm 0.0313 IoU. Removing patient reuse therefore reduces foreground Dice by 0.4599 absolute (69.00% relative) and foreground IoU by 0.3850 absolute (76.52% relative). CTSCAN packages the corrected benchmark with deterministic split manifests, explicit weak-supervision controls, a scripted multi-seed protocol sweep, and reproducible figure generation, providing a reusable basis for patient-disjoint chest CT evaluation. Keywords: reproducibility, chest CT segmentation, benchmark construction, patient-disjoint evaluation, medical imaging 1 Introduction Chest CT segmentation pipelines are often implemented on top of per-slice PNG exports from volumetric studies. That design is convenient, but it creates a serious evaluation risk: if slices are partitioned directly, train and test sets can share patients even when filenames differ. In that regime, reported metrics reflect patient reuse as much as they reflect model quality. This paper studies that problem through CTSCAN, a public multi-source chest CT segmentation benchmark designed to make patient-disjoint evaluation the default reproducible path. The central scientific question is not whether one architecture can be tuned to a strong number. It is whether the evaluation protocol changes the conclusion about what actually generalizes. Our thesis is that a large fraction of apparent chest CT segmentation performance can be explained by evaluation leakage, and that the right benchmarking question is what signal remains after enforcing patient-disjoint evaluation. This is consistent with broader warnings that leakage can change the apparent scientific conclusion in machine-learning-based research settings [ 2 ] . CTSCAN provides a reproducible way to answer that question through a deterministic case-disjoint benchmark built from a public multi-source artifact. We therefore ask three questions: 1. How much does slice-mixed evaluation inflate reported chest CT segmentation performance? 2. What performance remains once the same benchmark is evaluated with patient-disjoint splits? 3. What public artifact is needed to make patient-disjoint evaluation reproducible, auditable, and easy to rerun? Our answer is that protocol choice changes the absolute scientific conclusion, not just the benchmark number. We make three concrete contributions: • We expose a strong evaluation-leakage effect in a slice-derived chest CT segmentation benchmark: partitioning individual PNG pairs rather than case IDs induces near-complete case reuse across train, validation, and test. • We release a deterministic patient-disjoint benchmark protocol built from public chest CT sources, exact split manifests, overlap auditing, and a public benchmark page that ranks models only under patient-disjoint evaluation. • We pair that benchmark with a scripted multi-seed study that runs matched controls across evaluation protocols, learning curves, and controlled training variants so the leakage claim is supported by executable evidence rather than by one-off notebook results. 2 Benchmark Setting CTSCAN is a public multi-source chest CT segmentation benchmark implemented as a slice-derived training and evaluation stack. Each volumetric case is exported to aligned grayscale PNG slices and semantic masks, making the benchmark easy to reproduce but also easy to split incorrectly if patient identity is ignored. This paper studies the four-class public release used for the main benchmark: background, ground glass, consolidation, and pleural effusion. The same project also contains a broader eight-class pipeline, but the scientific claim in this paper is about evaluation protocol rather than ontology breadth. We therefore hold the training recipe fixed and compare two protocols: • Slice-mixed : train, validation, and test are formed directly over individual PNG pairs. • Patient-disjoint : slices are grouped by case before partitioning, and overlap is explicitly audited. The key question is which performance survives once the split is made patient-disjoint. 3 Benchmark Dataset and Protocols 3.1 Data composition The current four-class PNG benchmark contains 89 cases drawn from three public sources: 78 PleThora cases, 10 MedSeg SIRM cases, and 1 LongCIU case. The benchmark contains 10423 slices. Its label distribution is extremely imbalanced, as shown in Table 1 . Table 1: Pixel-frequency imbalance in the four-class PNG benchmark. Class Pixel frequency Background 0.996341 Class 1 0.001373 Class 2 0.000355 Class 3 0.001931 Foreground is therefore sparse, and the pleural-effusion-heavy class dominates among positive labels. 3.2 Best slice-mixed reference The strongest single-run slice-mixed reference on the four-class benchmark uses an FPN [ 3 ] with an EfficientNet-B0 encoder [ 5 ] , Lovász-Softmax plus cross-entropy loss [ 1 ] , and a rare-foreground sampler. Under the slice-mixed protocol, this configuration reaches 0.6811 foreground Dice and 0.5179 foreground IoU. At face value, these look like strong benchmark results. 3.3 Protocol comparison The reference training notebook operates on per-slice PNG pairs rather than case identifiers. In the notebook, all converted slice pairs are first pooled into a single list and train_test_split is then applied directly to that slice list. The resulting protocol is therefore slice-mixed rather than patient-disjoint. Collapsing slice stems back to case identifiers reveals near-complete case reuse across train, validation, and test, as shown in Table 2 . Table 2: Case reuse implied by the slice-mixed protocol. Partition summary Count Train cases 89 Validation cases 88 Test cases 89 Train/Test overlap 89 Train/Validation overlap 88 Validation/Test overlap 88 We therefore use a deterministic case-level protocol as the primary evaluation setting for patient-disjoint generalization. 4 Benchmark Release CTSCAN packages the patient-disjoint benchmark as a reproducible public release rather than as an undocumented split change. 4.1 Deterministic patient-disjoint split We added a scripted split builder that rebuilds a case-level split from the PNG slices plus benchmark metadata. The script groups slices by case ID, preserves source-aware allocation, writes CSV splits understood by the current trainer, and emits a machine-readable manifest that reports case counts, slice counts, per-source composition, and explicit case-overlap checks. The scripted split contains: • train: 70 cases, 8351 slices • validation: 9 cases, 1021 slices • test: 10 cases, 1051 slices All pairwise case overlaps are zero. 4.2 Explicit supervision controls The broader dataset builder supports pseudo-labeled sample studies, but the benchmark release disables them by default. Any weak-supervision run therefore requires an explicit command-line opt-in rather than being silently mixed into the public benchmark. 4.3 Reproducible study presets The benchmark release includes named presets for the shared-control and tuned reference settings. This removes the need to reconstruct long hyperparameter command lines by hand when rerunning the study. 4.4 Scripted study and public benchmark page We added a reproducible study driver that launches matched training runs under the slice-mixed and case-disjoint protocols, caches per-seed metrics, and aggregates both final test scores and per-epoch learning curves into a single machine-readable summary. The same artifacts feed a generated benchmark page