On the Role of Calibration in Algorithmic Fairness Benchmarking for Skin Cancer Detection DOI: 10.59275/j.melba.2025-ae66 Volume: 3
\firstname Brandon \surname Dominique \aff 1 \orcid 0009-0002-5163-6538, \firstname Prudence \surname Lam \aff 1 \orcid 0009-0000-9753-8234, \firstname Nicholas \surname Kurtansky \aff 2 \orcid 0000-0002-6745-0386, \firstname Jochen \surname Weber \aff 2 \orcid 000-0002-2397-4954, \firstname Kivanc · 2025
Read On the Role of Calibration in Algorithmic Fairness Benchmarking for Skin Cancer Detection DOI: 10.59275/j.melba.2025-ae66 Volume: 3 by \firstname Brandon \surname Dominique \aff 1 \orcid 0009-0002-5163-6538, \firstname Prudence \surname Lam \aff 1 \orcid 0009-0000-9753-8234, \firstname Nicholas \surname Kurtansky \aff 2 \orcid 0000-0002-6745-0386, \firstname Jochen \surname Weber \aff 2 \orcid 000-0002-2397-4954, \firstname Kivanc (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07700.txt.
Description
Brandon Dominique, Prudence Lam, and Nicholas Kurtansky argue that fairness audits of skin cancer AI miss half the story. Models can match expert melanoma detection overall while scoring demographic groups unevenly. Most audits use ranking accuracy, which says nothing about whether risk estimates themselves are trustworthy.
Calibration, whether a stated risk matches real outcomes, enters the benchmark beside ranking metrics. A model can rank fairly yet quote biased probabilities to one group, misleading triage. Adding calibration exposes such gaps before clinic use. The proposal pushes medical AI evaluation toward the numbers doctors actually act on, not just leaderboards.
Catalog record
| CREATOR | \firstname Brandon \surname Dominique \aff 1 \orcid 0009-0002-5163-6538, \firstname Prudence \surname Lam \aff 1 \orcid 0009-0000-9753-8234, \firstname Nicholas \surname Kurtansky \aff 2 \orcid 0000-0002-6745-0386, \firstname Jochen \surname Weber \aff 2 \orcid 000-0002-2397-4954, \firstname Kivanc |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07700