PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology
Yimin Zhao, Sheela R. Damle, Simone E. Dekker, Scott Geng, Karly Williams Silva, Jesse J Hubbard, Manuel F Fernandez, Fatima Zelada-Arenas, Alejandra Alvarez, Brianne Flores, Alexis Rodriguez, Stephen Salerno, Carrie Wright, Zihao Wang, Pang Wei Koh ∗, Jeffrey T. Leek ∗ · 2026
Read PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology by Yimin Zhao, Sheela R. Damle, Simone E. Dekker, Scott Geng, Karly Williams Silva, Jesse J Hubbard, Manuel F Fernandez, Fatima Zelada-Arenas, Alejandra Alvarez, Brianne Flores, Alexis Rodriguez, Stephen Salerno, Carrie Wright, Zihao Wang, Pang Wei Koh ∗, Jeffrey T. Leek ∗ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01343.txt.
Description
Zhao and fifteen colleagues build PanCanBench, a benchmark testing language models in pancreatic cancer care beyond general medical knowledge. A human-in-the-loop workflow creates expert rubrics for patient questions stripped of identifiers from the Pancreatic Cancer Action Network, yielding 3,130 question-specific checks across 282 real patient questions. The team evaluates 22 closed and open-source models with an LLM-as-a-judge setup, measuring care coverage, factual truth, and web-search use. Models vary a lot in rubric-based coverage.
Catalog record
| CREATOR | Yimin Zhao, Sheela R. Damle, Simone E. Dekker, Scott Geng, Karly Williams Silva, Jesse J Hubbard, Manuel F Fernandez, Fatima Zelada-Arenas, Alejandra Alvarez, Brianne Flores, Alexis Rodriguez, Stephen Salerno, Carrie Wright, Zihao Wang, Pang Wei Koh ∗, Jeffrey T. Leek ∗ |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72424 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.01343