PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology

Yimin Zhao, Sheela R. Damle, Simone E. Dekker, Scott Geng, Karly Williams Silva, Jesse J Hubbard, Manuel F Fernandez, Fatima Zelada-Arenas, Alejandra Alvarez, Brianne Flores, Alexis Rodriguez, Stephen Salerno, Carrie Wright, Zihao Wang, Pang Wei Koh ∗, Jeffrey T. Leek ∗ · 2026

Read PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology by Yimin Zhao, Sheela R. Damle, Simone E. Dekker, Scott Geng, Karly Williams Silva, Jesse J Hubbard, Manuel F Fernandez, Fatima Zelada-Arenas, Alejandra Alvarez, Brianne Flores, Alexis Rodriguez, Stephen Salerno, Carrie Wright, Zihao Wang, Pang Wei Koh ∗, Jeffrey T. Leek ∗ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01343.txt.

Description

Zhao and fifteen colleagues build PanCanBench, a benchmark testing language models in pancreatic cancer care beyond general medical knowledge. A human-in-the-loop workflow creates expert rubrics for patient questions stripped of identifiers from the Pancreatic Cancer Action Network, yielding 3,130 question-specific checks across 282 real patient questions. The team evaluates 22 closed and open-source models with an LLM-as-a-judge setup, measuring care coverage, factual truth, and web-search use. Models vary a lot in rubric-based coverage.

Catalog record

CREATORYimin Zhao, Sheela R. Damle, Simone E. Dekker, Scott Geng, Karly Williams Silva, Jesse J Hubbard, Manuel F Fernandez, Fatima Zelada-Arenas, Alejandra Alvarez, Brianne Flores, Alexis Rodriguez, Stephen Salerno, Carrie Wright, Zihao Wang, Pang Wei Koh ∗, Jeffrey T. Leek ∗
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72424
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.01343

Browse the full library · About the Tower of Babel