Mapping AI Benchmark Data to Quantitative Risk Estimates Through Expert Elicitation
Malcolm Murray, Henry Papadatos, Otter Quarks, Pierre-François Gimenez, Simeon Campos · 2025
Read Mapping AI Benchmark Data to Quantitative Risk Estimates Through Expert Elicitation by Malcolm Murray, Henry Papadatos, Otter Quarks, Pierre-François Gimenez, Simeon Campos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04299.txt.
Description
Malcolm Murray and co-authors try a way to turn AI test scores into risk estimates in a 2025 paper. Experts warn of many harms from large language models, yet direct measures of real harm are rare. Most work so far scores what models can do, and skill is only a sign of risk, not a measure of it.
In a pilot study, experts used results from Cybench, a benchmark of cyber tasks, to set chance estimates for harm scenarios. The authors describe the steps and judge the method promising for building such estimates. They also list ways to strengthen it. The aim is to link model skill to concrete harm in the world. That link could guide policy and safety work with firmer numbers than skill scores alone.
Catalog record
| CREATOR | Malcolm Murray, Henry Papadatos, Otter Quarks, Pierre-François Gimenez, Simeon Campos |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-04299