Mapping AI Benchmark Data to Quantitative Risk Estimates Through Expert Elicitation

Malcolm Murray, Henry Papadatos, Otter Quarks, Pierre-François Gimenez, Simeon Campos · 2025

Read Mapping AI Benchmark Data to Quantitative Risk Estimates Through Expert Elicitation by Malcolm Murray, Henry Papadatos, Otter Quarks, Pierre-François Gimenez, Simeon Campos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04299.txt.

Description

Malcolm Murray and co-authors try a way to turn AI test scores into risk estimates in a 2025 paper. Experts warn of many harms from large language models, yet direct measures of real harm are rare. Most work so far scores what models can do, and skill is only a sign of risk, not a measure of it.

In a pilot study, experts used results from Cybench, a benchmark of cyber tasks, to set chance estimates for harm scenarios. The authors describe the steps and judge the method promising for building such estimates. They also list ways to strengthen it. The aim is to link model skill to concrete harm in the world. That link could guide policy and safety work with firmer numbers than skill scores alone.

Catalog record

CREATORMalcolm Murray, Henry Papadatos, Otter Quarks, Pierre-François Gimenez, Simeon Campos
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-04299

Browse the full library · About the Tower of Babel