When No Benchmark Exists: Validating Comparative LLM Safety Scoring Without Ground-Truth Labels

Sushant Gautam, Finn Schwall, Annika Willoch Olstad, Fernando Vallecillos Ruiz Birk Torpmann-Hagen, Sunniva Maria Stordal Bjørklund, Leon Moonen, Klas Pettersen, Michael A. Riegler, [0.5em] Simula Metropolitan Center for Digital Engineering, Oslo Metropolitan University · 2026

Read When No Benchmark Exists: Validating Comparative LLM Safety Scoring Without Ground-Truth Labels by Sushant Gautam, Finn Schwall, Annika Willoch Olstad, Fernando Vallecillos Ruiz Birk Torpmann-Hagen, Sunniva Maria Stordal Bjørklund, Leon Moonen, Klas Pettersen, Michael A. Riegler, [0.5em] Simula Metropolitan Center for Digital Engineering, Oslo Metropolitan University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06652.txt.

Description

Sushant Gautam and eight co-authors tackle comparing language models for safety when no labeled benchmark exists for the language or sector at hand. They formalize benchmarkless comparative safety scoring and the fixed conditions, from scenario pack to rerun budget, under which scores count as deployment evidence. Validity rests on an instrumental chain rather than agreement with labels.

Their SimpleAudit tool checks that scores respond to a controlled safe-versus-weakened contrast, that model identity dominates scoring noise, and that results stabilize across reruns, shown on a Norwegian safety pack. A public-sector procurement case then shows the safer model can depend on scenario category, so scores must be reported with their full context.

Catalog record

CREATORSushant Gautam, Finn Schwall, Annika Willoch Olstad, Fernando Vallecillos Ruiz Birk Torpmann-Hagen, Sunniva Maria Stordal Bjørklund, Leon Moonen, Klas Pettersen, Michael A. Riegler, [0.5em] Simula Metropolitan Center for Digital Engineering, Oslo Metropolitan University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06652

Browse the full library · About the Tower of Babel