Who judges the judges? Governance from metrics: a runtime framework for continuous LLM compliance monitoring
Jehanne Dussert · 2026
Read Who judges the judges? Governance from metrics: a runtime framework for continuous LLM compliance monitoring by Jehanne Dussert (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24737.txt.
Description
Jehanne Dussert argues AI compliance should be a continuous runtime signal, not a binary audit-time verdict. The EU AI Act demands ongoing human oversight and drift detection in deployed systems. Her govllm framework routes model selection by compliance scores rather than latency or cost alone. A panel of regulatory judges, each specialized per criterion, turns disagreement into an uncertainty signal for arbitration. Tests on 49 pairs across five criteria show agreement topping out near 69 percent, with no model leading everywhere, and position bias costing up to 25 points.
Catalog record
| CREATOR | Jehanne Dussert |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 102996 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24737