s2n-bignum-bench: A practical benchmark for evaluating low-level code reasoning of LLMs
Balaji Rao, John Harrison, Soonho Kong, Juneyoung Lee, Carlo Lipizzi · 2026
Read s2n-bignum-bench: A practical benchmark for evaluating low-level code reasoning of LLMs by Balaji Rao, John Harrison, Soonho Kong, Juneyoung Lee, Carlo Lipizzi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14628.txt.
Description
Rao and co-authors benchmark language models on proofs about real cryptographic code. Math contest wins do not prove skill with production software. Their s2n-bignum-bench draws tasks from an AWS assembly library. Human experts already verified that library in HOL Light. Models get the formal spec and must write a proof script. HOL Light must accept the script inside a fixed time limit. The bench is the first public test of machine-checked proofs for industrial low-level crypto code, they state.
Catalog record
| CREATOR | Balaji Rao, John Harrison, Soonho Kong, Juneyoung Lee, Carlo Lipizzi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24829 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.14628