Automatic Legal Writing Evaluation of LLMs
Ramon Pires, Roseval Malaquias Junior, Rodrigo Nogueira · 2025
Read Automatic Legal Writing Evaluation of LLMs by Ramon Pires, Roseval Malaquias Junior, Rodrigo Nogueira (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21202.txt.
Description
Ramon Pires, Roseval Malaquias Junior, and Rodrigo Nogueira build in 2025 a benchmark that grades legal writing by language models using a real bar exam. Legal writing is open-ended and hard to score, so public tests are scarce. The Brazilian Bar Examination offers fresh questions, rubrics, and source texts.
Their set, oab-bench, holds 105 questions across seven law areas from recent exams. Four models take the test, and the best scores 7.93 of 10, passing all 21 exams. The team also asks whether models can grade as human examiners do. Top models track human scores closely on passed exams. That link backs AI help in grading, with care for the subjective core of legal judgment.
Catalog record
| CREATOR | Ramon Pires, Roseval Malaquias Junior, Rodrigo Nogueira |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-21202