UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoning
Volodymyr Ovcharov · 2026
Read UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoning by Volodymyr Ovcharov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29170.txt.
Description
Volodymyr Ovcharov introduces UA-Legal-Bench, a five-task benchmark for large language models on Ukrainian legal reasoning, built from nearly 100 million court rulings. Its tasks span case-type classification, judgment-form classification, outcome prediction, norm extraction, and cause prediction. Eleven models were tested under zero-shot and few-shot prompting, and few-shot prompting lifts judgment-form scores by about 39 points while barely moving outcome prediction. Accuracy misleads on imbalanced tasks: the highest-accuracy model is a majority-class predictor, while the genuinely best model scores only 44 percent.
Catalog record
| CREATOR | Volodymyr Ovcharov |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29872 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.29170