UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoning

Volodymyr Ovcharov · 2026

Read UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoning by Volodymyr Ovcharov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29170.txt.

Description

Volodymyr Ovcharov introduces UA-Legal-Bench, a five-task benchmark for large language models on Ukrainian legal reasoning, built from nearly 100 million court rulings. Its tasks span case-type classification, judgment-form classification, outcome prediction, norm extraction, and cause prediction. Eleven models were tested under zero-shot and few-shot prompting, and few-shot prompting lifts judgment-form scores by about 39 points while barely moving outcome prediction. Accuracy misleads on imbalanced tasks: the highest-accuracy model is a majority-class predictor, while the genuinely best model scores only 44 percent.

Catalog record

CREATORVolodymyr Ovcharov
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29872
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29170

Browse the full library · About the Tower of Babel