Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study
Jiuzhou Han Paul Burgess Ehsan Shareghi · 2024
Read Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study by Jiuzhou Han Paul Burgess Ehsan Shareghi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06272.txt.
Introduction
Jiuzhou Han, Paul Burgess, and Ehsan Shareghi wrote this paper, published in two thousand twenty-four, on predicting which laws and past cases a legal text will cite. The task is hard: it takes fine-grained reading and exact picks from a huge citation pool.
The team built the AusLaw Citation Benchmark, fifty-five thousand Australian legal cases tied to more than eighteen thousand distinct citations. They then raced many approaches: plain prompting of general and law-tuned language models, retrieval-only setups, fine-tuning, and hybrids that mix models with search. Plain models scored near zero on their own. Fine-tuning did far better, and a hybrid with a trained re-ranker did best of all. Even so, a gap of nearly half remains. The authors offer the benchmark as a stern test for legal AI.
Catalog record
| CREATOR | Jiuzhou Han Paul Burgess Ehsan Shareghi |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06272
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study Jiuzhou Han Paul Burgess Ehsan Shareghi Abstract Large Language Models (LLMs) have demonstrated strong potential across legal tasks, yet the problem of legal citation prediction remains under-explored. At its core, this task demands fine-grained contextual understanding and precise identification of relevant legislation or precedent. We introduce the AusLaw Citation Benchmark , a real-world dataset comprising 55 k 55k Australian legal instances and 18,677 18,677 unique citations which to the best of our knowledge is the first of its scale and scope. We then conduct a systematic benchmarking across a range of solutions: (i) standard prompting of both general and law-specialised LLMs, (ii) retrieval-only pipelines with both generic and domain-specific embeddings, (iii) supervised fine-tuning, and (iv) several hybrid strategies that combine LLMs with retrieval augmentation through query expansion, voting ensembles, or re-ranking. Results show that neither general nor law-specific LLMs suffice as stand-alone solutions, with performance near zero. Instruction tuning (of even a generic open-source LLM) on task-specific dataset is among the best performing solutions. We highlight that database granularity along with the type of embeddings play a critical role in retrieval-based approaches, with hybrid methods which utilise a trained re-ranker delivering the best results. Despite this, a performance gap of nearly 50 % 50\% remains, underscoring the value of this challenging benchmark as a rigorous test-bed for future research in legal-domain. 1 1 1 The data, code, and trained LLMs and re-rankers are available at: https://auslawbench.github.io 1 Introduction Recent advancements in utilising Large Language Models (LLMs) for legal domain have shown promising results across various tasks. For instance, Pont et al. (2023) leveraged LLMs for generating summaries of judicial decisions, identifying legal issues, decision-making criteria, and specifying keywords. Deroy et al. (2024) revealed that LLMs outperform extractive summarisation methods in quality metrics but suffer from inconsistencies and hallucinations, highlighting the importance of human-in-the-loop approaches for improved reliability. Jiang and Yang (2023) underscored that fine-tuning LLMs demonstrates state-of-the-art performance in legal judgment prediction, while Peng and Chen (2024) showed retrieval-augmentation leads to improved accuracy by integrating external knowledge, particularly for complex charges. Hou et al. (2024a) proposed a model to detect deviations between an AI-generated legal analysis and human as a way of quantifying their reliabilities. While these advancements highlight LLMs’ transformative potential in legal applications, challenges like ensuring factual accuracy, handling diverse tasks, and mitigating inherent issues such as hallucination remain. For instance, Dahl et al. (2024) reports that even state-of-the-art LLMs hallucinate between 69-88% of responses to legal queries, while Magesh et al. (2024) highlights that hallucination issue is mitigated with specialisation of tools but still persists as an unresolved issue. In this paper, we report progress with respect to a less-explored task in the legal domain, Legal Citation Prediction . Citations in legal cases, as in academic writing, serve two purposes: first, information necessary to locate, read, and verify the material; and second, information about the authority of the source is conveyed ( Axel-Lute, 1982 ) . The second is particularly crucial in legal cases in common law jurisdictions, as in those systems most decision makers are required to follow previous decisions ( Schauer, 1987 ) . Accordingly, whilst precedent plays a fundamental role in determining how courts behave and, therefore, in how societies function, citations to, of, and between authorities are the way in which precedent is communicated. In determining matters in court, judges use citations to give weight and authority to their decisions and also, crucially, to demonstrate that they are acting appropriately and are adhering to the legal precedent that already exists. When doing so, a judge or a panel of judges will often state a legal rule – as a principal or a proposition – and then support the existence of that rule with a citation to another source; frequently a prior court decision. In other words, judges making determinations today rely on citations to not only determine legal questions in court but also to show that they are acting within their (the judges’) lawful role. Formally, the citation prediction task can be defined as follows: Given a passage, the goal is to identify the correct legislation or precedent that applies and needs to be cited (i.e., to predict or retrieve the correct [CITATION] ). The following examples illustrate the citation prediction tasks considered in this paper: Example 1. Query : The distinction between a genuine offer of compromise and a demand to capitulate has to be recognised. See the discussion in [CITATION]. Answer : Leichhardt Municipal Council v Green [2004] NSWCA 341 Example 2. Query : The Tribunal is satisfied that the applicant does not fulfil the requirements of section 139(a) of the National Law, in that she lacks the mental capacity to practise medicine, as was considered in [CITATION]. Answer : Lindsay v Health Care Complaints Commission [2010] NSWCA 194 Example 3. Query : Whilst it is suggested that the offender’s mother and grandmother have difficulty paying rent without the offender’s assistance, there is no evidence of how they support themselves or their financial circumstances. There is no evidence of hardship that might meet the ‘truly, wholly or highly exceptional’ standard referred to in [CITATION]. Answer : Jinnette v R [2012] NSWCCA 217 We aim to compare, develop, and explore different solutions for the citation prediction task. We release, AusLaw Citation Benchmark , a real dataset of 55k instances specific to Australian law, covering 18,677 unique citations. To the best of our knowledge this is the first dataset of this scale and scope. We then conduct a thorough comparison along the following comprehensive dimensions: • Prompting general purpose LLMs (i.e., GPT-4o ( OpenAI, 2024 ; Achiam et al., 2023 ) , Claude Sonnet 3.5 ( Anthropic, 2024 ) , LLaMA-3.1-70B-instruct ( Grattafiori et al., 2024 ) , Command R+ ( Cohere, 2024 ) ) • Prompting law-specialised pre-trained LLMs (i.e., SaulLM-7B-instruct ( Colombo et al., 2024b ) , SaulLM-54B-instruct ( Colombo et al., 2024a ) ) • Retrieval-only setup with vectorised DB using general-purpose (i.e., text-embedding-3-large) and law-specific embeddings (i.e., AusLaw-embedding-v1.0) • Instruction fine-tuning LLMs (i.e., SaulLM-7B, and LLaMA-3.1-8B) for the citation prediction task • Different hybrid tactics that combine LLMs and retrieval systems (i.e., retrieval-augmented generation, query expansion, voting ensemble, and specialised re-rankers) While we aim to push all these solutions to their limits to better understand their effectiveness and limitations, a near 50\% gap is remained to be filled. We hope this benchmark and the baselines and methods investigated in this paper to encourage future developments in this important and challenging task. 2 Legal NLP in the Era of LLMs While a satisfying review demands a separate work, we attempt to provide a brief overview of existing work on the intersection of Law and LLMs. We also acknowledge that this only provides an overview of a small subset of the broader space of research in Legal NLP. The intersection of NLP and the legal domain has given rise to a wide array of research tackling tasks such as legal text classification, retrieval, summarisation, question answering, and reasoning. As legal texts are often complex and domain-specific, specialised benchmarks and models