Label Indeterminacy in AI & Law

Cor Steging, Tadeusz Zbiegień · 2025

Read Label Indeterminacy in AI & Law by Cor Steging, Tadeusz Zbiegień (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17463.txt.

Introduction

Court records look like solid ground truth for training legal prediction models, but Cor Steging and Tadeusz Zbiegień warn that the labels hide human choices. A recorded outcome may reflect a settlement, an appeal, or another step that could easily have gone the other way. They call that gap label indeterminacy: the same case could carry a different label if an intervention had or had not happened.

Methods exist to fill in such uncertain labels, yet all of them rest on assumptions that cannot be checked. Using cases from the European Court of Human Rights, Steging and Zbiegień show that how labels are built during training can strongly change model behaviour. They urge legal machine learning to treat label indeterminacy as a core concern, not a footnote.

Catalog record

CREATORCor Steging, Tadeusz Zbiegień
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-17463

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Label Indeterminacy in AI & Law Cor Steging, Tadeusz Zbiegień Abstract Machine learning is increasingly used in the legal domain, where it typically operates retrospectively by treating past case outcomes as ground truth. However, legal outcomes are often shaped by human interventions that are not captured in most machine learning approaches. A final decision may result from a settlement, an appeal, or other procedural actions. This creates label indeterminacy: the outcome could have been different if the intervention had or had not taken place. We argue that legal machine learning applications need to account for label indeterminacy. Methods exist that can impute these indeterminate labels, but they are all grounded in unverifiable assumptions. In the context of classifying cases from the European Court of Human Rights, we show that the way that labels are constructed during training can significantly affect model behaviour. We therefore position label indeterminacy as a relevant concern in AI & Law and demonstrate how it can shape model behaviour. keywords Label indeterminacy \sep machine learning \sep legal decision-making † † This manuscript has been accepted for presentation as a short paper at the 38th International Conference on Legal Knowledge and Information Systems (JURIX) in Turin, December 9 to 11 of 2025. 1 Introduction With recent advances in machine learning, artificial intelligence is increasingly deployed in law, yet many approaches inherently contain characteristics that are incompatible with legal tasks. Examples include the intrinsic retrospective nature of machine learning, and the inability to faithfully explain their reasoning [ 1 , 2 ] , and generative AI models suffer from issues such as hallucinations [ 3 ] . One crucial machine learning issue that has not yet been explored in the legal domain is label indeterminacy [ 4 ] , where the labels of certain training instances are inherently unknown because an intervention by a decision maker has affected the final outcome. An example comes from medicine, where doctors must decide whether to continue or withhold life-sustaining treatment. If treatment is withheld, the patient does not survive, and it is impossible to know whether they would have survived had treatment continued. The resulting data is therefore selectively labelled , as the observed outcomes are contingent on the decisions of the doctor [ 5 ] . Historical cases where treatment was withheld thus yield indeterminate labels, and any machine learning model trained on such data must take this indeterminacy into account. Research from other domains has shown that the way that we treat these indeterminate labels can have drastic effects on the behaviour and outcomes of AI models [ 4 ] . The legal domain also knows many scenarios in which the outcome has been affected by an intervention, and thus where label indeterminacy is at play. A famous example is that of granting bail: if an individual is not granted bail, we do not know how they would have acted if they were given bail [ 5 ] . If one were to train AI models on the historical cases, one would need to account for the pretrial detention cases, as those labels are indeterminate. If this is not accounted for, the model effectively learns to replicate past detention decisions rather than predicting the defendant’s likelihood of returning for a court appearance, thereby potentially perpetuating historical bias, as illustrated by concerns raised as in the COMPAS system [ 6 ] . Excluding all non-bail cases is likewise problematic, since they do not constitute a random sample of bail cases. Indeterminate labels can also arise in the case of settlements, where we do not know what the litigation outcome would have been if the case had been taken to court. Excluding settled cases from the training data can yield an unrepresentative model, as cases are not settled at random, and thus litigated cases alone may not be an accurate representative sample of all cases. Taking this one step further, in legal judgment prediction where cases could have been appealed but the judgment was accepted, we cannot know what the outcome would have been if the case had been appealed. The original judgment then may not reflect what an appellate court would have decided. Yet excluding original judgments from training data is problematic, since not all cases are or can be appealed based on the content. In this paper, we critically examine the legal task of court case predictions. We examine when and why decisions in non-appealed cases should be considered indeterminate rather than ground truth, and what methods exist that can be used to impute the labels of these indeterminate cases. In an experiment using court case data from the European Court of Human Rights (ECtHR), we show that the commonly used methods to handle label indeterminacy can lead to different prediction behaviours of the models that we train. 2 Background Machine learning approaches to automated legal prediction constitute an established subfield, with various models attempting to classify case outcomes across different legal domains [ 7 , 8 , 9 , 10 , 11 ] . Most of the previous research has used past case outcomes as fixed objective labels. This relies on an often implicit and rarely discussed modelling assumption that the final judgment represents an unambiguous resolution of the dispute and an objectively determined outcome. This assumption is particularly problematic in the context of label indeterminacy, where legal outcomes may depend on interventions such as appeals, settlements, or procedural decisions [ 12 , 13 ] . As the sub-field of automated legal prediction matures, core theoretical questions about what is meant by ‘legal prediction’ have emerged, along with considerations that must be addressed in practical applications [ 14 , 15 ] . In this context, the assumption that past case outcomes are always fixed and objective warrants revisiting, both from the computational and jurisprudential perspectives. To clarify the scope of this paper, it is first useful to distinguish label indeterminacy from selection bias , in which datasets capture only a non-representative slice of all disputes [ 16 ] . Both phenomena affect legal prediction, but in different ways. For example, excluding settled cases from a dataset introduces selection bias, as the remaining cases may not be representative of all disputes. In contrast, label indeterminacy arises when the outcome itself is unknowable: when settled cases are included in the dataset, the true court outcome of those cases cannot be determined. Consequently, a ‘final judgment’ in a dataset may therefore not reflect the whole story, depending on the qualities of a particular process. Distinct from these data-centric issues is the discussion on legal indeterminacy , a jurisprudential debate that should not be conflated with label indeterminacy. The indeterminacy debate developed from the realist critique of legal formalism [ 17 , 18 ] . It argues that legal reasoning often fails to yield a single inevitable answer because legal rules and principles are underdetermined [ 19 , 20 ] . Critical legal studies scholars have further emphasized that the law functions as a political practice with outcomes shaped by existing power dynamics rather than by mechanistic rule-following [ 20 , 21 ] . One of the most influential rebuttals to strong forms of legal indeterminacy comes from Dworkin [ 22 ]

Read the complete text