Temporal Decay of Co-Citation Predictability: A 20-Year Statute Retrieval Benchmark from 396M Ukrainian Court Citations
Volodymyr Ovcharov · 2026
Read Temporal Decay of Co-Citation Predictability: A 20-Year Statute Retrieval Benchmark from 396M Ukrainian Court Citations by Volodymyr Ovcharov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17639.txt.
Introduction
Volodymyr Ovcharov tests whether co-citation gives stable retrieval signal, building UA-StatuteRetrieval over 20 snapshots of Ukrainian court records. Using a leave-one-out protocol, Adamic-Adar rank falls 33 percent on a fixed set and 47 percent under a temporal split, confirming genuine decay, not a shifting mix. Decay is uneven: criminal procedure holds stable patterns, while civil law degrades around the 2017 reform. Hub items resist decay, but mid-frequency ones lose half their predictability. A BM25 baseline decays faster at 31 percent, and embedding drift reveals a semantic shift.
Catalog record
| CREATOR | Volodymyr Ovcharov |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21744 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.17639
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Temporal Decay of Co-Citation Predictability: A 20-Year Statute Retrieval Benchmark from 396M Ukrainian Court Citations Volodymyr Ovcharov Abstract Анотаця Co-citation structure is widely assumed to provide stable retrieval signal in legal information systems. We test this assumption longitudinally by constructing UA-StatuteRetrieval, a benchmark that measures co-citation predictability across 20 annual snapshots (2007–2026) of 396 million codex citations from 101 million Ukrainian court decisions. Using a leave-one-out protocol over the full bipartite citation graph, we find that Adamic-Adar MRR declines 33% on a fixed set of articles (from 0.43 to 0.29) and 47% under a train/test temporal split (from 0.51 to 0.27) – confirming genuine temporal decay rather than compositional shift or evaluation artifact. The decay is non-uniform: criminal procedure maintains stable co-citation patterns (MRR ∼ \sim 0.40), while civil law degrades from 0.35 to 0.15, coinciding with the 2017 judicial reform. Hub articles ( > > 100K citations) resist decay, but mid-frequency articles (1K–10K) – the practical retrieval frontier – lose half their predictability. A BM25 text baseline decays even faster (31%), and embedding drift analysis with E5-large reveals a 4.3% semantic shift in how articles are cited, providing a mechanistic explanation for the observed decay. The benchmark is released at https://huggingface.co/datasets/overthelex/ua-statute-retrieval . 1 Introduction Co-citation structure – the pattern of which statutory articles are cited together in court decisions – underpins many legal information retrieval systems [ 13 , 6 , 2 ] . The implicit assumption is that this structure provides a stable signal: if articles A and B were frequently co-cited in past decisions, they remain relevant to similar future cases. We test this assumption empirically. From the Ukrainian Unified State Register of Court Decisions (ДРСР, 101M decisions, 2007–2026), we extract 396 million codex-article citations and construct annual retrieval benchmarks spanning 20 years. Our evaluation protocol is leave-one-out citation prediction over the full bipartite citation graph: for each decision, we mask each cited article and measure whether co-citation-based methods can recover it from the remaining citations. While this is a proxy for statute retrieval rather than the end-user task itself (which starts from case facts, not partial citations), it directly measures the temporal stability of the co-citation signal that underpins many retrieval systems. The central finding is that co-citation predictability decays : the same retrieval methods applied to the same articles yield progressively worse results over time. This is not a trivial consequence of changing article composition – we verify through ablation that decay persists on a fixed article set (33% decline) and strengthens under proper train/test separation (47% decline). The decay concentrates along two dimensions: 1. Domain : criminal procedure resists decay while civil and administrative law degrade rapidly. 2. Frequency : hub articles ( > 100K citations) maintain predictability; mid-frequency articles (1K–10K) – the practical retrieval challenge – lose half their signal. Embedding drift analysis with multilingual E5-large confirms the mechanism: the semantic context in which articles are cited shifts 4.3% over 12 years, with civil procedure drifting fastest – directly mirroring MRR decay trajectories. These findings have direct implications for legal retrieval system design: graph-based methods should not be treated as static indexes but require continual updating, and complementary text-based methods are needed for the mid-frequency, civil/administrative tail where co-citation signal is weakest – though both BM25 and co-citation degrade over time. 2 Related Work Legal retrieval benchmarks. BSARD [ 8 ] provides 1,108 queries for Belgian statutory article retrieval. SCALE [ 12 ] and LeSICiN [ 11 ] evaluate retrieval in Swiss and Indian contexts respectively. LEXTREME [ 9 ] and LegalBench [ 5 ] cover classification but not retrieval. All evaluate at a single time point; none measure temporal stability. Legal citation networks. Fowler et al. [ 4 ] study US Supreme Court precedent networks; Coupette et al. [ 3 ] analyze European statute evolution; Winkels et al. [ 15 ] examine Dutch case law authority. CaseGNN++ [ 13 ] applies graph neural networks to legal retrieval. None release citation graphs as retrieval benchmarks or study temporal degradation. Link prediction. We adapt Common Neighbors and Adamic-Adar [ 1 ] from social network link prediction [ 7 ] to bipartite legal graphs. Our contribution is the longitudinal application and the discovery that predictability is non-stationary. 3 Data 3.1 Source Corpus The ДРСР publishes all Ukrainian court decisions since 2005. We process 101 million full-text decisions partitioned by adjudication year (2007–2026). Citation extraction uses compiled regex patterns for 13 codices with abbreviation normalization and article range expansion. The raw extraction yields 502 million citation records across all types (codex articles, case references, constitutional citations, law-by-number references). After restricting to codex-article citations and retaining only well-formed article references, we obtain 396 million validated records – the basis for our evaluation. 3.2 Annual Snapshots For each year y\in\{2007,\ldots,2026\} , we construct an evaluation snapshot: 1. Extract all codex citations from decisions adjudicated in year y . 2. Filter articles: minimum 50 citations, capped at 5,000 most frequent. 3. Filter cases: 3–200 cited articles per decision. The 2024 snapshot contains 3,671 articles, 1,801,481 cases, and 16.4M citation edges. 4 Method 4.1 Evaluation Protocol Let M\in\{0,1\}^{|U|\times|V|} be the binary case-article incidence matrix and C=M^{\top}M (with \text{diag}(C)=0 ) the article co-citation matrix. For each case u with cited articles S_{u} ( |S_{u}|\geq 3 ), and each target v_{t}\in S_{u} : score all candidates v\notin S_{u}\cup\{v_{t}\} and compute the rank of v_{t} . Since \text{diag}(C)=0 , the target’s self-contribution is naturally excluded. We sample 200,000 cases per year, yielding \sim 1.8M predictions per snapshot. 4.2 Scoring Functions Common Neighbors : \text{CN}(v_{t}\mid S)=\sum_{s\in S}C_{s,v_{t}} Adamic-Adar : \text{AA}(v_{t}\mid S)=\sum_{s\in S}C_{s,v_{t}}/\max(\log d_{s},1) Degree Baseline : \text{DG}(v_{t})=d_{v_{t}} (popularity only). 4.3 Ablation Controls To distinguish genuine temporal decay from confounds, we introduce two controls: Fixed-Article Ablation. We identify articles with \geq 50 citations in both 2012 (peak) and 2024 (recent), yielding a shared vocabulary. Evaluating only on this fixed set isolates temporal decay from compositional shift (new articles diluting performance). Temporal Train/Test Split. Within each year, we build C from the first 50% of cases (by document ID) and evaluate on the second 50%. This removes data leakage: the co-citation matrix never sees evaluation cases. 5 Results 5.1 Cross-Sectional Performance (2024) Табл. 1: Retrieval baselines on the 2024 snapshot (200K cases, 1.8M predictions, 3,671 articles). Metric Adamic-Adar Common Neighbors Degree Random Hit@1 0.145 0.141 0.030 < 0.001 Hit@5 0.406 0.398 0.063 0.001 Hit@10 0.545 0.534 0.111 0.003 Hit@20 0.675 0.664 0.175 0.005 MRR 0.272 0.266 0.059 < 0.001 Table 1 establishes the 2024 cross-section. Co-citation methods outperform degree by 4.6 \times on MRR, confirming that citation context provides retrieval signal beyond popularity. All differences are statistically significant: bootstrap 95% CIs on 1.8M predictions are \pm 0.0004 (AA: 0.272 \pm 0.0002, CN: 0.266 \pm 0.0002; paired difference z=108.9 ). These numbers represent the current state of predictability – subsequent sections show they were substantially higher in earlier years. 5.2 Difficulty Spectrum Табл. 2: Performance by article citation frequency (2024, Adamic-Adar). Bin Articles Predictions Hit@10 MRR Hub ( > 100K) 21 298,925 0.890 0.536 High (10K–100K) 354 1,127,489 0.601 0.274 Mid (1K–10K) 864 316,972 0.138 0.074 Low (100–1K) 1,689 68,807 0.041 0.020 Rare ( < 100) 739 5,338 0.012 0.010 0.0 0.2 0.4 0.6 0.8 1.0 100