DistilledGemma: Balanced Efficiency-Accuracy for Person-Place Relation Extraction from Multilingual Historical Articles
Youssef Aboelwafa, Ahmed Samir, Nagwa Elmakky, Marwan Torki · 2026
Read DistilledGemma: Balanced Efficiency-Accuracy for Person-Place Relation Extraction from Multilingual Historical Articles by Youssef Aboelwafa, Ahmed Samir, Nagwa Elmakky, Marwan Torki (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29130.txt.
Introduction
Aboelwafa, Samir, Elmakky and Torki present DistilledGemma, a system that extracts person-place relations from multilingual historical newspaper articles for the HIPE-2026 shared task. The articles span English, German and French, and the task is to decide which people connect to which places. Their design balances accuracy against computing cost through knowledge distillation, training a small model to imitate a large one.
The pipeline runs in three stages. First, prompt strategies were explored across eight large models to find the best reasoning setup. Second, a large Gemma teacher was fine-tuned efficiently to produce silver-standard reasoning traces. Third, those patterns were distilled into a compact Gemma student, with rule-based post-processing enforcing type and direction consistency. In official evaluation the team ranked third on the standard test and second on the binary test.
Catalog record
| CREATOR | Youssef Aboelwafa, Ahmed Samir, Nagwa Elmakky, Marwan Torki |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29130
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DistilledGemma: Balanced Efficiency-Accuracy for Person-Place Relation Extraction from Multilingual Historical Articles Youssef Aboelwafa, Ahmed Samir, Nagwa Elmakky, Marwan Torki Abstract We present DistilledGemma , an efficient and accurate system for the HIPE-2026 shared task on person-place relation extraction from multilingual historical newspaper articles in English, German, and French. Our approach adopts a three-stage knowledge distillation pipeline designed to balance classification accuracy with computational efficiency. In the first stage, we systematically explored prompt engineering strategies across eight large language models to identify the most effective reasoning architecture for this challenging task. In the second stage, we applied supervised fine-tuning (SFT) via QLoRA to a Gemma 4 26B A4B teacher model, leveraging its strong multilingual capabilities to generate silver-standard chain-of-thought traces across the training corpus. In the final stage, we performed response-level distillation to transfer these learned reasoning patterns into a compact Gemma 4 E2B student model. To address the class imbalance between positive and negative relation pairs and to enforce semantic consistency, we incorporated rule-based post-processing on all generated outputs. Specifically, this step enforced entity-type compatibility, relation-direction constraints, contextual trigger patterns, and the logical constraint. In the official evaluation, our team WHEREAMI ranked 3rd on the standard test set with an accuracy profile mean score of 0.688 , and 2nd on the binary test set with a mean score of 0.8156 . Notably, by distilling knowledge from the 26B teacher to the 2.3B student, we preserved strong reasoning capabilities while reducing the deployed model size to approximately 2.3B effective parameters ; the LoRA adapters used during training were merged into the student for inference. This configuration ranked 2nd in the balanced efficiency-accuracy profile across both the standard and binary test sets. These results demonstrate that knowledge distillation provides a practical and scalable solution for historical document processing, achieving competitive performance without excessive computational cost. [[email protected]] 1 [[email protected]] 1 [[email protected]] [[email protected]] † † venue: The Conference and Labs of the Evaluation Forum (CLEF) 2026 † † address: Alexandria University, Egypt † † Equal contribution. 1 Introduction Historical newspapers constitute invaluable resources for studying past societies, migration patterns, and geopolitical events. A critical step in unlocking this information is person-place relation extraction : determining whether a person mentioned in an article bears a geographic connection to a place also mentioned therein. The HIPE-2026 shared task 17 ; 19 ; 18 formalizes this problem as a classification task over (person, place) pairs extracted from multilingual newspaper articles in English, German, and French. Each pair must be labelled for two relations: at (whether the person has ever been associated with the place; TRUE / PROBABLE / FALSE ) and isAt (whether the person is at the place within the immediate temporal horizon of the article ; TRUE / FALSE ) as shown in Figure 1 and 2 . Figure 1: Illustration of the distinction between at and isAt relations. Figure 2: Illustration of the distinction between at and isAt relations in HIPE-2026. The at relation captures historical or general associations, while isAt denotes the person’s location at the time described in the article. This task presents several challenges: • Noisy OCR text : Historical documents contain frequent optical character recognition errors that degrade entity mention quality. • Class imbalance : The FALSE label dominates both relations ( {\sim} 55% for at , {\sim} 78% for isAt ), biasing models toward negative predictions. • Multilingual reasoning : Models must perform consistently across three typologically diverse languages with varying training data availability. • Implicit evidence : Many person-place connections are implied rather than explicitly stated, requiring world knowledge and contextual inference. Large language models (LLMs) have demonstrated impressive zero-shot and few-shot capabilities for relation extraction 27 ; 28 , but models with sufficient capacity (e.g., 26B+ parameters) are impractical for deployment in digital humanities pipelines that must process millions of documents. This motivates our central research question: Can we distill the relation extraction capabilities of a large, fine-tuned LLM into a model small enough for practical deployment without catastrophic performance loss? Our contributions are as follows: 1. A comprehensive evaluation of eight LLM configurations spanning prompt engineering, zero-shot inference, and supervised fine-tuning for person-place relation extraction across three languages. 2. A three-stage distillation pipeline that fine-tunes a Gemma 4 26B A4B teacher via QLoRA, generates silver-standard training data with chain-of-thought (CoT) reasoning, and trains a Gemma 4 E2B student on the teacher’s outputs. 3. Empirical evidence that knowledge distillation recovers {\sim} 88% of teacher performance, establishing a practical efficiency-accuracy trade-off for historical NLP applications. 2 Related Work Encoder-based relation extraction. Before the recent shift to generative LLMs, relation extraction was commonly framed as discriminative classification over marked entity pairs, using either hand-engineered features 34 or contextual encoders 20 . These lightweight baselines remain important for low-resource and imbalanced settings because they condition directly on entity spans and require only a small task head. Entity-aware representation learning, including task-agnostic relation representations from entity-linked text 2 , typed entity-marker models 36 , and PL-Marker span-pair packing 33 , provides strong baselines for sentence-level RE. Cross-lingual encoders such as XLM-R 4 and entity-aware multilingual encoders such as mLUKE 23 are especially relevant for multilingual HIPE-style data. For longer article-level evidence, document-level RE benchmarks such as DocRED highlight the need to synthesize evidence across sentences and entity mentions 32 . Recent multilingual and digital-humanities RE work has also used guided distant supervision to build German biographical relation data and evaluate multilingual transfer 21 . Low-resource RE benchmarks show that class balancing, data augmentation, and self-training are useful alternatives, but their gains can be inconsistent under long-tailed label distributions 30 . These approaches are therefore strong lightweight baselines and complementary alternatives to our LLM distillation pipeline. Relation extraction with LLMs. Recent work has demonstrated that LLMs can perform competitive relation extraction through in-context learning 28 and chain-of-thought prompting 29 , particularly when explicit reasoning steps are elicited before label prediction. Wadhwa et al. 27 showed that GPT-class models match or exceed supervised baselines on standard benchmarks, albeit at significantly higher computational cost. Historical NLP. The HIPE shared task series 6 ; 7 has driven substantial progress in named entity recognition and linking for historical texts, establishing standardized benchmarks across multiple languages and time periods. These tasks highlight unique challenges including OCR noise, archaic language usage, and domain-specific entity types that are absent from modern NLP benchmarks. The HIPE-2026 edition 19 extends the series to person-place relation extraction, introducing new evaluation profiles that jointly consider accuracy and computational efficiency. Knowledge distillation and teacher alignment. Hinton et al. 11 introduced knowledge distillation (KD) as a model compression technique in which a compact student network learns to mimic a larger teacher’s output distribution. For generative models, sequence-level KD trains students on complete teacher outputs rather than only token-level distributions 14 . This paradigm has been widely adopted for language models: DistilBERT 24 compresses BERT while retaining 97% of its performance, and MiniLLM 10 extends KD to autoregressive LLMs with a policy-level reverse KL objective. On-policy approaches such as GKD 1 further improve upon standard KD by training on the student’s own generations and allowing alternative KL divergences. Closer to our setting, rationale- and explanation-trace distillation methods show that small models can benefit from LLM-generated reasoning