Comparing LLM and Fine-Tuned Model Performance on NVDRS Circumstance Extraction with Varying Prompt Complexity

Geoffrey Martin, Xuan Zhong Feng, Yifan Peng · 2026

Read Comparing LLM and Fine-Tuned Model Performance on NVDRS Circumstance Extraction with Varying Prompt Complexity by Geoffrey Martin, Xuan Zhong Feng, Yifan Peng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21845.txt.

Introduction

Geoffrey Martin, Xuan Zhong Feng, and Yifan Peng compare language models with fine-tuned models on extracting death-circumstance information from violent-death narratives. Many circumstances need semantic inference beyond keyword matching, so the authors build a Complexity Score algorithm predicting when detailed prompts with coding guidelines beat name-only prompts. Their hybrid picks a prompt strategy per circumstance. Tested on twenty-five complex circumstances from the National Violent Death Reporting System, language models win on low-prevalence circumstances with thin data. Fine-tuned models suit common ones.

Catalog record

CREATORGeoffrey Martin, Xuan Zhong Feng, Yifan Peng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29955
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21845

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Comparing LLM and Fine-Tuned Model Performance on NVDRS Circumstance Extraction with Varying Prompt Complexity Geoffrey Martin, Xuan Zhong Feng, Yifan Peng Abstract Suicide is a leading cause of death in the United States, and understanding the circumstances that precede it requires extracting structured information from death investigation narratives. Many of these circumstances require semantic inference beyond simple keyword matching. We develop a “Complexity Score” algorithm that analyzes coding manual structure to predict when detailed prompts with full coding guidelines improve over name-only prompts. We then construct a hybrid approach that selects prompt strategy per circumstance. We evaluate large language models (LLMs) against fine-tuned RoBERTa on 25 inferentially complex circumstances from the National Violent Death Reporting System (NVDRS). We found that LLMs substantially outperform on low-prevalence circumstances where training data is insufficient. We further demonstrate that our framework generalizes across frontier LLMs, with GPT-5.2, Gemini 2.5 Pro and Llama-3 70B showing consistent performance patterns. These findings support a hybrid architecture where LLMs handle rare, inferentially complex circumstances while fine-tuned models handle common ones. keywords Social Determinants of Health Suicide Prevention Large Language Models Natural Language Processing NVDRS 1 Introduction The National Violent Death Reporting System (NVDRS) is the only state-based surveillance system linking data from death certificates, coroner/medical examiner reports, and law enforcement reports for violent deaths in the United States [ 1 ] . Since 2003, trained abstractors have manually coded over 60 circumstance variables from free-text narratives (referred to as death investigation reports) for each case, creating a rich dataset for epidemiological research on suicide risk factors [ 16 ] . However, manual abstraction is time-intensive and subject to inter-rater variability [ 18 ] , and the CDC’s constrained budget threatens the sustainability of this coding effort [ 6 ] . Automated extraction of circumstances from death investigation reports could reduce abstractor burden while maintaining data quality. Recent work has increasingly applied natural language processing (NLP) to extract Social Determinants of Health (SDOH) from clinical text [ 12 , 11 , 2 ] . Ontological frameworks like SDoHO provide structured vocabularies for representing social factors [ 4 ] , while transformer-based models have achieved strong performance on SDOH extraction from electronic health records [ 3 , 8 , 14 ] . More recently, large language models have shown particular promise. For example, Guevara et al. [ 7 ] demonstrated that GPT-4 achieves strong performance on extracting SDOH categories such as housing and employment from clinical notes. Consoli et al. [ 3 ] demonstrated that LLMs capture nuanced social factors that rule-based systems miss. Further, Gabriel et al. [ 5 ] validated LLM-based classifiers for clinical social needs screening, and Keloth et al. [ 8 ] showed that LLMs can extract SDOH across multiple institutions with minimal adaptation. Collectively, these studies show that LLMs can extract social factors from clinical narratives, but focus on relatively straightforward SDOH categories where the target concept is explicitly stated (e.g., “patient is homeless” \rightarrow housing instability), leaving inferentially complex and low-prevalence circumstances underexplored. For NVDRS specifically, Wang et al. [ 17 ] fine-tuned BioBERT for NVDRS circumstance extraction, achieving strong performance on high-prevalence categories but poor results on rare circumstances and those requiring multi-step reasoning. Xu et al. [ 19 ] extended this work to analyze social factors across population groups. However, fine-tuned classifiers struggle with such circumstances because they learn surface patterns rather than the underlying inference rules [ 13 ] . Supervised models need sufficient examples to learn each category’s decision boundary, and many NVDRS circumstances have fewer than 1000 positive cases. Circumstances coded in NVDRS death investigation reports present a more challenging extraction task for NLP because many circumstances require semantic inference beyond surface pattern matching—the text describes a situation, but the abstractor must reason about what that situation implies under a specific context [ 15 ] . For example, the circumstance ‘Household Substance Abuse’ is coded only if three criteria are met: (1) the victim is a child under 18, (2) the substance use must involve someone other than the victim, and (3) that person resides in the same household. Therefore, a model seeing “alcohol bottles found in the home” must infer that for adult victims, this reflects the victim’s own use and does not meet the coding criteria of ‘Household Substance Abuse’. Similarly, “Caregiver Burden” requires role inference: a statement such as “wife had terminal cancer” implies the victim was providing care, not receiving it. To address these challenges, we hypothesize that LLMs, with their capacity for zero-shot reasoning [ 9 ] , outperform fine-tuned models on circumstances requiring deep semantic inference, particularly when training data is scarce. We propose a framework with three components: (1) we extract definitions, coding guidance and examples from the NVDRS coding manual to construct detailed prompts; (2) we develop a Complexity Score algorithm that analyzes the structure of coding manual examples to predict when detailed prompts are needed versus when simpler name-only prompts suffice; and (3) we construct a hybrid approach that uses the Complexity Score to select prompt strategy per circumstance, without requiring empirical tuning on each task. We evaluate our proposed framework to extract 25 inferentially complex circumstances on the NVDRS. Our hybrid approach achieves macro F1 of 0.893 compared to fine-tuned RoBERTa’s 0.800, with the largest gains on low-prevalence circumstances. We compare GPT-5.2, Gemini 2.5 Pro, and Llama-3 70B and observe consistent performance patterns across all three frontier LLMs, demonstrating that our framework is model-agnostic. Among circumstances predicted to require detailed guidance, complex prompts improve F1 by 9.5 percentage points over name-only prompts, while the Complexity Score algorithm achieves 87% accuracy on non-tie circumstances in predicting optimal prompt strategy. 2 Materials and Methods 2.1 Framework Overview Our framework uses the NVDRS coding manual to guide prompt construction and selection. For each circumstance, we first extract the definition, coding guidance, and positive/negative examples from the manual. We then compute a Complexity Score based on the linguistic structure of the negative examples, which identifies circumstances where the LLM’s default interpretation is likely to produce false positives without explicit guidance. If the Complexity Score exceeds a predefined threshold, we use a complex prompt containing full coding guidelines; otherwise, we use a simple prompt containing only the circumstance name. This hybrid approach automatically selects appropriate prompts in real time. 2.2 Dataset and Circumstance Selection We utilized restricted-access death investigation reports from the NVDRS [ 1 ] . Each report combines the coroner/medical examiner (CME) and law enforcement (LE) narratives for each suicide case. Our dataset comprises 191,696 narratives. To isolate circumstances that require semantic inference under NVDRS coding guidelines, we excluded those that can be extracted from keyword matching or strongly lexical cues (e.g., “Mental Health Problem” indicated by terms like ‘depression’ or ‘schizophrenia’), as well as crisis-timing variables requiring temporal reasoning beyond our current scope. Our final analysis focuses on 25 circumstances that require semantic inference. These circumstances were selected because they involve entity resolution (determining who is affected by a mentioned condition), role inference (understanding relationship directionality), or implicit reasoning (recognizing that described situations imply unstated circumstances or applying non-obvious exclusion criteria). 2.3 Prompting Strategy We developed two prompting approaches based on the NVDRS Coding Manual. We used an LLM (GPT-5.2 in

Read the complete text