Taxonomy-Aligned Risk Extraction from 10-K Filings with Autonomous Improvement Using LLMs DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies
Rian Dolphin, Joe Dursun, Jarrett Blankenship, Katie Adams, Quinton Pike · 2026
Read Taxonomy-Aligned Risk Extraction from 10-K Filings with Autonomous Improvement Using LLMs DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies by Rian Dolphin, Joe Dursun, Jarrett Blankenship, Katie Adams, Quinton Pike (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15247.txt.
Introduction
Rian Dolphin, Joe Dursun, and co-authors pull structured risk data out of corporate filings. Their 2026 arXiv paper deals with 10-K reports, the yearly filings firms must make. Each filing lists risk factors in free text, which is hard to use at scale.
The authors build a three-stage pipeline for the task. A language model first extracts risks with supporting quotes. Embedding-based mapping then ties each risk to a fixed taxonomy. A second model, acting as judge, filters out false assignments. The pipeline also improves itself on its own over time.
Catalog record
| CREATOR | Rian Dolphin, Joe Dursun, Jarrett Blankenship, Katie Adams, Quinton Pike |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-15247
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Taxonomy-Aligned Risk Extraction from 10-K Filings with Autonomous Improvement Using LLMs DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Information extraction Rian Dolphin, Joe Dursun, Jarrett Blankenship, Katie Adams, Quinton Pike Abstract . We present a methodology for extracting structured risk factors from corporate 10-K filings while maintaining adherence to a predefined hierarchical taxonomy. Our three-stage pipeline combines LLM extraction with supporting quotes, embedding-based semantic mapping to taxonomy categories, and LLM-as-a-judge validation that filters spurious assignments. To evaluate our approach, we extract 10,688 risk factors from S&P 500 companies and examine risk profile similarity across industry clusters. Beyond extraction, we introduce autonomous taxonomy maintenance where an AI agent analyzes evaluation feedback to identify problematic categories, diagnose failure patterns, and propose refinements, achieving 104.7% improvement in embedding separation in a case study. External validation confirms the taxonomy captures economically meaningful structure: same-industry companies exhibit 63% higher risk profile similarity than cross-industry pairs (Cohen’s d = 1.06 d=1.06 , AUC 0.82, p < 0.001 p<0.001 ). The methodology generalizes to any domain requiring taxonomy-aligned extraction from unstructured text, with autonomous improvement enabling continuous quality maintenance and enhancement as systems process more documents. 1. Introduction Corporate risk disclosure in annual reports (10-K filings) provides insightful commentary on the uncertainties facing public companies. Item 1A of these filings, the “Risk Factors” section, contains detailed discussions of material risks written in natural language by company management and legal teams. For investors, analysts, and risk managers, systematically analyzing these disclosures across hundreds or thousands of companies could reveal sector-wide risk patterns, emerging threats, and company-specific vulnerabilities. However, the unstructured nature of these documents and their substantial length (often spanning dozens of pages per company) makes manual systematic analysis at scale impractical. A natural solution is to use large language models (LLMs) to extract structured risk information from these documents. Modern LLMs can indeed process 10-K risk sections and generate coherent lists of identified risks. However, this naïve approach creates a fundamental problem: without constraints, each LLM extraction produces idiosyncratic risk labels that may differ in terminology, granularity, and framing, even when describing the same underlying risk. One company might be tagged with “currency fluctuation risk” while another receives “foreign exchange exposure,” making systematic comparison impossible. The result is unstructured tags that defeat the purpose of automation—we trade one form of unstructured data for another. Producing actionable data requires a method that maps LLM-extracted risks to a predefined, curated taxonomy. This allows consistent risk categorization across thousands of filings while preserving the flexibility of LLMs to identify diverse risk expressions in natural language. The challenge lies in the mapping itself: how do we reliably connect free-form risk descriptions extracted by an LLM to fixed categories in a hierarchical taxonomy, while ensuring that only genuinely relevant mappings are retained? We propose a methodology that combines semantic embeddings with LLM-based validation to achieve robust taxonomy-aligned risk extraction. Our approach operates in three stages. First, we use an LLM with structured output to extract comprehensive risk factors and supporting quotes from the raw text. Second, we map these extracted risks to our predefined taxonomy using embedding-based similarity between the extracted quote and taxonomy category descriptions. Third, we employ an LLM-as-a-judge to validate each mapping, scoring how well the supporting quote actually matches the assigned taxonomy category and filtering out weak or spurious mappings. This validation stage serves dual purposes: it filters spurious mappings in production while generating systematic feedback for continuous taxonomy improvement. This multi-stage pipeline ensures that extracted risks adhere to a consistent taxonomy while maintaining high precision in their assignments. Our contributions are: A robust methodology for taxonomy-aligned risk extraction. We present a complete pipeline that combines LLM extraction, embedding-based similarity mapping, and LLM validation to reliably map free-form risk descriptions to predefined taxonomy categories. The embedding-based mapping provides computational efficiency and semantic grounding, while the LLM-as-a-judge validation prevents false positives that would arise from naïve nearest-neighbor assignment alone. A three-tier risk factors taxonomy for investment analysis. We introduce a hierarchical taxonomy specifically designed for risk analysis in the investment context, with primary, secondary, and tertiary categories that organize risks at appropriate granularity for investment decision-making. Unlike existing taxonomies such as the Cambridge Taxonomy of Business Risks ( 7 ) , which subdivides categories like extreme weather into overly specific types (“Tropical Windstorm” vs. “Temperature Windstorm”), our taxonomy balances comprehensiveness with practical utility for financial analysis. Autonomous taxonomy refinement workflow. We present a systematic method for continuous taxonomy improvement using LLM-as-a-judge evaluation scores as feedback signals. An AI agent autonomously identifies problematic taxonomy categories by analyzing patterns in low-quality mappings, diagnoses root causes through examination of evaluation reasoning, generates candidate description refinements, and validates improvements using embedding-based testing that directly mirrors production matching. This creates a continuous improvement loop where taxonomy quality increases as the system processes more documents, reducing the human expertise burden in taxonomy maintenance. A case study demonstrates 104.7% improvement in embedding separation for a pharmaceutical approval category through this autonomous workflow, transitioning taxonomy development from a static design problem to an iteratively improving system. Empirical validation on S&P 500 companies. We evaluate our approach on 2024 10-K filings for S&P 500 companies, extracting 10,688 validated risk factors mapped to our taxonomy. Industry clustering analysis demonstrates that companies in the same industry exhibit significantly more similar risk profiles than cross-industry pairs (63% relative improvement in similarity, Cohen’s d=1.06 , p<0.001 across four statistical tests). This clustering emerges despite the taxonomy mapping having no access to industry codes, validating that extracted categories capture genuine economic risk dimensions. Finer industry granularity strengthens the signal (AUC improves from 0.733 for broad sectors to 0.822 for narrow industry definitions), and sector-specific analysis reveals intuitive patterns such as 83% of banks tagged with interest rate risk versus 22% of all companies. The production system has been deployed at Massive.com, processing companies across five years of historical filings and serving results via API. The methodology we present is not specific to risk factors or 10-K filings. It provides a general framework for extracting structured information from unstructured text while maintaining adherence to predefined categorical systems—a problem that arises in many domains where systematic analysis requires consistent labeling but source documents use unconstrained natural language. 2. Related Work Financial document analysis and risk extraction. The analysis of corporate disclosures using computational methods has been extensively studied. Early work focused on sentiment analysis of financial documents and identifying risk-related language through dictionary-based approaches ( 16 ) . More recent work has applied deep learning to classification and prediction based on financial text ( 5 ; 19 ; 10 ) . Other work has explored using pre-trained language models fine-tuned on financial corpora ( 2 ; 23 ; 8 ) . However, existing work primarily focuses on document-level sentiment analysis and broad classification tasks rather than structured extraction of specific risk factors aligned with pre-defined taxonomies. Risk taxonomies for corporate analysis. Several taxonomies have been proposed for classifying business risks. The Cambridge Taxonomy of Business Risks ( 7 ) provides a comprehensive framework but is designed for general risk management rather than investment analysis, leading to