Building Data-Driven Occupation Taxonomies: A Bottom-Up Multi-Stage Approach via Semantic Clustering and Multi-Agent Collaboration
Nan Li, Bo Kang, Tijl De Bie · 2025
Read Building Data-Driven Occupation Taxonomies: A Bottom-Up Multi-Stage Approach via Semantic Clustering and Multi-Agent Collaboration by Nan Li, Bo Kang, Tijl De Bie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15786.txt.
Introduction
Nan Li, Bo Kang, and Tijl De Bie build job taxonomies straight from job postings. A taxonomy of occupations guides job matching and labor market study. Hand made versions are slow. Top down auto versions miss local markets, while bottom up ones can tangle on noisy data.
Their builder, CLIMB, starts with global meaning based clustering to distill core jobs. A team of software agents then reflects and revises, step by step, to grow a coherent tree. On three real data sets from varied places, the trees are more coherent and scale better than past methods. They also capture traits unique to each region. Code and data are released.
Catalog record
| CREATOR | Nan Li, Bo Kang, Tijl De Bie |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-15786
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Building Data-Driven Occupation Taxonomies: A Bottom-Up Multi-Stage Approach via Semantic Clustering and Multi-Agent Collaboration Nan Li, Bo Kang, Tijl De Bie Abstract Creating robust occupation taxonomies, vital for applications ranging from job recommendation to labor market intelligence, is challenging. Manual curation is slow, while existing automated methods are either not adaptive to dynamic regional markets (top-down) or struggle to build coherent hierarchies from noisy data (bottom-up). We introduce CLIMB (CLusterIng-based Multi-agent taxonomy Builder), a framework that fully automates the creation of high-quality, data-driven taxonomies from raw job postings. CLIMB uses global semantic clustering to distill core occupations, then employs a reflection-based multi-agent system to iteratively build a coherent hierarchy. On three diverse, real-world datasets, we show that CLIMB produces taxonomies that are more coherent and scalable than existing methods and successfully capture unique regional characteristics. We release our code and datasets at https://anonymous.4open.science/r/CLIMB . 1 Introduction A taxonomy is a hierarchical structure that organizes information. In the labor market, a robust taxonomy is critical for organizing job postings, guiding job seekers, and informing policy. It improves search and guidance for job seekers, provides governments a framework to analyze labor trends and inform policy, and underpins corporate workforce planning and skill gap analysis. However, because labor markets are dynamic and regionally diverse, there is a strong need for taxonomies that are tailored to specific markets and easily updatable. The goal of this work is to fully automate the construction of such data-driven, hierarchical taxonomies directly from a raw corpus of job postings. Existing methods struggle to meet this need. Manual curation is slow, expensive, and unscalable. As detailed in Section 2 , automated approaches are either top-down , expanding an existing structure, or bottom-up , building a new one from data. Many traditional and recent LLM-based methods follow a top-down approach. These systems are designed to expand upon a pre-existing seed , typically a small, expert-curated taxonomy or a list of initial terms extracted and filtered using classic natural language processing techniques. While effective for enriching existing knowledge structures, this reliance on a seed makes them ill-suited for building a taxonomy from scratch in a new domain and and fails to fully remove the need for expert knowledge and potential for human bias. In contrast, bottom-up approaches aim to build a hierarchy directly from a raw corpus but face fundamental challenges. First , they struggle to distill a globally consistent set of core concepts (e.g., distinct occupations) from the data; feeding an entire corpus to an LLM is often infeasible, while incremental processing sacrifices the global perspective of the whole corpus needed to identify concepts consistently. Second , even with a clean set of concepts, constructing a deep and logically coherent hierarchy is a complex reasoning task where single-LLM systems often fail, producing inconsistent groupings and flawed structures. Figure 1: High-level overview of CLIMB. To address these challenges, we propose CLIMB ( CL uster I ng-based M ulti-agent taxonomy B uilder), a multi-stage framework that solves both problems in sequence. To overcome the first challenge, CLIMB begins with the raw corpus and uses semantic clustering over the entire dataset to automatically distill a globally-informed set of leaf-node concepts. To solve the second, CLIMB employs a novel reflection-based multi-agent framework for the hierarchical construction. It proceeds level-by-level, using a “Generator” to propose parent concepts and an “Evaluator” to critique them for logical consistency. This iterative refinement is essential for building a complex yet coherent taxonomy. Our contributions are: • We propose a novel multi-stage methodology, CLIMB, that fully automates bottom-up taxonomy generation. Its key innovations are (1) a global clustering stage to distill concepts directly from a raw corpus and (2) a multi-agent, reflection-based framework to solve the complex reasoning task of hierarchical construction. This makes the process objective by removing the need for expert seeds or manual curation. • We demonstrate that CLIMB produces taxonomies that are not only coherent and scalable, but also highly adaptive. Unlike static, generic taxonomies, CLIMB’s data-driven approach creates structures that are customized to specific regional labor markets at a specific point in time, capturing unique local roles and emerging trends. • We release our code and the datasets used in this study to the public to encourage reproducible research and further innovation in this area ( https://anonymous.4open.science/r/CLIMB ). 2 Related Work The task of taxonomy generation is broadly divided into two main methods: top-down and bottom-up. Top-Down Approaches. A significant body of work, from both the pre-LLM era Shen et al. 2018 ; Zhang et al. 2018 ; Huang et al. 2020 ; Shang et al. 2020 ; Lee et al. 2022 ; Le et al. 2023 and the recent LLM era Zeng et al. 2024 ; Gunn et al. 2024 ; Marchenko and Dvoichenkov 2024 ; Kargupta et al. 2025 , operates top-down by expanding a pre-existing seed taxonomy. While effective for enriching established knowledge structures, this reliance on a seed makes them less suitable for creating a taxonomy from scratch in a new domain. Bottom-Up Approaches. In contrast, bottom-up approaches aim to construct a taxonomy directly from a corpus, aligning with CLIMB’s objective of creating data-driven, adaptive taxonomies. Recent works have explored several strategies to this end. Some domain-specific methods require a pre-compiled list of terms Sas and Capiluppi 2024 ; Moraes et al. 2024 , precluding full automation from raw text. To the best of our knowledge, the state-of-the-art method that operates in a fully automated, bottom-up, and seed-free manner is TnT-LLM Wan et al. 2024 , which we select as our primary baseline. TnT processes data in small batches to iteratively build its taxonomy. While scalable, this local view (i.e., seeing only the data in the current batch) can result in redundant concepts or a fragmented hierarchy, especially with noisy data. Another strategy uses a human-in-the-loop to refine an LLM-generated draft, sacrificing full automation for quality control Shah et al. 2023 . In contrast, CLIMB’s fully automated solution resolves the local-view limitation by performing global clustering on the corpus, ensuring its foundation of leaf nodes is comprehensive and consistent. Reflection for Complex Reasoning. Once base concepts are identified, constructing a logically sound hierarchy is a complex reasoning task where a single LLM struggles to ensure overall logical coherence. In other fields require such reasoning, such as code generation Madaan et al. 2023 ; Chen et al. 2023 or mathematical problem-solving Yao et al. 2023 , reflection-based multi-agent frameworks using a “generator” and “evaluator” for iterative refinement have proven highly effective Anthropic 2024 ; Kalyanpur et al. 2024 ; Yuan and Xie 2025 . To our knowledge, CLIMB is the first to apply this powerful paradigm to coherent and robust taxonomy construction. 3 Method Given a corpus of documents, our goal is to automatically construct a hierarchical taxonomy. As aforementioned, building a taxonomy from a raw corpus has two challenges: (1) distilling a consistent set of base concepts from a large and noisy corpus, and (2) constructing a coherent hierarchy from them. CLIMB’s multi-stage pipeline (Figure 1 ) is designed to address these. First , the pipeline generates leaf nodes from raw job postings. This involves optional text distillation and embedding (Section 3.1 ), global semantic clustering to identify occupations (Section 3.2