S k i l l V e r s e : Assessing and Enhancing LLMs with Tree Evaluation

Yufei Tian, Jiao Sun, Nanyun Peng, Zizhao Zhang, [7pt] University of California, Los Angeles · 2025

Read S k i l l V e r s e : Assessing and Enhancing LLMs with Tree Evaluation by Yufei Tian, Jiao Sun, Nanyun Peng, Zizhao Zhang, [7pt] University of California, Los Angeles (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00319.txt.

Introduction

Yufei Tian and co-authors present SkillVerse, a tree based method for judging what large language models can and cannot do. Their 2025 paper notes that as models take on complex, many sided tasks, a single score hides too much. SkillVerse uses a language model as judge to critique answers, then sorts the critiques into a branching tree of skills.

From that tree, users can read skill at any level of detail, from broad traits down to narrow ones. The team shows two uses. A tree search picked better few-shot examples and lifted in-context learning by 25 percent. The tree also predicted new model weaknesses with a 55 percent success rate, well above the rate without it. The catalog title is spaced out oddly, so the head title governs.

Catalog record

CREATORYufei Tian, Jiao Sun, Nanyun Peng, Zizhao Zhang, [7pt] University of California, Los Angeles
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00319

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

S k i l l V e r s e : Assessing and Enhancing LLMs with Tree Evaluation Yufei Tian, Jiao Sun, Nanyun Peng, Zizhao Zhang, [7pt] University of California, Los Angeles Abstract As language models evolve to tackle complex and multifaceted tasks, their evaluation must adapt to capture this intricacy. A granular, skill-specific understanding of model capabilities can empower researchers to make informed model development plans. In this paper, we introduce SkillVerse , an unsupervised tree-structured diagnosis framework for understanding model proficiency in specific abilities. With LLM as a judge, SkillVerse first critiques the model responses, and then organizes them into a hierarchical structure termed dendrogram. Given proficiency at arbitrary levels of granularity, SkillVerse is flexible to produce insights of behaviors of modern large models. We also demonstrate its efficacy in two downstream tasks: 1) improving model in-context learning by 25% using a tree-search algorithm to select more informative few shots, and 2) accurately predicting new model weaknesses with a 55% success rate, 22% higher than the baseline. 1 Introduction In recent years, leaderboard and benchmark results such as ChatbotArena Chiang et al. 2024 and MMLU Hendrycks et al. 2020 have become the dominant practice for evaluating the potency of language models. While these results provide a high-level snapshot of a model’s rank, their limited interpretability makes it difficult to identify subtle behavioral traits and derive actionable insights Murahari et al. 2024 ; Moayeri et al. 2024 . The limited interpretability of the current evaluation paradigm makes it hard to compare the relative strengths and weaknesses of different models. For instance, does a higher-ranked model consistently outperform lower-ranked counterparts across the entire benchmark? Do comparable scores transfer to equivalent model performance on all subdomains? Addressing such questions typically requires manual inspections, which is both time-consuming and costly. These challenges highlight the need for automatic, granular analyses that provide valuable insights to enrich our understanding of model behavior, which paves the road for targeted model improvement of specific capabilities. Figure 1: Up: The input-output flow of SkillVerse . Skill-specific critiques are extracted, structured into a dendrogram, and sliced at varying granularities to reveal nested clusters of skills and model proficiency. Bottom: Versatile applications of SkillVerse , from selecting informative few-shot demonstrations to uncovering hidden model weaknesses. Figure 2: The overall framework of SkillVerse . A set of critiques on model responses is parsed into atomic judgments and organized using bottom-up clustering into a dendrogram, which is then unfolded at varying levels of granularity to form nested clusters, allowing for detailed analysis of model proficiency. Thanks to the hierarchical structure, this novel pipeline is highly flexible in interpreting model capabilities. Lately, LLM-based evaluations Li et al. 2023 ; Zheng et al. 2023 , where LLMs critique and judge model responses, have emerged as a scalable approach to approximate human preferences Wang et al. 2023 ; Yuan et al. 2024 . These methods enable detailed analysis with rich contextual feedback, forming the foundation of our diagnosis framework, SkillVerse . Orthogonal to developing more reliable auto-raters, this paper contributes to structuring contextual feedback to generate actionable insights for model model evaluation, comparison, debugging, and improvement. As shown in Figure 1 and Figure 2 , SkillVerse generates personalized insights through a tree-structured model assessment, tailored to the level of detail preferred by human scientists. To quantify and extract actionable insights from critiques to diverse real-world data, we introduce atomic judgment : an assessment of an indivisible aspect of model capability. Next, we conduct agglomerative clustering on these atomic judgments based on their semantic distance, resulting in a dendrogram. This tree can be chopped at different levels, resulting into clusters of varying sizes or granularities. Each cluster represents a specific skill, for which we calculate success rates to evaluate model performance ( \lx@sectionsign 2 ). SkillVerse produces insights of current model behaviors, such as Gemini, Claude, and GPT-4 ( \lx@sectionsign 3 ). For instance, on Arena-Hard benchmark Li et al. 2024 where Claude-3.5-Sonnet Anthropic 2024 ranks 2nd and Gemini-1.5-Pro Google 2024a ranks 6th, 1 1 1 Under Arena-Hard-Auto on https://lmarena.ai , as of Nov 18th, 2024. SkillVerse finds out Gemini needs improvement in debugging, writing command lines, and business analysis. On the other hand, the higher-ranked Claude, falls short in providing analogical examples, debate, and evaluate arguments. Notably, by comparing different sized models within the same family, we also identify instances of inverse scaling, where larger models underperform smaller ones due to strong parameterized knowledge McKenzie et al. . Such tasks include handling word inclusion/exclusion, word counts, and adherence to specific formats. To validate SkillVerse ’s ability to identify true model errors and its potential values for model improvement, we design a few extended improvement explorations that lead to promising gains. In \lx@sectionsign 4 , we show that the dendrogram enhances in-context learning by enabling a tree search algorithm that adaptively selects challenging and relevant examples as contrastive few-shot demonstrations. This approach achieves a 25% relative improvement over the standard contrastive in-context learning method (C-ICL, Yan et al. 2021 ). In \lx@sectionsign 5 , we demonstrate that a strong reasoner such as GPT-4o can digest the model proficiency report generated from SkillVerse to predict weaknesses in unseen scenarios. For example, the model proficiency on ten hypothesized tasks is only 55%, 22% lower than uninformed predictions by the same reasoner. We showcase that SkillVerse can serve as a powerful tool for providing fine-grained interpretation of model behaviors and developing targeted improvement of discovered model deficiencies during inference. Future research could also leverage the actionable feedback derived from SkillVerse to a wide range tasks: such as model routing, curatiing targeted training data for specific subdomains where the current model underperforms, and etc. Figure 3: Upper right : A dendrogram produced by SkillVerse on a combination of two datasets: ChatbotArena Chiang et al. 2024 and IF Eval Zhou et al. 2023 . Main figure : Multiple layers of nested clusters that represent model proficiencies from coarse-grained to fine-grained, by horizontally slicing the dendrogram at different levels. For each group, we can then calculate skill-level model proficiency based on the atomic judgments, as demonstrated in the parenthesis. An LLM summarizes all members in the same cluster and generate the skill-level description. 2 S k i l l V e r s e : Diagnosis Framework 2.1 Overview Figure 2 presents the overall framework of SkillVerse . Starting with a large set of user prompts and model responses, we collect critiques that evaluate model responses in detail ( \lx@sectionsign 2.2 ). These critiques are then parsed into atomic judgments, enabling efficient organization and large-scale quantification. Using bottom-up clustering algorithms, the atomic judgments are structured into a dendrogram. To interpret the results, the dendrogram is unfolded at varying levels of granularity based on the detail preferred by engineers and researchers ( \lx@sectionsign 2.3 and Figure 3 ). Finally, clusters derived independently are anchored to support multi-party analyses ( \lx@sectionsign 2.4 ). 2.2 Collecting Accurate Critiques A straightforward way to collect critiques is adopting language models off-the-shelf or finetune them on domain specific data—a reliable approach for evaluating content relevance, style. etc. For a comprehensive representation of model capabilities, our critique model takes in both positive and negative samples and is capable

Read the complete text