Adaptively profiling models with task elicitation

Davis Brown, Prithvi Balehannina, Helen Jin, Shreya Havaldar · 2025

Read Adaptively profiling models with task elicitation by Davis Brown, Prithvi Balehannina, Helen Jin, Shreya Havaldar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01986.txt.

Introduction

Standard benchmarks miss the ways language models actually fail in the wild, leaving experts to poke at outputs by hand. Brown and co-authors automate that poking with task elicitation, a method that invents new tests aimed at a model's weak spots.

The method turns up hundreds of natural-language tasks, about ten times more than earlier approaches, where leading models fail in consistent, systematic ways. The failures span forecasting, harassment handling, and more. Two examples stand out: one top model too readily links quantum computing with artificial general intelligence, and another hallucinates more when false claims are repeated inside the prompt. The authors present task elicitation as a profiling instrument that adapts to each model instead of freezing the test in advance.

Catalog record

CREATORDavis Brown, Prithvi Balehannina, Helen Jin, Shreya Havaldar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE112 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01986

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptively profiling models with task elicitation Davis Brown, Prithvi Balehannina, Helen Jin, Shreya Havaldar Abstract Language model evaluations often fail to characterize consequential failure modes, forcing experts to inspect outputs and build new benchmarks. We introduce task elicitation , a method that automatically builds new evaluations to profile model behavior. Task elicitation finds hundreds of natural-language tasks---an order of magnitude more than prior work---where frontier models exhibit systematic failures, in domains ranging from forecasting to online harassment. For example, we find that Sonnet 3.5 over-associates quantum computing and AGI and that o3-mini is prone to hallucination when fabrications are repeated in-context 1 1 1 We release our datasets on HuggingFace and our code https://github.com/davisrbr/adaptive_evals . 1 Introduction Language models often have failure modes that are difficult to identify with evaluations ( Karpathy, 2024 ) . As language models reach billions of users and these behaviors are not caught, they pose significant safety problems. Today, a performant model subtly lacking in legal domain knowledge can hallucinate and provide incorrect arguments to a lawyer ( Magesh et al., 2024 ) and chatbots harass teenagers communicating with them ( Hinduja, 2023 ) . Looking forward, expert forecasters anticipate risks ranging from cyber- to bio-security to escalate to large-scale harm ( Phuong et al., 2024 ) . Why are current evaluations inadequate? The paradigm of ‘static’ benchmarks, where fixed sets of questions are curated by humans, faces two main challenges. First, constructing evaluations that challenge capable frontier models requires actively involving leading subject matter experts, with costs sometimes reaching hundreds or thousands of dollars per question ( Rein et al., 2023 ; Glazer et al., 2024 ) . And yet despite this, model evaluations still have limited coverage: e.g., a prominent AI lab recently released a model that was overly agreeable to hundreds of millions users ( OpenAI, 2025 ) ; this bug slipped past extensive offline evaluations and surfaced only on deployment. Second, even after an evaluation is constructed, performance measures are often misleading ( Dunlap et al., 2024 ) . Models often cheat by taking advantage of scaffolding issues ( Meng et al., 2025 ) , e.g., by modifying a test-case instead of writing correct code. Summarizing these two challenges, we arrive at the question: How can we automatically generate and validate descriptions of LLM behavior? Elicited Tasks o3-mini gpt-4o 4o-mini Llama 3.3 Sonnet 3.5 Domain Reasoning Extensive Carveouts Precise Timing Quantum and AGI Environmental and Finance Alignment Fake Brainstorming Historical/Harmful Contexts Repeating Falsities False Details Social Harms Reverse-Psychology ‘Merciless’ Mode Formal and Dismissive Courteous and Sarcastic Task Elicited (model struggles on task) Task Not Elicited (model performs well) Table 1: Tasks elicited by gpt-4o to profile five models. We apply task elicitation to domain reasoning ( Legal Reasoning, Forecasting Consistency), alignment benchmarks ( Jailbreaking, Truthfulness), and social harms ( Cyberharassment, Cultural Politeness). For details, see Section 3 and Appendix B for full task descriptions. Two lines of work point towards solutions to these evaluation issues. First, adaptive evaluations ( Li et al., 2025b ) automatically create new problems that challenge the language model under evaluation—this increases the coverage and scalability of benchmarking. However, it is not obvious how to interpret scores from adaptive evaluations, which produce questions adversarially difficult for a model. Second, another line of work goes beyond summary statistics and attempts to find richer natural language explanations of language model performance within some domain ( Yang et al., 2024 ; Dunlap et al., 2024 ) . However, existing frameworks for natural language descriptions are based on only a single observation of model behavior, i.e. they do not adaptively create new questions to test whether the natural language description is faithful. Our approach: Task elicitation— adaptive and interpretable profiling. Neither adaptive benchmarks nor qualitative reports alone give a faithful picture of a frontier model’s behaviour: the former are hard to interpret, while the latter are easy to overfit. We close this gap with task elicitation , an adaptive framework that automatically (i) hypothesizes failure modes in natural language, (ii) generates new questions to test those hypotheses, and finally (iii) clusters these descriptions into tasks that describe model behavior and failure modes. Figure 1 shows an overview of the framework and Figure 2 shows example questions and the topic diversity of the tasks created during the adaptive profiling. Table 1 provides examples of the tasks elicited, comparing across five different models. In summary, we make the following contributions: • Profiling models adaptively : We profile the behavior of language models with natural language tasks . Our profiles are novel in that they are adaptive , found with multiple rounds of hypothesizing and testing. This brings together recent work on qualitative evaluations ( Yang et al., 2024 ) and automated benchmarking ( Li et al., 2025b ) . • Scalability : Our framework finds hundreds of targeted natural language descriptions that diversely profile a model’s weaknesses, compared to previous work that finds only a single-digit number of descriptions. • Generalizability : We show that task elicitation generalizes across domains by identifying hundreds of failure modes in forecasting, legal reasoning, hallucination, jailbreaking, cultural politeness, and a new cyber-harassment evaluation. We also demonstrate that the discovered tasks often transfer to other models. Figure 1: Task elicitation generates new natural language profiles of model capabilities and weaknesses, found adaptively. First, the results from an initial static evaluation (for example, TruthfulQA ( Lin et al., 2021 ) ) are retrieved– this includes the target LLM’s (the model being evaluated) CoT and whether it solved the question correctly. The profiling model generates new evaluation questions using the pattern of the incorrect/correct examples and the target LLM’s CoT. Finally, the questions (filtered for diversity/difficulty/correctness) are clustered and summarized ( Bravansky et al., 2025 ) to form higher-level tasks. For details, see Section 2 . 2 Task Elicitation Task elicitation works by hypothesizing model failures and then testing these hypotheses by synthesizing new natural language tasks. In existing work, the natural language profiles of language model behaviors are observational , built by distilling a single static snapshot of language model outputs into natural language ( Yang et al., 2024 ; Dunlap et al., 2024 ) . However, this will miss out on important domain behavior inadequately captured by the evaluation– for example, general knowledge benchmarks can only cover a small fraction of facts. Motivated by this, we optimize for questions that are particularly useful for understanding model behavior (e.g., are difficult to answer or elicit a hallucination) ( Li et al., 2025b ; Chen et al., 2024 ) for the purpose of creating more relevant natural language profiles. We provide more related work in Section 4 . Task elicitation therefore adaptively generates a sets of ‘tasks’ that describe, in easily understandable natural language, groups of inputs that are informative concerning domain performance. A model profiling run (see Figure 1 for the schematic) has three steps: (1) defining the domain with a dataset and natural language prompt, (2) using the profiler model to generate new questions via an adaptive evaluation, and (3) grouping the successful questions into higher-level tasks that profile the model’s behavior on the domain. These tasks, unlike the standard outputs from automated adaptive benchmarks, are in natural

Read the complete text