From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs

Itay Itzhak Eliya Habba Gabriel Stanovsky Yonatan Belinkov, Technion – Israel Institute of Technology The Hebrew University of Jerusalem [email protected] { \{ eliya.habba,gabriel.stanovsky } \} @mail.huji.ac.il [email protected] · 2026

Read From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs by Itay Itzhak Eliya Habba Gabriel Stanovsky Yonatan Belinkov, Technion – Israel Institute of Technology The Hebrew University of Jerusalem [email protected] { \{ eliya.habba,gabriel.stanovsky } \} @mail.huji.ac.il [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14137.txt.

Introduction

Itzhak and three colleagues study vibe-testing, the informal way users judge language models through hands-on trials tied to their own work. Drawing on a survey and on public comparison reports, they formalize the habit as two personal choices: what to test and how to judge answers. A trial pipeline built on that framing pairs tailored prompts with user-aware scoring. It shifts which model wins on coding tests, suggesting formalized vibe-testing can bridge benchmark scores and lived experience.

Catalog record

CREATORItay Itzhak Eliya Habba Gabriel Stanovsky Yonatan Belinkov, Technion – Israel Institute of Technology The Hebrew University of Jerusalem [email protected] { \{ eliya.habba,gabriel.stanovsky } \} @mail.huji.ac.il [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE119498
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14137

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs Itay Itzhak Eliya Habba Gabriel Stanovsky Yonatan Belinkov, Technion – Israel Institute of Technology The Hebrew University of Jerusalem [email protected] { \{ eliya.habba,gabriel.stanovsky } \} @mail.huji.ac.il [email protected] Abstract Evaluating LLMs is challenging, as benchmark scores often fail to capture models’ real-world usefulness. Instead, users often rely on “vibe-testing”: informal experience-based evaluation, such as comparing models on coding tasks related to their own workflow. While prevalent, vibe-testing is often too ad hoc and unstructured to analyze or reproduce at scale. In this work, we study how vibe-testing works in practice and then formalize it to support systematic analysis. We first analyze two empirical resources: (1) a survey of user evaluation practices, and (2) a collection of in-the-wild model comparison reports from blogs and social media. Based on these resources, we formalize vibe-testing as a two-part process: users personalize both what they test and how they judge responses. We then introduce a proof-of-concept evaluation pipeline that follows this formulation by generating personalized prompts and comparing model outputs using user-aware subjective criteria. In experiments on coding benchmarks, we find that combining personalized prompts and user-aware evaluation can change which model is preferred, reflecting the role of vibe-testing in practice. These findings suggest that formalized vibe-testing can serve as a useful approach for bridging benchmark scores and real-world experience. 1 1 1 See code and study artifacts at: https://technion-cs-nlp.github.io/vibe-testing-llms . 1 Introduction Evaluating LLMs has been a long-standing challenge in NLP research ( 19 ; 2 ) , as popular evaluation suites typically report performance as aggregated scores on standardized tasks ( 36 ; 12 ) . However, these scores often miss the usefulness of models in real-world workflows ( 17 ; 25 ; 29 ) . In practice, model usefulness often depends on context-dependent criteria, such as clarity, ease of use, or workflow fit ( 34 ; 31 ) . As a result, strong benchmark scores do not necessarily imply a good fit for users’ needs in everyday tasks. Figure 1: Anatomy of a “vibe-test”. In practice, users evaluate LLMs by “vibe-testing” them: – writing personalized prompts that test specific behaviors and judging models’ responses using personal subjective criteria. We analyze recurring patterns of vibe-testing in real-world user comparisons, formalize them into a two-part structure, and present a proof-of-concept pipeline for automated vibe-testing. Example taken from Tom’s Guide . In response, many users turn to “ vibe-testing ”: an informal practice of evaluating models through targeted experiments or extended personal use ( 6 ; 11 ) . Instead of relying solely on benchmark scores, users compare models on tasks that resemble their own workflows and judge the responses qualitatively. These comparisons often focus on practical aspects of model behavior, such as writing style, clarity of explanations, ease of use, or how well the output fits a specific workflow (Figure 1 ). Vibe-testing can be performed by individual users or shared by community members on blogs, forums, and social media. Users’ reliance on vibe-testing implies that it captures valuable aspects of model performance that benchmarks often miss. However, vibe-testing is inherently informal and subjective – different users test different tasks and judge responses from their own perspective. As a result, insights from these evaluations remain scattered and fragmented, making them difficult to compare or transfer across settings. This informality leaves a gap between the practical insights of vibe-testing and our ability to study them systematically. Recent work has suggested assessing model “vibe” and personalized evaluation, but none have empirically studied vibe testing itself as a user practice or suggested an evaluation framework grounded in it (Section 2 ). In this work, we empirically study the practice of vibe-testing, formalize its distinctive patterns, and propose a proof-of-concept evaluation pipeline for systematic analysis. We begin by examining vibe testing in practice, drawing on two empirical sources. First, we conduct a survey on evaluation practices, asking questions such as “What do you look for when testing a model?” . Second, we collect an “in-the-wild” corpus of model comparisons from blogs, forums, tech articles, and YouTube reviews. We annotate these examples to identify recurring patterns in users’ test design and response evaluation. Together, these sources provide real-world evidence of how users vibe-test models: what they test and what they look for when judging responses. Building on these empirical sources, we formalize vibe-testing as an evaluation practice defined by two recurring types of dimensions. Input dimensions capture what users test and how they construct prompts, while Output dimensions capture how users judge model responses. For example, in coding assistance, input dimensions can include the type of coding task or the amount of context provided (e.g., debugging a codebase). Output dimensions can include clarity, adherence to constraints, and fit to the user’s workflow (e.g., production-ready code). This formalization makes vibe-testing easier to compare and analyze, and provides a basis for systematic reproduction. We leverage this formalization and introduce a proof-of-concept evaluation pipeline that mirrors the two-part structure of vibe-testing. Given a brief user description, the pipeline first rewrites benchmark prompts to reflect that user’s likely context and preferences. It then compares models head-to-head by judging their responses along user-relevant output dimensions from that user’s perspective. We apply the pipeline to coding benchmarks and find that personalizing both the prompt and the judgment criteria can change which model is preferred. In several head-to-head comparisons, the preferred model flips relative to the original benchmark prompts, while non-personal rewrites largely preserve the original ordering. These results echo the core idea behind vibe-testing: model preferences can change when both the task framing and response judgment are tailored to the user. Overall, this work takes a first step toward turning vibe-testing from an informal practice into a structured form of user-centered evaluation. (a) (b) Figure 2: Benchmarks vs. vibe-testing in practice. Left: What benchmarks miss. Survey participants selected real-world qualities that benchmarks fail to capture (multi-select), including workflow and style fit, handling ambiguity, stability, clarity, and trust. Right: How users test models. Common strategies include trying tasks from one’s own workflow, side-by-side comparisons, probing style, stress-testing ambiguity handling, checking repeated runs stability, and using recurring “test prompts,”. Notably, the qualities participants say benchmarks miss largely match what they explicitly probe during vibe-testing. 2 Background: personal vibe evaluation Recent work has shown that standard benchmarks often miss aspects of model performance that matter to users in real-world use, motivating more qualitative user-centered evaluation ( 2 ; 34 ) . However, systematically capturing vibe-testing evaluation requires a framework that is subjective, personalized, and grounded in empirical evidence. To our knowledge, prior work typically captures only one of these properties. Vibe-based subjective evaluation. A few recent papers study “vibe”-like aspects of model behavior, but they do so in different ways. Most relevant is VibeCheck ( 7 ) , which measures qualitative “vibe” differences between models, but at the population level rather than for individual users. Vibe Checker ( 38 ) adds verifiable instruction checks to coding tasks, but covers only automatically checkable traits and leaves out softer subjective dimensions, limiting potential coverage. Beyond these, HELM Instruct ( 36 ) offers general stylistic evaluation while ChatBench ( 3 ) focuses on evaluating interactive conversations, both relying on predefined

Read the complete text