Evaluating Ill-Defined Tasks in Large Language Models

Yi Zhou, Basel Shbita · 2026

Read Evaluating Ill-Defined Tasks in Large Language Models by Yi Zhou, Basel Shbita (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17067.txt.

Description

Zhou and Shbita examine why evaluating language models on ill defined tasks so often misleads. Such tasks have unclear input and output spaces and fuzzy success criteria. They analyze two case studies. For complex instruction following, they find thin coverage of real world complexity, sensitivity to phrasing, mismatched metrics, and unstable LLM judges. For turning text into Mermaid sequence diagrams, they show that multi part criteria can give useful, actionable signals. They conclude that current scores often mix distinct failure modes. Better designs should be robust and interpretable.

Catalog record

CREATORYi Zhou, Basel Shbita
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30972
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.17067

Browse the full library · About the Tower of Babel