Evaluating Ill-Defined Tasks in Large Language Models
Yi Zhou, Basel Shbita · 2026
Read Evaluating Ill-Defined Tasks in Large Language Models by Yi Zhou, Basel Shbita (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17067.txt.
Description
Zhou and Shbita examine why evaluating language models on ill defined tasks so often misleads. Such tasks have unclear input and output spaces and fuzzy success criteria. They analyze two case studies. For complex instruction following, they find thin coverage of real world complexity, sensitivity to phrasing, mismatched metrics, and unstable LLM judges. For turning text into Mermaid sequence diagrams, they show that multi part criteria can give useful, actionable signals. They conclude that current scores often mix distinct failure modes. Better designs should be robust and interpretable.
Catalog record
| CREATOR | Yi Zhou, Basel Shbita |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30972 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17067