Decision-Oriented Text Evaluation
Yu-Shiang Huang, Chuan-Ju Wang, Chung-Chi Chen · 2025
Read Decision-Oriented Text Evaluation by Yu-Shiang Huang, Chuan-Ju Wang, Chung-Chi Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01923.txt.
Description
Yu-Shiang Huang and two co-authors argue that generated text should be graded by the decisions it leads to, not by how closely it matches a reference. Standard scores for machine writing, such as word overlap, barely predict whether a text helps anyone act well. Their test bed is finance: morning market digests and closing commentaries were given to human investors and to language-model agents, who traded on the texts alone.
Neither humans nor agents beat chance when working only from plain summaries. Richer analytical commentary changed the picture: mixed teams of humans and agents using it outperformed either side working alone by a clear margin. The authors frame this as a call to evaluate writing by decision outcomes, especially in high-stakes fields. Their framework also exposes where summary text quietly fails its readers, a weakness that surface metrics cannot see.
Catalog record
| CREATOR | Yu-Shiang Huang, Chuan-Ju Wang, Chung-Chi Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-01923