Decision-Oriented Text Evaluation

Yu-Shiang Huang, Chuan-Ju Wang, Chung-Chi Chen · 2025

Read Decision-Oriented Text Evaluation by Yu-Shiang Huang, Chuan-Ju Wang, Chung-Chi Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01923.txt.

Description

Yu-Shiang Huang and two co-authors argue that generated text should be graded by the decisions it leads to, not by how closely it matches a reference. Standard scores for machine writing, such as word overlap, barely predict whether a text helps anyone act well. Their test bed is finance: morning market digests and closing commentaries were given to human investors and to language-model agents, who traded on the texts alone.

Neither humans nor agents beat chance when working only from plain summaries. Richer analytical commentary changed the picture: mixed teams of humans and agents using it outperformed either side working alone by a clear margin. The authors frame this as a call to evaluate writing by decision outcomes, especially in high-stakes fields. Their framework also exposes where summary text quietly fails its readers, a weakness that surface metrics cannot see.

Catalog record

CREATORYu-Shiang Huang, Chuan-Ju Wang, Chung-Chi Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-01923

Browse the full library · About the Tower of Babel