An Empirical Comparison of Text Summarization: A Multi-Dimensional Evaluation of Large Language Models
Anantharaman Janakiraman, Behnaz Ghoraani, Ph.D · 2025
Read An Empirical Comparison of Text Summarization: A Multi-Dimensional Evaluation of Large Language Models by Anantharaman Janakiraman, Behnaz Ghoraani, Ph.D (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-04534.txt.
Description
Janakiraman and Ghoraani present a 2025 benchmark study pitting seventeen large language models against each other at summarizing text. Seven datasets spanned news, patents, law, medicine, and chat. Each model wrote summaries at three set lengths. Scores covered factual soundness, meaning match, word overlap, and human-like quality, plus speed and cost.
No single model won everywhere. One open model led on factual accuracy, while a Claude model led on human-like quality. Google's Flash models won on speed and price. Technical texts tripped models up; chat text suited them. Shorter outputs held facts best, yet longer ones read better. The authors translate the split results into picks for common use cases.
Catalog record
| CREATOR | Anantharaman Janakiraman, Behnaz Ghoraani, Ph.D |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-04534