Systematic assessment of text summarization methods for biomedical literature from frequency methods to language models Authors: Fabio Baumgärtel, Enrico Bono, Lucas Fillinger, Louiza Galou, Kinga Kęska-Izworska, Samuel M Walter, Peter Andorfer, Klaus Kratochwill, Paul Perco, Matthias Ley Published: 2026 ABSTRACT SummaryThe rapid expansion of biomedical literature demands automated summarization tools that reliably condense research articles into concise, accurate summaries. We benchmarked 62 summarization methods, ranging from frequency-based and TextRank extractors to encoder-decoder models (EDMs) and large language models (LLMs), on 1,000 biomedical abstracts from 20 journals across ScienceDirect and Cell Press, using author-written highlights as reference summaries. Models were evaluated with a composite suite of lexical, semantic, and factual metrics, including ROUGE, BLEU, METEOR, embedding-based similarity, and factuality scores. General-purpose models (e.g., Mistral, GPT, and Llama) achieved the highest overall performance across lexical and semantic dimensions, outperforming reasoning-oriented (e.g., DeepSeek and Magistral) and domain-specific (e.g., BioGPT and BioMistral) models. Notably, medium-sized models outperformed large-scale models, suggesting an optimal balance between model capacity and efficiency, while classical extractive methods lagged behind neural approaches. These findings provide a systematic reference for selecting biomedical summarization tools and highlight that broad pretraining outperforms narrow domain adaptation.Keywords: biomedical text summarization, natural language processing, large language models, benchmarking Introduction The exponential growth of scientific literature has created a demand for text summarization methods to support scientists in prioritizing articles, extracting relevant information, and interpreting research findings. Automatic text summarization (ATS) methods have evolved from statistical approaches to deep learning-based models, thus becoming increasingly sophisticated and reliable in capturing the essential parts of complex research articles. Although ATS methods have been previously evaluated and described,1,2 only a few have focused on scientific literature.3,4 Recent work has begun to benchmark large language models (LLMs) on biomedical summarization, though only for a limited set of models and typically against abstract-based references.5,6,7 A systematic comparison of word-frequency methods to state-of-the-art LLMs is still lacking. Over time, these methods have developed into distinct approaches that differ in how they identify or generate summary content. Early pre-neural summarization was mainly characterized by unsupervised extractive approaches that rank sentences by word or concept frequencies to identify relevant sentences. The field was initially shaped by Luhn’s assumption that recurrent words signal importance8 and Edmundson’s use of cue words, title words, and sentence position.9 Later work adopting weighting term frequency-inverse document frequency (TF-IDF) refined this idea by down-weighting frequent, low-specificity terms,10 laying the foundation for more advanced methods in scientific text summarization.11 Graph-based methods judged the importance against the document’s global structure rather than local word counts. TextRank, widely used in the biomedical domain, applies the PageRank algorithm to a sentence-similarity graph to select top-ranked sentences.12 The advent of sequence-to-sequence (Seq2seq) frameworks shifted the field toward neural approaches capable of paraphrasing and condensing text using encoder-decoder architectures, originally implemented with recurrent neural networks (RNNs), long short-term memory (LSTM) networks, and gated recurrent units (GRUs),13,14 while self-attention enabled parallel processing of sequences and modeling of long-range context.15 This laid the foundation for transformer architectures that quickly gained popularity in performing a wide range of natural language processing (NLP) tasks. Building on bidirectional encoder representations from transformers (BERT),16 several abstractive summarization models have emerged, including bidirectional and auto-regressive transformer (BART), a denoising autoencoder for pretraining Seq2seq models17 and its biomedical adaptations18,19; the unified text-to-text transfer transformer (T5) model20; Pre-training with extracted gap-sentences for abstractive summarization sequence-to-sequence (PEGASUS),21 including domain-specific PubMed variants; and Longformer-based models22 that can handle longer inputs.23 These models are typically pretrained independently, while Longformer-based variants may additionally leverage a robustly optimized BERT approach (RoBERTa), an optimized version of BERT trained on a larger corpus. More recently, the field of ATS has quickly moved toward decoder-only architectures at the core of LLMs, which capture semantic relations with greater flexibility and specificity. LLMs can be classified as (1) general-purpose models, which apply broad domain knowledge across diverse NLP tasks, such as the proprietary generative pre-trained transformer (GPT)24 and Claude25 series and the open-source LLM Meta AI (Llama)26 and Gemma27 families; (2) reasoning-oriented models, characterized by logical text understanding through iterative chain-of-thought processing and instruction tuning28 such as DeepSeek-R1,29 Qwen3,30 and Mistral’s Magistral,31 together with the reasoning tiers of the proprietary families (e.g., GPT-5 and Claude Opus 4); and (3) domain-specific models, tailored for specialized tasks or specific scientific domains, such as the biomedical models OpenBioLLM-Llama3,32 BioGPT,33 and BioMistral,34 and SciLitLLM35 for scientific literature. The gold-standard dataset used for benchmarking and the complete set of models evaluated in this study are listed in Tables 1 and 2 in the STAR Methods section. Table 1Overview of journals included in the gold-standard datasetPublisherJournalsScienceDirectCytokine; Developmental Cell; Drug Discovery Today; FEBS Letters; Gene; Genomics; Journal of Biotechnology; Journal of Molecular Biology; Journal of Proteomics; The International Journal of Biochemistry & Cell BiologyCell PressCancer Cell; Cell; Cell Chemical Biology; Cell Genomics; Cell Host & Microbe; Cell Metabolism; Cell Reports; Cell Reports Medicine; Cell Stem Cell; Cell SystemsEach journal contributed 50 articles, resulting in 500 articles from ScienceDirect and 500 articles from Cell Press. Overview of journals included in the gold-standard dataset Each journal contributed 50 articles, resulting in 500 articles from ScienceDirect and 500 articles from Cell Press. Table 2Overview of summarization methods evaluated in this studyCategoryMethods/ModelsTraditional modelsfrequency; textrankGeneral-purpose EDMsfacebook/bart-base; google/pegasus-large; google-t5/t5-base; google-t5/t5-largeDomain-specific EDMscsebuetnlp/mT5_multilingual_XLSum; facebook/bart-large-cnn; google/bigbird-pegasus-large-pubmed; google/pegasus-cnn_dailymail; google/pegasus-pubmed; google/pegasus-xsum; led_large_16384_arxiv_summarizationGeneral-purpose SLMschat_swiss-ai/Apertus-8B-Instruct-2509; gemma3:1b; gemma3:270M; gemma3:4b; gpt-4.1-mini; gpt-4o-mini; granite3.3:2b; granite3.3:8b; granite4:micro; granite4:micro-h; granite4:small-h; granite4:tiny-h; llama3.1:8b; llama3.2:1b; llama3.2:3b; mistral:7b; PetrosStav/gemma3-tools:4b; phi3:3.8bGeneral-purpose LLMsclaude-3-5-haiku-20241022; gemma3:12b; gpt-3.5-turbo; gpt-4.1; gpt-4o; mistral-large-2411; mistral-medium-2505; mistral-nemo:12b; mistral-small-2506; mistral-small3.2:24b; phi4:14bReasoning-oriented SLMsdeepseek-r1:1.5b; deepseek-r1:7b; deepseek-r1:8b; qwen3:4b; qwen3:8bReasoning-oriented LLMsclaude-opus-4-1-20250805; claude-opus-4-20250514; claude-sonnet-4-20250514; deepseek-r1:14b; gpt-5-2025–08–07; gpt-5-mini-2025–08–07; gpt-5-nano-2025–08–07; gpt-oss:20b; magistral-medium-2509Domain-specific SLMschat_aaditya/OpenBioLLM-Llama3-8B; chat_Uni-SMART/SciLitLLM1.5-7B; completion_microsoft/biogpt; conversational_BioMistral/BioMistral-7B; medllama2:7bDomain-specific LLMschat_Uni-SMART/SciLitLLM1.5-14BModels organized by category. Overview of summarization methods evaluated in this study Models organized by category. Despite this range of approaches, these method families have not been evaluated together on biomedical literature, leaving little guidance on which to choose. This study addresses this gap by systematically evaluating 62 text summarization models, ranging from word-frequency methods to state-of-the-art LLMs, using a dataset of 1,000 biomedical abstracts and corresponding highlights sections as reference summaries. We provide recommendations for selecting appropriate tools to accelerate knowledge discovery in biomedical sciences and discuss the strengths and limitations of each approach. Results Our benchmark results offer a comparative view of summarization performance across all evaluated models. We first examined the correlations among the chosen evaluation metrics and, based on the findings, identified the best-performing model across lexical, semantic, and factual dimensions. Next, we compared model performance across categories and families to identify significant differences. We additionally performed a data leakage analysis to ensure that the inclusion of benchmarking data in training data had no impact on the results. We then analyzed how well automatic metrics aligned with human judgment based on expert-level result assessment, and finally assessed whether an LLM-as-a-judge panel could reproduce these rankings. Figure 1 provides an overview of the complete benchmarking workflow of this study from dataset construction and model inference to metric computation, score aggregation, and statistical analysis. Overview of the benchmarking workflow (1) Construction of a gold-standard dataset comprising 1,000 biomedical titles, abstracts, and author-provided highlight sections. (2) Suite of 62 diverse summarization models. (3) Uniform prompting of all models to generate summaries for each abstract. (4–5) Computation of summary quality metrics capturing lexical overlap, semantic similarity, and factual consistency. (6) Standardization of metric scores using Z score normalization to ensure comparability across evaluation dimensions. (7) Ranking of models based on overall performance scores. (8) Quality assessment of generated summaries across four quality dimensions (coherence, fluency, relevance, and consistency), combining expert human evaluation on a subset of models with (9) LLM-as-a-judge evaluation from three provider-independent judges run across all models. Metric correlations Summary quality was quantified with 12 automatic metrics spanning 3 dimensions: lexical overlap, semantic similarity, and factual consistency (see “evaluation metrics” in STAR Methods). Pairwise Spearman correlations were computed to analyze the relationships between the different evaluation metrics (Figure 2). Strong positive correlations were observed among most lexical-based metrics (recall-oriented understudy for gisting evaluation [ROUGE]-1/2/L, metric for evaluation of translation with explicit ordering [METEOR], and bilingual evaluation understudy [BLEU]), with the correlation between METEOR and BLEU marking an exception (ρ = 0.23). ROUGE variants showed almost identical behavior (ρ ≥ 0.88), while BLEU and METEOR demonstrated slightly weaker but still substantial alignment with ROUGE measures (ρ = 0.51–0.82). Spearman correlation between evaluation metrics Spearman correlation coefficients (ρ) between two metrics based on their mean scores across all models are given. Metric categories (lexical, semantic, and factual) are indicated on the x and y axes. For visualization purposes, cells with values higher than 0.8 are shown with white text. Most semantic-based metrics (RoBERTa, decoding-enhanced BERT with disentangled attention [DeBERTa], and all-mpnet-base-v2) showed high internal consistency (ρ > 0.5), reflecting their shared focus on semantic similarity. When compared with lexical-based metrics, correlations were moderate to strong in most cases, indicating that both categories capture related but not identical dimensions of summary quality. Concerning factual consistency metrics, AlignScore and the two MiniCheck variants exhibited very strong correlations with each other (ρ = 0.82–0.94), while SummaC showed moderately positive correlations with the other factual metrics (ρ = 0.41–0.69). This likely reflects methodological differences, as SummaC relies on zero-shot natural language inference over sentence pairs, while MiniCheck and AlignScore use models specifically optimized for factual consistency assessment. When compared with lexical- and semantic-based metrics, correlations were generally weaker and, in some cases, even slightly negative. This pattern can largely be attributed to the different point of reference used by factual metrics. Overall, these relationships demonstrate that the various metrics are broadly consistent while providing complementary perspectives. This supports the use of an aggregated “overall performance score” as a balanced indicator of overall summarization performance. Overall model performance Based on overall performance ranks, referred to as “performance rank” and derived from our multi-metric evaluation framework (see “overall performance score” in STAR Methods), models from the Mistral family were top-ranked across the three evaluated metric dimensions, as depicted in Figure 3. Ranking of all evaluated summarization models across metric dimensions Overview of the performance of all evaluated models across lexical, semantic, and factual metric dimensions. Each row corresponds to one model. Columns show the overall performance rank and independent ranks for each metric dimension, with lower ranks indicating better performance. The figure displays each model’s family and category. Models are sorted by the overall performance rank. Ranks are displayed using a continuous diverging color gradient, ranging from navy-blue tones for top-performing models through light neutral shades for mid-ranked models to dark red tones for low-performing models. The mistral-small3.2:24b model ranked first, followed by mistral-small-2506 and mistral-medium-2505. The lowest-ranked models included OpenBioLLM-Llama3-8B, bigbird-pegasus-large-pubmed and pegasus-pubmed from the PEGASUS family, and mT5_multilingual_XLSum from the T5 family. Overall, domain-specific small language models (SLMs) and encoder-decoder models (EDMs) showed poor performance across all the different dimensions of metrics. Among the 10 top-ranked models, 6 were general-purpose LLMs and 4 were general-purpose SLMs. A similar trend was observed in the top half of the ranking (positions 1–31) with the presence of just a few reasoning-oriented and domain-specific models. In contrast, in the lower half of the ranking, where models performed poorly across most metric dimensions, the majority were reasoning-oriented SLMs, general-purpose EDMs, domain-specific EDMs, and traditional models. Considering each metric dimension separately, several Mistral models ranked high in the lexical dimension, with GPT-5-nano, a reasoning-oriented LLM from the GPT family, ranking third. In the semantic dimension, Mistral models achieved high ranks, but the highest positions were occupied by the Phi4:14b general-purpose LLM and several Granite models, including granite4:tiny-h and granite3.3:2b. Finally, the highest ranks based on the factual dimension were covered by traditional models and some EDMs. Detailed performance indications for each model across all individual ranked metrics are provided in Figure S2. Models could also decline to summarize by returning the INSUFFICIENT_FINDINGS token (see “prompt design” in STAR Methods). This was rare overall, occurring in 736 of 62,000 generated summaries (1.2%) and exclusively among prompt-based models. Usage was concentrated in the two domain-specific biomedical models OpenBioLLM-Llama3-8B (20.0%) and medllama2 (9.7%), while all other models returned it in fewer than 6% of cases. Category comparisons To compare model performance across categories, we displayed the distribution of overall performance scores within each category, along with the best- and worst-performing individual models (Figure 4A). Results of the Games-Howell post-hoc comparisons between individual groups are given in Figure 4B. Overall performance and pairwise statistical comparison of the nine model categories (A) Distributions of overall performance scores across the nine model categories, with each model represented by a black dot. Categories are ordered from lowest to highest-performing according to the mean performance score. Boxes span the 25th to 75th percentile, the center line denotes the mean, and whiskers extend to the minimum and maximum values. The best- and worst-performing models of each category are highlighted. (B) Games-Howell pairwise adjusted p value matrix comparing all category pairs. Each cell shows the adjusted p value for the difference in metric mean scores between two categories, with white-to-gray shading indicating smaller p values. All general-purpose LLMs achieved high overall scores, thus forming the top category of models, with general-purpose SLMs following second, however showing a slightly wider spread in performance scores. Domain-specific LLMs and reasoning-oriented LLMs followed in third and fourth place based on average overall performance. Domain-specific EDMs, domain-specific SLMs, traditional models, and general-purpose EDMs performed significantly worse as compared with general-purpose language models (LMs). Domain-specific EDMs and domain-specific SLMs displayed the widest spread of performance (Figure 4A). Family comparisons To complement the category-level findings, we examined performance across model families, since models within the same family often share architectural features or training strategies that may cause consistent performance trends. Figure 5A shows the distribution of overall performance scores across all model families. Similar to the category-level analysis, there were clear performance differences between architectural lineages. The Mistral family achieved the strongest overall results, with several members ranking among the top performers in the benchmark. Families dominated by modern LLM or SLM architectures, such as Granite, Claude, and Gemma, consistently outperformed more traditional extractive approaches and families built on encoder-decoder architectures such as longformer encoder-decoder (LED), BART, T5, and PEGASUS. Overall performance and pairwise statistical comparison of all model families (A) Distributions of overall performance scores across all model families, with each model represented by a black dot. Families are ordered from lowest to highest-performing according to the mean performance score. Boxes span the 25th to 75th percentile, the center line denotes the mean, and whiskers extend to the minimum and maximum values. The best- and worst-performing models of each family are highlighted. (B) Games-Howell pairwise adjusted p value matrix comparing all family pairs. Each cell shows the adjusted p value for the difference in metric mean scores between two families, with white-to-gray shading indicating smaller p values. In particular, the GPT and Llama families showed lower performance scores than expected, given the competitive performance of their top models. This was primarily due to the inclusion of domain-specific small models, such as BioGPT and OpenBioLLM-Llama3-8B, which performed substantially worse than their general-purpose counterparts and therefore pulled down the family-level averages. Regarding the Games-Howell post-hoc matrix in the family comparison (Figure 5B), a combination of significant and non-significant differences between families was observed, reflecting the greater heterogeneity within some lineages. While many high-performing families differed significantly from traditional and encoder-decoder dominated families, several comparisons among modern SLMs and LLMs dominated families did not reach statistical significance. Data leakage assessment To rule out that benchmark articles were present in the models’ training data, we applied a two-step assessment: first quantifying each model’s potential overlap, defined as the share of benchmark articles published on or before its training cutoff date, and then probing the models with high overlap for actual memorization (see “data leakage analysis” in STAR Methods). Cutoff overlap results Across the evaluated models, potential overlap ranged from minimal levels of 5% for earlier models with training cutoffs prior to 2024, up to 40.2% for more recent LLMs with reported cutoff dates in early 2025. Most models showed low overlap (≤20%), indicating that the majority of benchmark articles were published after their respective training cutoff dates. A subset of eight models, including recent versions of Claude, Deepseek, and one GPT model, exhibited moderate levels of potential overlap (24–40.2%) and were further analyzed via an abstract completion probe in the next step. A complete overview of training cutoff dates and corresponding potential overlap values for all models is provided in Table S1. Abstract completion probe To assess whether high potential overlap translated into memorization effects, we conducted an abstract completion probe for the eight models showing moderate levels of potential overlap. For each model, ROUGE-L scores between generated completions and the original abstracts were compared for articles published before and after the respective training cutoff dates. Average ROUGE-L scores were nearly identical for pre-cutoff and post-cutoff articles (both at approximately 13.5%), and there were only negligible differences at the individual model level (Table 3). Statistical testing using a one-sided Mann-Whitney U test revealed no significant increase in similarity for pre-cutoff articles in any of the tested models. Effect sizes were consistently close to zero, which further indicates that there is no systematic difference between the two groups. Table 3Abstract completion probeModelCutoffPre ROUGE-LPost ROUGE-LΔ95% CIp valueEffect(r)claude-sonnet-4-20250514 (closed-source)2025–0315.215.2+0.0[-0.5, +0.4]0.598+0.01claude-opus-4-20250514 (closed-source)2025–0315.315.0+0.2[-0.3, +0.8]0.239−0.03claude-opus-4-1-20250805 (closed-source)2025–0315.214.9+0.3[-0.2, +0.8]0.161−0.04deepseek-r1:1.5b2025–0110.310.5−0.1[-0.7, +0.4]0.598+0.01deepseek-r1:7b2025–0111.711.9−0.2[-0.6, +0.3]0.524+0.00deepseek-r1:8b2025–0113.213.5−0.3[-0.7, +0.2]0.834+0.04deepseek-r1:14b2025–0113.413.4−0.1[-0.4, +0.3]0.662+0.02gpt-5-2025–08–07 (closed-source)2024–1013.713.5+0.2[-0.2, +0.7]0.399−0.01ROUGE-L scores for completions of abstracts published before (pre-cutoff) and after (post-cutoff) the respective training cutoff dates. Δ denotes the difference between both groups. Statistical significance was assessed using a one-sided Mann-Whitney U test. Abstract completion probe ROUGE-L scores for completions of abstracts published before (pre-cutoff) and after (post-cutoff) the respective training cutoff dates. Δ denotes the difference between both groups. Statistical significance was assessed using a one-sided Mann-Whitney U test. If memorization effects had occurred, substantially higher similarity scores would be expected for abstracts from pre-cutoff articles. As this was not the case, we concluded that the models were not reproducing memorized content from their training data. Expert assessment results Eight blinded domain experts rated summaries from four models along coherence, fluency, relevance, and consistency on a 1–5 scale (see “expert assessment” in STAR Methods). The results of the expert assessment are summarized in Figure 6 and Table 4. Values are reported as mean ± standard error of the mean (SEM). The two Mistral-based models (mistral-small3.2:24b and mistral-small-2506) consistently outperformed the two EDMs (bigbird-pegasus-large-pubmed and mT5_multilingual_XLSum) across all four evaluation dimensions. Expert assessment of summary quality across four dimensions Distribution of expert ratings for coherence, fluency, relevance, and consistency for each model are shown as violin plots with overlaid boxplots, where the center line denotes the median, the box spans the interquartile range, and whiskers extend to 1.5 times the interquartile range (outliers not shown). Mean ratings are given above each plot. Table 4Expert assessment results across four quality dimensionsModelCoherenceFluencyRelevanceConsistencyBigBird-Pegasus2.08 ± 0.082.23 ± 0.071.40 ± 0.051.41 ± 0.05mT5-XLSum3.20 ± 0.103.67 ± 0.091.54 ± 0.062.17 ± 0.09Mistral-Small-3.2 (24B)4.83 ± 0.034.86 ± 0.034.07 ± 0.074.56 ± 0.06Mistral-Small-25064.82 ± 0.044.90 ± 0.034.09 ± 0.074.59 ± 0.05Gwet’s AC20.7350.7940.7130.735Spearman’s ρ (mean)0.7070.7220.7990.801Values are mean ± SEM (n = 160 per model). Gwet’s AC2 (quadratic weights) and mean pairwise Spearman’s ρ measure inter-rater agreement. Overall AC2 = 0.725, overall ρ = 0.765. Expert assessment results across four quality dimensions Values are mean ± SEM (n = 160 per model). Gwet’s AC2 (quadratic weights) and mean pairwise Spearman’s ρ measure inter-rater agreement. Overall AC2 = 0.725, overall ρ = 0.765. For coherence and fluency, both Mistral models scored substantially higher than the EDMs. In coherence, they reached scores of 4.83 ± 0.03 and 4.82 ± 0.04, compared with 3.20 ± 0.10 for mT5 and 2.08 ± 0.08 for bigbird-pegasus. The same pattern was observed for fluency, where the Mistral models achieved scores of 4.86 ± 0.03 and 4.90 ± 0.03, clearly exceeding mT5 (3.67 ± 0.09) and bigbird-pegasus (2.23 ± 0.07). All pairwise comparisons between Mistral models and EDMs were statistically significant (all p < 0.001 after Bonferroni correction). The two EDMs also differed significantly from each other in both dimensions, while no significant differences were found between the two Mistral models. Differences were even more pronounced for relevance and consistency. The Mistral models again achieved high scores in relevance (4.07 ± 0.07 and 4.09 ± 0.07), while mT5 (1.54 ± 0.06) and bigbird-pegasus (1.40 ± 0.05) scored markedly lower. A similar pattern was seen for consistency, where the Mistral models reached scores of 4.56 ± 0.06 and 4.59 ± 0.05, compared to 2.17 ± 0.09 for mT5 and 1.41 ± 0.05 for bigbird-pegasus. Again, all Mistral versus EDM comparisons were statistically significant (p < 0.001). The EDMs differed significantly in consistency, but not in relevance (p = 0.7308), and no significant differences were found between the two Mistral models. Inter-rater agreement was strong on both measures. Gwet’s AC2 indicated substantial absolute agreement across all dimensions, ranging from 0.713 (relevance) to 0.794 (fluency), with an overall value of AC2 = 0.725. Mean pairwise Spearman correlations were likewise strong across all dimensions, ranging from ρ = 0.707 (coherence) to ρ = 0.801 (consistency), with an overall mean of ρ = 0.765 (all p < 0.001). These results show that evaluators consistently agreed both on the absolute quality of the summaries and on the relative ranking between models. LLM-as-a-judge evaluation A panel of three provider-independent judge models scored the same four dimensions, with the median across judges taken as the panel score (see “LLM-as-a-judge evaluation” in STAR Methods). The LLM-as-a-judge panel agreed closely with expert assessments on every dimension (Table 5). To benchmark against human reliability, we computed a leave-one-out ceiling on the same target the judges were scored against: each expert’s ratings were correlated against the mean of the remaining experts. The judge’s mean agreement (ρ = 0.87) exceeded this ceiling (ρ = 0.82), leading to the assumption that it tracked the expert consensus at least as closely as a typical individual expert. Table 5Judge-expert agreement against the human ceilingDimensionJudge ρLeave-one-out ceilingPairwiseCoherence0.880.780.71Fluency0.870.800.72Relevance0.880.840.80Consistency0.870.840.80Mean0.870.820.76LLM-as-a-judge panel vs. expert assessments (Spearman’s ρ, n = 80 summaries). Judge ρ: judge panel vs. experts, same dimension. Leave-one-out ceiling: each expert vs. the mean of the others (fair, same-target human upper bound). Pairwise: raw inter-rater agreement between two individual experts. Judge-expert agreement against the human ceiling LLM-as-a-judge panel vs. expert assessments (Spearman’s ρ, n = 80 summaries). Judge ρ: judge panel vs. experts, same dimension. Leave-one-out ceiling: each expert vs. the mean of the others (fair, same-target human upper bound). Pairwise: raw inter-rater agreement between two individual experts. The judge panel did not separate the four dimensions well. Its agreement with the matching expert dimension (diagonal ρ = 0.87) was barely higher than its agreement with the other dimensions (off-diagonal ρ = 0.85), a gap of only 0.02. In effect, the judge measured a single overall-quality signal rather than four independent constructs, so its per-dimension scores should not be over-interpreted. We questioned whether the LLM-as-a-judge panel was capable of ranking the 62 models in a comparable manner to the evaluation metrics (Table 6). The judge ranking agreed strongly with the lexical, semantic, and composite performance ranks (ρ = 0.75–0.83). The factual metrics were an exception (ρ = 0.09, n.s.), as they reward verbatim copying, so extractive systems (TextRank, frequency, T5, PEGASUS) ranked highly on factuality but performed poorly under the judge, reflecting a construct difference rather than a judge failure. Table 6Rank concordance of judge and metric rankingsMetric familyρτpLexical0.750.57<10−11Semantic0.830.64<10−15Performance0.780.61<10−12Factual0.090.060.48 (n.s.)System-level rank concordance between the LLM-as-a-judge ranks and each evaluation metric category across all n = 62 models. Judge ranks derive from the 20-publication subset, metric ranks use the full benchmark. This measures agreement between two automatic methods, not human validation. Rank concordance of judge and metric rankings System-level rank concordance between the LLM-as-a-judge ranks and each evaluation metric category across all n = 62 models. Judge ranks derive from the 20-publication subset, metric ranks use the full benchmark. This measures agreement between two automatic methods, not human validation. To analyze the tendency of judges to favor their own model family, we computed, for each judge, how far its score sat above or below the mean of the other two judges, and contrasted its own-family models with all others (Table 7). Anthropic exhibited a clear self-preference effect (+0.30 on the 1–5 scale), whereas OpenAI and Mistral showed no detectable differential effect (≤0.08 in magnitude). The direction is consistent with self-preference reported for rubric-based judging.36,37 Taking the median across the three judges damped this bias in the final scores. Table 7LLM-as-a-judge self-preferenceJudge providerSelf-preference effectAnthropic+0.30OpenAI−0.08Mistral−0.02A judge’s mean score for its own model family minus its score for other systems, each measured relative to the other two judges (1–5 scale). Positive values indicate inflation of the judge’s own family. LLM-as-a-judge self-preference A judge’s mean score for its own model family minus its score for other systems, each measured relative to the other two judges (1–5 scale). Positive values indicate inflation of the judge’s own family. Discussion Our benchmarking analysis of 62 text summarization models revealed clear performance differences. General-purpose LLMs achieved the highest summarization quality across all metric dimensions, closely followed by general-purpose SLMs, domain-specific LLMs, and reasoning-oriented LLMs. In contrast, domain-specific SLMs, encoder-decoder architectures, and traditional extractive methods performed significantly worse regarding overall performance. These results highlight the clear progression from extractive and encoder-decoder approaches toward transformer-based models, while also showing that domain-specific fine-tuning alone does not necessarily lead to improved summarization quality. Comparing models by architecture, size, and domain focus shows that, overall, LLMs perform best, likely due to their large number of parameters, which enable them to better understand the complex context typical for biomedical literature. While lightweight models like Gemma3:270M may lack the capacity to handle this complexity, SLMs remain competitive, with some models, such as mistral:7b and granite4:tiny-h, even outperforming several LLMs across the three metric dimensions. This may be attributed to the fact that smaller datasets are often more curated and of higher quality compared to the large amount of data required to train a large model.38 Interestingly, medium-sized models (e.g., many models in the Mistral family) outperform larger proprietary ones. These models seem to reach an optimal balance between parameter count and performance, where additional parameters could disrupt this equilibrium, potentially leading to over-fitting or plateaus.39 We also found that overall general-purpose models outperform both domain-specific models specialized in the biomedical domain and those specialized for text summarization, regardless of size. A possible explanation might be that domain-specific models fine-tuned on biomedical text might be better for learning and understanding the complex biomedical terminology or lexical patterns but fail in summarization tasks. On the other hand, models specifically designed for text summarization might be good at summarizing in general but fail at capturing the complex biomedical meaning. Therefore, generalist models, leveraging their broad knowledge, seem to perform better.40 Additionally, domain-specific models can “forget” the general knowledge acquired during the pre-training phase, experiencing a phenomenon called “catastrophic forgetting,” which represents an issue when the task requires both biomedical knowledge and context understanding for text summarization.41 Most reasoning-oriented models ranked in the middle, indicating moderate performance. This likely reflects their multi-step reasoning nature, which can be advantageous for tasks that require breaking down problems into sequential steps, such as mathematical operations or computer programming, but seems less suited for text summarization, which requires semantic compression and factual grounding.42 Traditional models ranked highest on the factual dimension, likely because their extractive approach generates summaries closely aligned with the source abstracts, which factuality metrics reward by design.43 The expert assessment supported the findings of the automated benchmark. The two highest-ranked models consistently outperformed the two lowest-ranked models across all four evaluation dimensions. This alignment suggests that the automatic metrics capture relevant aspects of summary quality and are broadly consistent with human judgments. The results of this benchmark provide useful guidance for selecting summarization models in biomedical and scientific settings. The strong performance of general-purpose LMs indicates that broad, diverse pretraining is often more advantageous than narrow domain adaptation when dealing with unseen scientific content. Another key consideration is the trade-off between output quality and processing efficiency. While LLMs achieved the highest overall scores, SLMs deliver competitive results at substantially lower computational cost,44 which makes them especially attractive for large-scale or resource-constrained applications. Choosing between large and small models therefore depends not only on desired output quality but also on the intended scale of summarization. Overall, the findings suggest that general-purpose LMs currently offer the most reliable and practical choice for biomedical text summarization. Their consistent performance across evaluation criteria demonstrates that broad generalization outweighs the marginal gains from more narrowly specialized or fine-tuned approaches, many of which are not primarily optimized for text summarization. This benchmark provides a comprehensive evaluation of 62 text summarization methods on a dataset of 1,000 biomedical abstracts paired with standardized highlights sections. Across all lexical overlap, semantic similarity, and factual consistency metrics, general-purpose LLMs were the best-performing models. Their ability to incorporate broad pretraining with strong semantic understanding enabled them to outperform reasoning-oriented models, domain-specific models, and encoder-decoder architectures. Especially medium-sized models achieved the highest overall rankings, which indicate that increased scale beyond this range does not automatically translate to improved summarization quality for biomedical text. SLMs remain competitive for settings where computational resources are limited as they offer balance between efficiency and output quality. In contrast, domain-specific models did not show a systematic advantage, showing that narrow fine-tuning alone is insufficient to match the generalization of broadly trained LLMs for this task. Overall, this benchmark provides a systematic reference point for selecting summarization models in biomedical research. By assessing a broad spectrum of architectures under a unified framework, we underline the strengths and limitations of currently available systems and highlight the superiority of general-purpose LLMs for scientific text summarization. These insights can support researchers and practitioners in choosing models that balance output quality, computational cost, and practical usability in biomedical workflows. Limitations of the study Even though our results are based on a robust evaluation framework, several limitations of our study warrant discussion. Model access methods varied across the evaluation due to differing application programming interface (API) capabilities and requirements. Hugging Face models were accessed through their supported interfaces: the pipeline API with task = “summarization” where available, or chat/completion formats for models that did not support the pipeline approach. Ollama models required use of the generate endpoint with merged prompts, while OpenAI, Anthropic, and Mistral models each mandated their respective provider-specific APIs (responses.create, messages.create, and chat.complete) with distinct message structures. We applied hyperparameter normalization where possible, though API-level constraints prevented full standardization. For example, GPT-5 does not support temperature control, instead offering only reasoning-specific parameters. Additionally, proprietary middleware layers may transform requests and responses in undocumented ways, potentially affecting outputs independently of the underlying model architectures. These necessary methodological variations warrant consideration when interpreting performance differences across models. Furthermore, all models received an identical summarization prompt. While this ensures a controlled comparison, it may disadvantage models whose instruction tuning differs from the chosen phrasing. Per-model prompt optimization (e.g., via DSPy45,46) could yield different absolute scores and rankings. Our results therefore reflect performance under a fixed instruction, not each model’s attainable ceiling. Reference-based evaluation cannot fully capture every valid way a good summary might be written, as summarization is largely a subjective exercise. To account for this, we tried to capture summary quality along diverse, complementary dimensions by combining lexical, semantic, and factual metrics, and as a further assessment included a blinded expert evaluation as an independent check on the automatic scores. Nonetheless, since automatic metrics and human ratings have not fully aligned in other settings,5,6 we cannot formally guarantee that our metrics capture the full space of valid summaries, which remains an important direction for future work. Regarding our LLM-as-a-judge results, it is notable that the three judges are not independent: correlated errors mean the panel yields far fewer effective votes than its size suggests, and a strong single judge can match it.47 We therefore treat the panel median as bias-control, not a route to higher accuracy, and its edge over the human leave-one-out ceiling reflects median smoothing rather than above-human reliability. Furthermore, the summary-level agreement pools models of very different quality. Within-model agreement is much lower, so the judge ranks models reliably but separates similar-quality summaries weakly. Finally, the system-level agreement compares two automatic methods, not validation against humans (infeasible with only four expert-rated models). Another limitation of this benchmarking study is that we focused on a single summarization task: generating concise summaries from biomedical abstracts. This setup provides a clear and well-defined evaluation framework, but the findings may not fully extend to other forms of scientific or biomedical summarization, including full-text articles, clinical trial data, or lay-oriented summaries. Given the rapid evolution of LLMs, these results just capture a specific snapshot in time and may change as newer architectures and models become available. A related consideration is our choice of reference summary. We used author-written highlights as the reference against which summaries were evaluated, but condensing an abstract (let alone a full publication) into a few bullet points is inherently subjective, and highlights inevitably emphasize some findings over others. We nonetheless regard them as the most suitable reference available, since they are author-generated, standardized across both publishers, and well-matched to our target of concise summaries within a limited length. Finally, although we deliberately drew articles from 20 journals spanning a broad range of biomedical and translational fields, our corpus inevitably covers only part of this very diverse domain. We cannot rule out that summarization performance, and the resulting model rankings, would differ for subfields not well presented in our selection, leaving open how well these results generalize across the biomedical domain. Resource availability Lead contact Further information and requests for resources should be directed to and will be fulfilled by the lead contact, Matthias Ley (matthias.ley@delta4.ai). Materials availability This study did not generate new unique reagents. Data and code availability •All data reported in this paper are publicly available as of the date of publication under a CC-BY 4.0 licence with no access restrictions at Zenodo: https://doi.org/10.5281/zenodo.21840194. The deposit comprises the gold-standard dataset of 1,000 titles, abstracts, and author-written highlights; all generated summaries together with per-paper and per-model metric scores across the lexical, semantic, and factual dimensions; the raw ratings of all eight expert evaluators; the LLM-as-a-judge panel scores; the data leakage analyses; and the aggregated ranking, concordance, and agreement tables reported in the figures and tables. All data reported in this paper are publicly available as of the date of publication under a CC-BY 4.0 licence with no access restrictions at Zenodo: https://doi.org/10.5281/zenodo.21840194. The deposit comprises the gold-standard dataset of 1,000 titles, abstracts, and author-written highlights; all generated summaries together with per-paper and per-model metric scores across the lexical, semantic, and factual dimensions; the raw ratings of all eight expert evaluators; the LLM-as-a-judge panel scores; the data leakage analyses; and the aggregated ranking, concordance, and agreement tables reported in the figures and tables. •All original code is deposited alongside the data and publicly available as of the date of publication under an MIT licence at Zenodo: https://doi.org/10.5281/zenodo.21840194. Development versions are maintained at https://github.com/Delta4AI/LLMTextSummarizationBenchmark. All original code is deposited alongside the data and publicly available as of the date of publication under an MIT licence at Zenodo: https://doi.org/10.5281/zenodo.21840194. Development versions are maintained at https://github.com/Delta4AI/LLMTextSummarizationBenchmark. •Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request. Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request. Acknowledgments E.B. is supported by a grant from the European Union’s Horizon Europe Marie Skłodowska-Curie Actions Doctoral Networks program project PICKED (HORIZON–MSCA–2023–DN-01, grant no. 101168626). L.G. is supported by a grant from the European Union’s Horizon Europe Marie Skłodowska-Curie Actions Doctoral Networks Industrial Doctorates program project PROMOTE (HORIZON–MSCA–2023–DN-01, grant no. 101169245). P.P. and M.L. are members of and would like to cordially thank the COST Action PerMediK, CA21165, supported by COST (European Cooperation in Science and Technology). The authors gratefully acknowledge Dorota Wojenska for her support in optimizing the overview workflow figure (Figure 1). Author contributions Conceptualization, M.L., P.P., E.B., and F.B.; methodology, M.L., P.P., E.B., and F.B.; software, M.L.; formal analysis, F.B., E.B., P.P., and M.L.; validation, E.B., F.B., L.F., P.P., M.L., L.G., K.K.-I., S.M.W., P.A., and K.K.; writing – original draft, F.B., E.B., M.L., and P.P.; writing – review and editing, E.B., F.B., P.P., M.L., K.K., L.G., L.F., K.K.-I., P.A., and S.M.W.; visualization, F.B. and E.B.; supervision, M.L. and P.P. All authors have read and agreed to the published version of the manuscript. Declaration of interests K.K. is co-founder and shareholder of Delta4 GmbH (Vienna, Austria). F.B., E.B., L.F., L.G., K.K.-I., S.M.W., P.A., P.P., and M.L. are employees of Delta4 GmbH (Vienna, Austria). Declaration of generative AI and AI-assisted technologies in the writing process During the preparation of this work, the author(s) used Anthropic (Claude Opus 4.5) and OpenAI (GPT-5) to enhance textual clarity and readability without introducing new hypotheses or data. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the published article. STAR★Methods Key resources table REAGENT or RESOURCESOURCEIDENTIFIERDeposited dataGold-standard dataset: 1,000 biomedical titles, abstracts, and author-written highlightsThis paperZenodo: https://doi.org/10.5281/zenodo.21840194Generated summaries and per-paper, per-model metric scores for all 62 summarization methodsThis paperZenodo: https://doi.org/10.5281/zenodo.21840194Raw expert ratings from all eight evaluators across four quality dimensionsThis paperZenodo: https://doi.org/10.5281/zenodo.21840194LLM-as-a-judge panel scores and data leakage analysesThis paperZenodo: https://doi.org/10.5281/zenodo.21840194Software and algorithmsOriginal benchmarking and analysis code (this study)This paperZenodo: https://doi.org/10.5281/zenodo.21840194; https://github.com/Delta4AI/LLMTextSummarizationBenchmarkPython 3.12Python Software Foundationhttps://www.python.org; RRID:SCR_008394pandas v2.3.3McKinney48https://pandas.pydata.org; RRID:SCR_018214scikit-learn v1.8.0Pedregosa et al.49https://scikit-learn.org; RRID:SCR_002577NetworkX v3.6.1Hagberg et al.50https://networkx.org; RRID:SCR_016864NLTK v3.9.2Bird et al.51https://www.nltk.org; RRID:SCR_025101rouge-score v0.1.2Lin52https://pypi.org/project/rouge-scorebert-score v0.3.13Zhang et al.53https://github.com/Tiiiger/bert_scoresentence-transformers v5.2.0Reimers and Gurevych54https://www.sbert.netAlignScore v0.1.6Zha et al.55https://github.com/yuh-zha/AlignScoresummac v0.0.4 (SummaC-ZS)Laban et al.56https://github.com/tingofurro/summacMiniCheck v0.1.0Tang et al.57https://github.com/Liyan06/MiniChecktransformers v4.50.3Wolf et al.58https://huggingface.co/docs/transformers; RRID:SCR_027381SciPy v1.17.0SciPy developershttps://scipy.org; RRID:SCR_008058pingouin v0.6.1Pingouin developershttps://pingouin-stats.org; RRID:SCR_022261OllamaOllama Inc.https://ollama.comR v4.6.1R Core Teamhttps://www.r-project.org; RRID:SCR_001905ggplot2 (R)tidyversehttps://ggplot2.tidyverse.org; RRID:SCR_014601ComplexHeatmap (R)Bioconductorhttps://bioconductor.org/packages/ComplexHeatmap; RRID:SCR_017270circlize (R)CRANhttps://CRAN.R-project.org/package=circlize; RRID:SCR_002141roberta-large (contextual encoder for BERTScore)Liu et al.59Hugging Face: roberta-largemicrosoft/deberta-xlarge-mnli (contextual encoder for BERTScore)He et al.60Hugging Face: microsoft/deberta-xlarge-mnliall-mpnet-base-v2 (sentence embedding model)Song et al.61Hugging Face: sentence-transformers/all-mpnet-base-v2AlignScore-large checkpoint (roberta-large backbone)Zha et al.55Hugging Face: yzha/AlignScore, file AlignScore-large.ckpttals/albert-xlarge-vitaminc-mnli (NLI model behind SummaC-ZS)Laban et al.56Hugging Face: tals/albert-xlarge-vitaminc-mnliMiniCheck-Flan-T5-Large (MiniCheck-FT5)Tang et al.57Hugging Face: lytang/MiniCheck-Flan-T5-LargeBespoke-MiniCheck-7B, quantized GGUF build (MiniCheck-7B)Tang et al.57Ollama: bespoke-minicheckclaude-sonnet-4-6 (LLM-as-a-judge panel member)AnthropicAnthropic API: claude-sonnet-4-6gpt-5.4–2026–03–05 (LLM-as-a-judge panel member)OpenAIOpenAI API: gpt-5.4–2026–03–05mistral-medium-2604 (LLM-as-a-judge panel member)Mistral AIMistral AI API: mistral-medium-2604frequency (summarization method evaluated in this study; traditional extractive method)This paperZenodo: https://doi.org/10.5281/zenodo.21840194textrank (summarization method evaluated in this study; traditional extractive method)This paperZenodo: https://doi.org/10.5281/zenodo.21840194facebook/bart-base (summarization method evaluated in this study; general-purpose encoder-decoder model)Hugging FaceHugging Face: facebook/bart-basegoogle-t5/t5-base (summarization method evaluated in this study; general-purpose encoder-decoder model)Hugging FaceHugging Face: google-t5/t5-basegoogle-t5/t5-large (summarization method evaluated in this study; general-purpose encoder-decoder model)Hugging FaceHugging Face: google-t5/t5-largegoogle/pegasus-large (summarization method evaluated in this study; general-purpose encoder-decoder model)Hugging FaceHugging Face: google/pegasus-largefacebook/bart-large-cnn (summarization method evaluated in this study; domain-specific encoder-decoder model)Hugging FaceHugging Face: facebook/bart-large-cnngoogle/pegasus-xsum (summarization method evaluated in this study; domain-specific encoder-decoder model)Hugging FaceHugging Face: google/pegasus-xsumgoogle/pegasus-cnn_dailymail (summarization method evaluated in this study; domain-specific encoder-decoder model)Hugging FaceHugging Face: google/pegasus-cnn_dailymailgoogle/pegasus-pubmed (summarization method evaluated in this study; domain-specific encoder-decoder model)Hugging FaceHugging Face: google/pegasus-pubmedgoogle/bigbird-pegasus-large-pubmed (summarization method evaluated in this study; domain-specific encoder-decoder model)Hugging FaceHugging Face: google/bigbird-pegasus-large-pubmedcsebuetnlp/mT5_multilingual_XLSum (summarization method evaluated in this study; domain-specific encoder-decoder model)Hugging FaceHugging Face: csebuetnlp/mT5_multilingual_XLSumAlgorithmicResearchGroup/led_large_16384_arxiv_summarization (summarization method evaluated in this study; domain-specific encoder-decoder model)Hugging FaceHugging Face: AlgorithmicResearchGroup/led_large_16384_arxiv_summarizationgemma3:270M (summarization method evaluated in this study; general-purpose small language model)Ollama model registryOllama: gemma3:270Mgemma3:1b (summarization method evaluated in this study; general-purpose small language model)Ollama model registryOllama: gemma3:1bgemma3:4b (summarization method evaluated in this study; general-purpose small language model)Ollama model registryOllama: gemma3:4bPetrosStav/gemma3-tools:4b (summarization method evaluated in this study; general-purpose small language model)Ollama model registryOllama: PetrosStav/gemma3-tools:4bgranite3.3:2b (summarization method evaluated in this study; general-purpose small language model)Ollama model registryOllama: granite3.3:2bgranite3.3:8b (summarization method evaluated in this study; general-purpose small language model)Ollama model registryOllama: granite3.3:8bgranite4:tiny-h (summarization method evaluated in this study; general-purpose small language model)Ollama model registryOllama: granite4:tiny-hgranite4:small-h (summarization method evaluated in this study; general-purpose small language model)Ollama model registryOllama: granite4:small-hgranite4:micro (summarization method evaluated in this study; general-purpose small language model)Ollama model registryOllama: granite4:microgranite4:micro-h (summarization method evaluated in this study; general-purpose small language model)Ollama model registryOllama: granite4:micro-hllama3.1:8b (summarization method evaluated in this study; general-purpose small language model)Ollama model registryOllama: llama3.1:8bllama3.2:1b (summarization method evaluated in this study; general-purpose small language model)Ollama model registryOllama: llama3.2:1bllama3.2:3b (summarization method evaluated in this study; general-purpose small language model)Ollama model registryOllama: llama3.2:3bmistral:7b (summarization method evaluated in this study; general-purpose small language model)Ollama model registryOllama: mistral:7bphi3:3.8b (summarization method evaluated in this study; general-purpose small language model)Ollama model registryOllama: phi3:3.8bgpt-4o-mini (summarization method evaluated in this study; general-purpose language model)OpenAIOpenAI API: gpt-4o-minigpt-4.1-mini (summarization method evaluated in this study; general-purpose language model)OpenAIOpenAI API: gpt-4.1-miniswiss-ai/Apertus-8B-Instruct-2509 (summarization method evaluated in this study; general-purpose small language model)Hugging FaceHugging Face: swiss-ai/Apertus-8B-Instruct-2509gemma3:12b (summarization method evaluated in this study; general-purpose large language model)Ollama model registryOllama: gemma3:12bmistral-nemo:12b (summarization method evaluated in this study; general-purpose large language model)Ollama model registryOllama: mistral-nemo:12bmistral-small3.2:24b (summarization method evaluated in this study; general-purpose large language model)Ollama model registryOllama: mistral-small3.2:24bmistral-medium-2505 (summarization method evaluated in this study; general-purpose large language model)Mistral AIMistral AI API: mistral-medium-2505mistral-large-2411 (summarization method evaluated in this study; general-purpose large language model)Mistral AIMistral AI API: mistral-large-2411phi4:14b (summarization method evaluated in this study; general-purpose large language model)Ollama model registryOllama: phi4:14bgpt-3.5-turbo (summarization method evaluated in this study; general-purpose large language model)OpenAIOpenAI API: gpt-3.5-turbogpt-4o (summarization method evaluated in this study; general-purpose large language model)OpenAIOpenAI API: gpt-4ogpt-4.1 (summarization method evaluated in this study; general-purpose large language model)OpenAIOpenAI API: gpt-4.1claude-3-5-haiku-20241022 (summarization method evaluated in this study; general-purpose large language model)AnthropicAnthropic API: claude-3-5-haiku-20241022deepseek-r1:1.5b (summarization method evaluated in this study; reasoning-oriented small language model)Ollama model registryOllama: deepseek-r1:1.5bdeepseek-r1:7b (summarization method evaluated in this study; reasoning-oriented small language model)Ollama model registryOllama: deepseek-r1:7bdeepseek-r1:8b (summarization method evaluated in this study; reasoning-oriented small language model)Ollama model registryOllama: deepseek-r1:8bqwen3:4b (summarization method evaluated in this study; reasoning-oriented small language model)Ollama model registryOllama: qwen3:4bqwen3:8b (summarization method evaluated in this study; reasoning-oriented small language model)Ollama model registryOllama: qwen3:8bdeepseek-r1:14b (summarization method evaluated in this study; reasoning-oriented large language model)Ollama model registryOllama: deepseek-r1:14bgpt-oss:20b (summarization method evaluated in this study; reasoning-oriented large language model)Ollama model registryOllama: gpt-oss:20bgpt-5-nano-2025–08–07 (summarization method evaluated in this study; reasoning-oriented large language model)OpenAIOpenAI API: gpt-5-nano-2025–08–07gpt-5-mini-2025–08–07 (summarization method evaluated in this study; reasoning-oriented large language model)OpenAIOpenAI API: gpt-5-mini-2025–08–07gpt-5-2025–08–07 (summarization method evaluated in this study; reasoning-oriented large language model)OpenAIOpenAI API: gpt-5-2025–08–07claude-sonnet-4-20250514 (summarization method evaluated in this study; reasoning-oriented large language model)AnthropicAnthropic API: claude-sonnet-4-20250514claude-opus-4-20250514 (summarization method evaluated in this study; reasoning-oriented large language model)AnthropicAnthropic API: claude-opus-4-20250514claude-opus-4-1-20250805 (summarization method evaluated in this study; reasoning-oriented large language model)AnthropicAnthropic API: claude-opus-4-1-20250805magistral-medium-2509 (summarization method evaluated in this study; reasoning-oriented large language model)Mistral AIMistral AI API: magistral-medium-2509microsoft/biogpt (summarization method evaluated in this study; domain-specific small language model)Hugging FaceHugging Face: microsoft/biogptmedllama2:7b (summarization method evaluated in this study; domain-specific small language model)Ollama model registryOllama: medllama2:7baaditya/OpenBioLLM-Llama3-8B (summarization method evaluated in this study; domain-specific small language model)Hugging FaceHugging Face: aaditya/OpenBioLLM-Llama3-8BBioMistral/BioMistral-7B (summarization method evaluated in this study; domain-specific small language model)Hugging FaceHugging Face: BioMistral/BioMistral-7BUni-SMART/SciLitLLM1.5-7B (summarization method evaluated in this study; domain-specific small language model)Hugging FaceHugging Face: Uni-SMART/SciLitLLM1.5-7BUni-SMART/SciLitLLM1.5-14B (summarization method evaluated in this study; domain-specific large language model)Hugging FaceHugging Face: Uni-SMART/SciLitLLM1.5-14B Method details Study design This study was designed as a systematic evaluation and comparative benchmark of ATS methods for biomedical literature and is reported in accordance with the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD)-LLM reporting guideline.62 Summarization quality was assessed quantitatively using a suite of lexical, semantic, and factual metrics, normalized and aggregated into an Overall Performance Score for ranking, and complemented by a blinded expert assessment of a subset of models. Differences across model categories and families were then compared statistically. The evaluation metrics and statistical tests were chosen during the analysis rather than pre-specified in a study protocol. Gold-standard dataset We generated a gold-standard benchmarking dataset comprising 1,000 biomedical peer-reviewed articles from ScienceDirect and Cell Press, as these publishers provide standardized highlights sections, which are concise bullet points capturing an article’s main findings.63,64 The concatenated highlights served as reference summaries in our evaluation, while the corresponding titles and abstracts were used as input texts for the summarization task. While summarization is inherently subjective, author-generated highlights represent the most credible and standardized source, ensuring the captured content reflects the study’s intended key messages. Articles were collected systematically across a variety of journals from the two publishers to ensure coverage of different fields within molecular sciences, including among others drug discovery, genomics, proteomics, biotechnology, and biochemistry. We considered only English-language, research-focused content that provided an author-written highlights section together with a DOI, title, and abstract. From each of the 20 journals, we then selected the 50 most recent eligible articles available at the time of dataset construction, yielding 1,000 papers (Table 1) with publication dates ranging from October 2015 to November 2025. Because selection was performed per journal, journals with fewer highlights-bearing articles contributed some older publications, though 95% of the corpus was published from 2024 onward. Selecting the most recent articles also minimizes overlap with the models’ pretraining data and thus the risk of data leakage. This setup provided standardized pairs of abstracts and reference summaries subsequently used in our systematic evaluation and benchmarking of ATS methods. Summarization methods We evaluated 62 models for text summarization, ranging from simple frequency-based algorithms to SLMs and LLMs. These models were classified into five categories, as listed in Table 2. We obtained the pre-trained EDMs through the Hugging Face library, selecting architectures widely used for abstractive summarization to ensure established neural approaches are covered in our benchmark. Additionally, we evaluated a range of SLMs and LLMs, defining SLMs as models with fewer than 10 billion parameters.65 Both SLMs and LLMs with advanced reasoning capabilities were categorized as “reasoning-oriented” to investigate how multi-step problem solving affects summarization performance. Similarly, models fine-tuned on scientific/biomedical data or specifically tailored for text summarization were classified as “domain-specific” to assess the impact of domain adaptation. Overall, this selection covers various model sizes and release dates, ensuring a representative mix of both, widely adopted and recent, architectures. Exceptionally large models, such as LLaMA 3.1 405B, were excluded to keep the benchmark reproducible on single-GPU setups typical of biomedical research groups. Each of the 62 models was tasked with generating summaries for the 1,000 abstracts in the dataset, resulting in a total of 62,000 generated summaries for evaluation. Prompt design We used an identical prompt across all prompt-based models to ensure compatibility, asking for concise summaries focused on the main findings of each publication while excluding unnecessary background or methodological details. The prompt was developed iteratively on a small number of sample abstracts and then frozen. Each model received the publication title and abstract as an input and was asked to produce an output of 15–100 words. A post-hoc check confirmed that models adhered well to this length constraint, with the large majority of summaries falling within the target range across methods. A full per-method breakdown is provided in Figure S1. If the abstract did not contain any substantive results or conclusions, the model was instructed to return the predefined token INSUFFICIENT_FINDINGS. This gave models an explicit exit path for cases where an abstract contained no actual findings to summarize, preventing them from fabricating content simply to fill the response. The token was scored identically to any other generated summary: no metric treats it as a special case, so a model invoking this option is evaluated against the reference summary and source abstract like any other output, rather than being excluded from scoring or penalized separately. The exact prompt used for all models was as follows: Summarize the provided publication (title and abstract) in 15-100 words. Key requirements: •Identify main findings, results, or contributions Identify main findings, results, or contributions •Preserve essential context and nuance Preserve essential context and nuance •Exclude background, methods unless crucial to conclusions Exclude background, methods unless crucial to conclusions •Write concisely and objectively Write concisely and objectively •Avoid repetition and unnecessary qualifiers If no substantial findings exist, respond: ‘INSUFFICIENT_FINDINGS’ Avoid repetition and unnecessary qualifiers If no substantial findings exist, respond: ‘INSUFFICIENT_FINDINGS’ Evaluation metrics As there is no single metric that can fully reflect summary quality, especially in the biomedical field where both coverage of key information and factual correctness are critical, we employed both surface-level metrics based on lexical overlap, referred to as lexical-based metrics, and embedding-level metrics. The latter includes metrics based on semantic similarity, denoted as semantic-based metrics, as well as metrics that evaluate factual consistency. Surface-level metrics Surface-level metrics compare the generated summaries with the reference summaries mainly at the word or phrase level. While they do not capture meaning beyond surface overlap, they remain common metrics in summarization research and provide a straightforward foundation for evaluation. We used three ROUGE variants (ROUGE-1, ROUGE-2, ROUGE-L),52 BLEU,66 and METEOR.67 ROUGE-1 and ROUGE-2 measure how many unigrams (single words) or bigrams (word pairs) from the reference appear in the generated output, while ROUGE-L identifies the longest sequence of words shared between the two. BLEU calculates how many n-grams in the output also occur in the reference, emphasizing precision over recall and applying a brevity penalty to counteract the tendency toward overly short summaries. METEOR extends n-gram matching by considering word stems and synonyms, making it more robust to wording variations. Together, these metrics offer a simple but transparent point of reference. Embedding-based metrics To capture similarity beyond surface-level word overlap, we included a set of embedding-based metrics built on pre-trained transformer models. These methods generate vector representations of text, allowing them to capture semantic similarity rather than just word overlap. We measured semantic similarity using BERT-score,53 which compares two texts by matching their contextual token embeddings and averaging the resulting cosine similarities. Because BERT-score needs a pre-trained encoder to produce these embeddings, we ran it with two well-established choices: RoBERTa (roberta-large)59 and DeBERTa (microsoft/deberta-xlarge-mnli).60 This lets the metrics recognize when two summaries express the same content even if the wording differs. We further included all-mpnet-base-v2,61 a transformer model fine-tuned for sentence similarity. Unlike BERT-score, which matches summaries token by token, masked and permuted pre-training (MPNet) was trained with a focus on alignment at the sentence-level. This characteristic makes it a useful complement to the other metrics, as it is particularly sensitive to whether the overall meaning of a reference summary is preserved in the system output. Finally, to evaluate factual consistency, we included metrics designed to assess whether statements in a generated summary are supported by the source text. These included AlignScore (AlignScore-large checkpoint),55 SummaC (tals/albert-xlarge-vitaminc-mnli),56 and two variants of MiniCheck (lytang/MiniCheck-Flan-T5-Large and Bespoke-MiniCheck-7B).57 In contrast to the other metrics, these approaches compare the generated output directly to the source text itself (i.e., the publication title and abstract) rather than the reference summary (i.e., the highlights section), as factual accuracy can only be assessed relative to the original input text. This addition ensures that our evaluation captures hallucinations and unsupported statements that might otherwise be overlooked. Data leakage analysis To assess whether benchmark articles were part of the models’ pretraining data, we conducted a data leakage analysis by comparing their publication dates against reported model training cutoff dates. If an article was published before a model’s training cutoff date, it may have been included in the model’s internal knowledge, potentially giving the model access to information beyond the input provided (title and abstract). Publication dates for all benchmark articles were retrieved using the Crossref API. For each model, the reported training cutoff date was collected from official model documentation (i.e., provider-released model descriptions) or, when unavailable, from community-curated sources.68 In total, 21 cutoff dates were confirmed from primary sources, 9 were obtained from community sources, 17 were estimated based on publicly available information (e.g., Hugging Face documentation or associated publications), and 13 (primarily from newer and less-well documented models) remained unknown. We then compared article publication dates with model cutoff dates to determine the proportion of benchmark articles published on or before each model’s training cutoff. This proportion represents the theoretical maximum share of benchmark articles that could have been present in the model’s training data. To further investigate models with high potential overlap, we performed an abstract completion probe. Notably, all models included in this abstract completion probe had cutoff dates confirmed from either official or community sources. In this test, models were prompted in completion mode with the first sentence of an article abstract and asked to generate the continuation. For each model, this procedure was done separately for abstracts published before and after the model’s training cutoff date. The similarity between generated summaries and the original abstracts were measured using ROUGE-L scores, as ROUGE-L is based on the longest common subsequence (LCS), which makes it sensitive to long in-order token overlaps. If models had memorized benchmark articles during training, completions for pre-cutoff abstracts would be expected to show significantly higher similarity scores compared to post-cutoff abstracts. Differences in ROUGE-L similarity between pre- and post-cutoff articles were assessed using one-sided Mann-Whitney U tests. This two-stage analysis provides a practical estimation of whether benchmark articles have influenced model outputs through previous exposure during pretraining. Expert assessment To assess whether automatic evaluation metrics align with human judgments, we conducted an expert evaluation of the generated summaries. Eight evaluators with professional backgrounds in biomedical research and drug discovery participated. A web-based questionnaire was developed and evaluators were asked to rate generated summaries using the four quality dimensions defined in the SummEval framework,69 a widely adopted standard for summarization evaluation that has also been used in recent LLM-based evaluation frameworks such as G-Eval.74 The evaluated dimensions were: coherence (logical structure and organization of the summary), fluency (grammatical correctness and readability), relevance (coverage of important information), and consistency (factual alignment with the reference). Each dimension was rated on a five-point Likert scale ranging from 1 to 5. The assessment was conducted on summaries generated by the two highest-ranked and the two lowest-ranked models identified in the automated benchmark: mistral-small3.2:24b, mistral-small-2506, google/bigbird-pegasus-large-pubmed, and mT5_multilingual_XLSum. For evaluation, one article was randomly selected from each journal category, resulting in 20 biomedical articles and 80 assessments per expert. Each evaluator assessed the same set of summaries, but to reduce potential ordering effects, the presentation order was randomized. Model identities were hidden to ensure an unbiased, blinded evaluation. For each paper, the evaluation followed a two-step process. In the first step, evaluators were shown the author-written highlights along with the generated summary and asked to rate coherence, fluency, and relevance. The author-written highlights served as a reference for the key findings, allowing evaluators to assess whether the generated summaries captured the most important information and enabling comparison of human judgments with lexical and semantic evaluation metrics. In the second step, evaluators were shown the title and abstract of the original article alongside the generated summary to assess factual consistency and detect potential hallucinations. This mirrors the input used for the automatic factuality metrics, enabling direct comparison. Inter-rater agreement was quantified using Gwet’s AC2 coefficient71 with quadratic weights, chosen for its robustness to skewed score distributions and capturing absolute agreement, together with the mean pairwise Spearman’s ρ, capturing agreement in relative ranking. Pairwise differences in ratings between models were tested using Mann-Whitney U tests with Bonferroni correction for multiple comparisons. LLM-as-a-judge evaluation We complement the evaluation metrics with an LLM-as-a-judge approach72 using three models from distinct providers (Anthropic’s claude-sonnet-4-6, OpenAI’s gpt-5.4-2026-03-05 and Mistral’s mistral-medium-2604) in a panel-of-evaluators design73 to reduce intra-model bias relative to any single judge. We deliberately used mid-tier models while limiting reasoning efforts, rather than using each provider’s flagship model, to prevent overthinking. Each judge emitted a brief rationale before scoring (explain-then-rate, which aids alignment with human ratings74), rating each of the four rubric dimensions on a 1 to 5 scale via forced structured outputs and receiving the same evidence as the human experts (title, abstract, reference highlights, and the summary). The panel score for each summary-dimension is the median across the three judges. This test was introduced as a bias-control and robustness measure rather than as a source of additional independent signal (see Limitations). We validated the judge panel against the 80 expert assessments using summary-level Spearman correlation and then applied it to all 62 models on the same 20 publications (one per journal), rather than the full 1,000-document benchmark. All correlations use scipy.stats.spearmanr. Benchmarking framework The benchmark framework, including all downstream statistical tests and supplementary analyses, was implemented in Python 3.12. Gold standard data were retrieved from open-access articles published by ScienceDirect and Cell Press through manual extraction of titles, abstracts, and highlights sections, along with metadata including publication URLs, identifiers, section types, and article types where available. Extraction followed an identical procedure for both publishers, and each record was checked for completeness of the required fields. All data were stored in machine-readable JSON format. The framework was implemented using the Python standard library supplemented by several specialized packages: pandas48 for data import and export, scikit-learn49 for computing cosine similarities of embeddings and TF-IDF vectors, networkx50 for graph construction and the PageRank algorithm.75 Additional evaluation metrics were computed using NLTK51 for METEOR and BLEU scores, ROUGE-score, BERT-score,53 AlignScore,55 the summac package implementing the SummaC-ZS model,56 MiniCheck,57 and sentence-transformers54 with the all-mpnet-base-v2 model.61 Statistical analyses used pingouin for Welch’s analysis of variance (ANOVA) and Games-Howell tests, SciPy for Spearman correlations and Mann-Whitney U tests, and a custom implementation of Gwet’s AC2 available in the project repository. Figures were generated in R (v4.6.1) using ggplot2, ComplexHeatmap, and circlize. Communication with proprietary closed-source LLMs was facilitated through the official Python APIs provided by Anthropic, Mistral AI, and OpenAI. Local LLM execution was performed on a workstation equipped with a NVIDIA RTX A4000 GPU (16GB VRAM) running Ollama as a backend service, accessed through its Python API along with the transformers library.58 All LLMs were configured with a temperature parameter of 0.2 to optimize reproducibility while avoiding completely deterministic outputs. For the latest generation of OpenAI models featuring adaptive reasoning capabilities, the configuration was set to text.verbosity = low and reasoning.effort = minimal. The full set of parameters and prompts are documented in the config.py file in the GitHub repository. Quantification and statistical analysis Overall performance score To comprehensively assess the performance of each model on the summarization task, we employed a multi-metric framework covering three dimensions: lexical (n = 5), semantic (n = 3), and factual (n = 4). To prevent the impact of dimension imbalance, we computed an average score for each dimension as follows: To examine how the different evaluation metrics relate to each other, we computed pairwise Spearman correlation coefficients across all models (Figure 2). Averaging metrics within each dimension ensured that highly correlated measures such as ROUGE-1, ROUGE-2, and ROUGE-L do not dominate the overall evaluation. As the different metric dimensions operate on different value ranges, we applied z-score normalization to the dimension-wise averages across all models, ensuring that lexical, semantic, and factual metrics contribute comparably to the final evaluation. The Overall Performance Score (OPS) was computed as the mean of the normalized dimension scores: We finally ranked models by overall performance, resulting in the performance rank, with lower ranks indicating better performance. To examine which model performed best in each dimension (lexical, semantic, factual), we additionally computed per-dimension ranks, reported alongside the overall rank. Statistical analysis Differences in overall performance between model categories and families were assessed using Welch’s ANOVA, followed by pairwise post-hoc comparisons with Games-Howell tests. This procedure is well suited for groups with unequal sample sizes and heterogeneous variances, conditions that apply to our benchmark due to the heterogeneity of models within each category and family. Adjusted p-values were used to determine the statistical significance of between-group differences. Footnotes Supplemental information can be found online at https://doi.org/10.1016/j.isci.2026.117438. Supplemental information Document S1. Figures S1 and S2 and Table S1 References Associated Data Supplementary Materials Document S1. Figures S1 and S2 and Table S1 Data Availability Statement •All data reported in this paper are publicly available as of the date of publication under a CC-BY 4.0 licence with no access restrictions at Zenodo: https://doi.org/10.5281/zenodo.21840194. The deposit comprises the gold-standard dataset of 1,000 titles, abstracts, and author-written highlights; all generated summaries together with per-paper and per-model metric scores across the lexical, semantic, and factual dimensions; the raw ratings of all eight expert evaluators; the LLM-as-a-judge panel scores; the data leakage analyses; and the aggregated ranking, concordance, and agreement tables reported in the figures and tables. All data reported in this paper are publicly available as of the date of publication under a CC-BY 4.0 licence with no access restrictions at Zenodo: https://doi.org/10.5281/zenodo.21840194. The deposit comprises the gold-standard dataset of 1,000 titles, abstracts, and author-written highlights; all generated summaries together with per-paper and per-model metric scores across the lexical, semantic, and factual dimensions; the raw ratings of all eight expert evaluators; the LLM-as-a-judge panel scores; the data leakage analyses; and the aggregated ranking, concordance, and agreement tables reported in the figures and tables. •All original code is deposited alongside the data and publicly available as of the date of publication under an MIT licence at Zenodo: https://doi.org/10.5281/zenodo.21840194. Development versions are maintained at https://github.com/Delta4AI/LLMTextSummarizationBenchmark. All original code is deposited alongside the data and publicly available as of the date of publication under an MIT licence at Zenodo: https://doi.org/10.5281/zenodo.21840194. Development versions are maintained at https://github.com/Delta4AI/LLMTextSummarizationBenchmark. •Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request. Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.