How Much Structure Do LLMs Need? Evaluating LLMs for Bibliometric Cluster Description

Abraham Camelo-Guerrero, Jairo Diaz-Rodriguez · 2026

Read How Much Structure Do LLMs Need? Evaluating LLMs for Bibliometric Cluster Description by Abraham Camelo-Guerrero, Jairo Diaz-Rodriguez (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24351.txt.

Introduction

Camelo-Guerrero and Diaz-Rodriguez test whether bibliometric structure helps language models write cluster descriptions. Language models can aid literature synthesis but invent references and organize themes weakly. The team compares six pipelines at different levels of evidence and structure, rebuilding Scopus corpora from 100 analyses. Model descriptions land close to human ones in meaning, but models prove unreliable when asked to infer structure from scratch. Results improve when algorithms define the clusters and the model interprets them, pointing to a hybrid workflow.

Catalog record

CREATORAbraham Camelo-Guerrero, Jairo Diaz-Rodriguez
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79762
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24351

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Much Structure Do LLMs Need? Evaluating LLMs for Bibliometric Cluster Description Abraham Camelo-Guerrero, Jairo Diaz-Rodriguez Abstract Large language models (LLMs) can support scientific literature synthesis, but remain prone to hallucinated references, uneven coverage, and weakly grounded thematic organization. We evaluate whether bibliometric structure improves LLM-assisted synthesis by comparing six pipelines for generating cluster descriptions under different levels of evidence and structure. Using 100 published bibliometric analyses, we reconstruct Scopus corpora, extract human-written cluster descriptions, and assess outputs by human alignment, semantic coverage, clustering quality, graph quality, and reference grounding. Results show that LLMs produce descriptions semantically close to human-written ones, but are unreliable when asked to infer bibliometric structure from scratch. Performance improves when bibliometric algorithms define the clusters and the LLM interprets them. Overall, LLM-assisted bibliometric synthesis is most promising as a hybrid workflow in which algorithms provide auditable structure and LLMs generate readable descriptions. 1 Introduction Large language models (LLMs) are increasingly used for scientific literature synthesis, including related-work generation, scientific summarization, and automatic literature review generation ( 13 ; 6 ; 23 ; 17 ; 34 ) . Retrieval-augmented generation improves grounding in external corpora ( 21 ; 9 ) , and recent systems target scientific synthesis directly ( 1 ) . However, generating a literature review from scratch remains difficult: LLMs may hallucinate references, provide uneven coverage, or impose an organization on the literature that is not supported by the scholarly record ( 34 ) . A natural alternative is to separate organization from writing: first construct a structured map of the literature, then use the LLM to synthesize descriptions within that structure. Bibliometric analysis provides such a structured approach. It uses publication and citation metadata to organize papers through relations such as bibliographic coupling, co-citation, and direct citation ( 19 ; 33 ; 5 ) . Widely used in management, information science, health sciences, environmental studies, education, and scientometrics, bibliometric science mapping provides an auditable way to identify research streams, intellectual foundations, influential works, and scholarly communities ( 7 ) . Recent work has begun to use LLMs around bibliometric analysis, mainly for auxiliary tasks such as search support, summarization, topic classification, and thematic mapping ( 31 ; 11 ; 18 ) . However, these studies do not systematically evaluate how different levels of LLM responsibility affect the quality of the bibliometric workflow itself. This makes bibliometric analysis a useful test case for LLM-assisted synthesis: if LLMs struggle to organize a literature from scratch, how much external structure should they receive? Figure 1: Workflow for evaluating LLM-assisted bibliometric synthesis. We address this question by evaluating six LLM-assisted pipelines for generating bibliometric cluster descriptions under different levels of evidence and structure (Figure 1 ). The pipelines range from Blind , where the model receives only the search query, to structured settings such as Labeled and Ranked , where the model receives papers grouped by bibliographic coupling or direct citation clusters. We build the benchmark from 100 published bibliometric analyses by manually extracting author queries, reconstructing Scopus corpora, and collecting human-written cluster descriptions. We assess the outputs in terms of human alignment, semantic coverage, clustering quality, agreement with the underlying bibliometric graph, and reference grounding. This design contributes to a broader shift in LLM-assisted literature review research from isolated single-task applications toward structured, multi-stage workflows. Recent work has examined LLMs for tasks such as screening, query generation, search, extraction, organization, and synthesis under human supervision within these multi-stage workflows ( 26 ; 35 ; 37 ; 28 ; 32 ) . Other studies emphasize structured outputs, including review tables, schemas, hierarchical maps, and extracted evidence, because these outputs are easier to inspect than free-form prose ( 27 ; 12 ; 16 ; 15 ) . Our study extends this direction by testing structure itself as an experimental condition. Our results show that: (i) LLMs can generate cluster descriptions that are semantically close to human-written ones, but are unreliable when asked to infer bibliometric structure from scratch; (ii) when given appropriate bibliometric structure, LLM-generated descriptions can score higher than human descriptions on our corpus-level, clustering, and graph-based metrics, with the strongest performance occurring when bibliometric algorithms first define the clusters and the LLM is used to interpret them; and (iii) the best form of structure depends on the relation: bibliographic coupling benefits from full cluster context, while citation analysis can often be summarized from compact link-ranked evidence. (iv) Overall, LLM-based bibliometric analysis is most promising as a hybrid workflow in which algorithms provide auditable structure and LLMs translate that structure into readable descriptions. 2 Related Work NLP work on scientific literature synthesis includes related-work generation, scientific multi-document summarization, and automatic literature review generation ( 13 ; 6 ; 23 ; 17 ; 34 ) . Recent retrieval-augmented systems further ground scientific generation in external corpora ( 21 ; 9 ; 1 ) , while recent evaluations show that LLM-generated literature reviews still suffer from hallucinated references and uneven coverage ( 34 ) . Our work differs by studying not full review generation, but the more specific task of generating cluster descriptions with the principles of bibliometric analysis ( 7 ) . Our task connects topic labeling with bibliometric science mapping. Topic models and automatic labeling methods induce and verbalize latent themes from text ( 3 ; 24 ; 20 ; 2 ) , and recent work uses LLMs to improve topic modeling and topic interpretability ( 22 ; 36 ) . Bibliometric methods instead cluster papers using relations such as bibliographic coupling, co-citation, and direct citation ( 19 ; 33 ; 5 ) , often with community detection algorithms such as Louvain ( 4 ) . We use bibliometric clusters as an auditable scaffold for LLM generation, testing both whether LLMs can infer literature structure on their own and how much external structure is needed to improve cluster descriptions. 3 Background: Bibliometric Analysis 3.1 Bibliometric Analysis as Science Mapping Bibliometric analysis is a structured approach for studying scientific literatures through metadata such as titles, abstracts, keywords, references, and citations. In science mapping, the goal is to reveal the organization of a research field: its main themes, intellectual foundations, influential works, and research communities ( 7 ) . Rather than producing a single linear summary, bibliometric analysis typically organizes a literature into clusters that can be interpreted as research streams or thematic areas. A simplified bibliometric workflow consists of four stages 8 : \text{query}\rightarrow\text{corpus}\rightarrow\text{clustering}\rightarrow\begin{array}[]{c}\text{cluster}\\ \text{descriptions}\end{array}. Query. The query defines the scope of the analysis. It specifies which topic, keywords, time period, document types, or database fields are included in the search. Because all later stages depend on the retrieved papers, the query strongly shapes the resulting analysis. In published bibliometric reviews, the query is part of the methodological record and provides a reproducible entry point into the literature. Corpus. Executing the query in a bibliographic database such as Scopus produces a corpus of papers. The corpus is the universe of documents to be organized. It provides textual evidence, such as titles and abstracts, as well as bibliographic evidence, such as references and citation links. Importantly, the corpus is not itself a clustering. It is a collection of papers that still needs to be structured. Clustering. The clustering stage

Read the complete text