Bridging Social Psychology and LLM Reasoning: Conflict-Aware Meta-Review Generation via Cognitive Alignment

Wei Chen, Han Ding, Meng Yuan, Zhao Zhang, Deqing Wang, Fuzhen Zhuang · 2025

Read Bridging Social Psychology and LLM Reasoning: Conflict-Aware Meta-Review Generation via Cognitive Alignment by Wei Chen, Han Ding, Meng Yuan, Zhao Zhang, Deqing Wang, Fuzhen Zhuang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13879.txt.

Introduction

Wei Chen and co-authors built a frame in 2025 to help language models write fair meta-reviews of science papers. Journals drown in new work, and a meta-review must weigh clashing referee views into one call. Past model tools blur those clashes and pick up biases of their own, such as clinging to the first view read. Their frame copies the mind's two modes: a fast first take, then slow, checked thought.

Three steps run in turn: a first read, a slow merge of views, and an alignment pass that settles clashes. On tests, the frame lifted the match of tone with human meta-reviews by up to a fifth. Content match rose by up to an eighth. The team pitches it as a steadier hand for editors.

Catalog record

CREATORWei Chen, Han Ding, Meng Yuan, Zhao Zhang, Deqing Wang, Fuzhen Zhuang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-13879

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bridging Social Psychology and LLM Reasoning: Conflict-Aware Meta-Review Generation via Cognitive Alignment Wei Chen, Han Ding, Meng Yuan, Zhao Zhang, Deqing Wang, Fuzhen Zhuang Abstract The rapid growth of scholarly submissions has overwhelmed traditional peer review systems, driving the need for intelligent automation to preserve scientific rigor. While large language models (LLMs) show promise in automating manuscript critiques, their ability to synthesize high-stakes meta-reviews, which require conflict-aware reasoning and consensus derivation, remains underdeveloped. Existing methods fail to effectively handle conflicting viewpoints within differing opinions, and often introduce additional cognitive biases, such as anchoring effects and conformity bias. To overcome these limitations, we propose the Cognitive Alignment Framework (CAF), a dual-process architecture that transforms LLMs into adaptive scientific arbitrators. By operationalizing Kahneman’s dual-process theory, CAF introduces a three-step cognitive pipeline: review initialization, incremental integration, and cognitive alignment. Empirical validation shows that CAF outperforms existing LLM-based methods, with sentiment consistency gains reaching up to 19.47% and content consistency improving by as much as 12.95%. 1 Introduction Academic peer review is fundamental to ensuring scientific rigor, with experts assessing research validity through structured evaluation protocols Chubin et al. (1990) ; Rowland (2002) ; Alberts et al. (2008) . However, the rapid rise in scholarly submissions has created challenges in maintaining these standards, leading to imbalances between reviewer workload and quality assurance Lee et al. (2013) ; Fox et al. (2017) . This scalability issue has driven a shift towards intelligent automation, with large language models (LLMs) showing great promise in enhancing peer review through multimodal reasoning and domain-specific text generation Tyser et al. (2024) ; Jin et al. (2024) ; Zhou et al. (2024) . Figure 1: Illustrating the “Fast Thinking” and “Slow Thinking” processes in human meta-review. While current LLM-driven review automation mainly targets the peer review process, particularly the generation of detailed comments for individual manuscripts, a crucial yet underexplored aspect is meta-review generation Bhatia et al. (2020) ; Santu et al. (2024) . This higher-order task involves integrating conflicting evaluations, and deriving consensus through evidence-based reasoning Li et al. (2023) ; Jin et al. (2024) . Unlike standard summarization, effective meta-review generation requires conflict-aware reasoning that mirrors human decision-making, balancing psychological heuristics and logical deduction Katsikopoulos (2011) . Recently, a few studies Li et al. (2024) ; Zeng et al. (2024) have explored simulating human deliberation in meta-review generation, primarily leveraging the Chain-of-Thought (CoT) reasoning framework Wei et al. (2022) . These approaches decompose the meta-review process into sequential steps: extracting evidence-based opinions, summarizing areas of consensus and controversy, and composing an AC/REJ meta-review. However, the absence of cognitive conflict modeling makes these systems prone to biased decision-making, especially when dealing with opposing opinions in peer reviews. Additionally, we observe that current LLM-based methods unintentionally inherit, and at times amplify, fundamental cognitive biases from human decision-making. These patterns are specifically reflected in: ❶ Anchoring Effect Wilson et al. (1996) : Initial reviewer comments exhibit a stronger anchoring effect on final decisions, with LLMs exhibiting an anchoring coefficient of 0.255, higher than the 0.193 observed in humans. ❷ Conformity Bias Allen and Levine (1968) : Majority opinions tend to suppress minority perspectives through pseudo-consensus reinforcement. The conformity coefficient for LLMs is 0.25, significantly lower than that for humans (1.00), indicating that LLMs are more susceptible to conformity bias. To bridge this gap, we draw inspiration from dual-process theory Groves and Thompson (1970) ; Chaiken (1999) , which posits two distinct modes of cognitive processing: “Fast Thinking” and “Slow Thinking”. As illustrated in Figure 1 , the “Fast Think” quickly handles low-conflict cases with immediate responses, while the “Slow Thinking” is engaged for high-conflict decisions, requiring deeper analysis and reflection. Building on this idea, we propose the Cognitive Alignment Framework (CAF) for meta-review generation, which reimagines the role of LLMs in managing multi-perspective discourse in peer review. Specifically, the CAF architecture involves three key phases: 1) Initialization Phase, where the model extracts key points from each comment; 2) Incremental Integration Phase, which progressively incorporates subsequent reviews while maintaining a conflict map; and 3) Cognitive Alignment Phase, which employs dual-process reasoning to balance heuristic pattern recognition with analytical evaluation. By introducing this adaptive arbitration mechanism, CAF extends the capabilities of LLMs beyond traditional summarization, enabling them to systematically reconcile conflicting scientific opinions. In summary, our contributions are threefold: • We uncover inherent cognitive biases, such as the anchoring effect and conformity bias, within existing LLM-driven meta-review systems. To our knowledge, this is the first instance in which these biases have been identified in this task. • We propose a novel three-phase framework that integrates dual-process theory into LLMs. This framework combines iterative conflict mapping and dual reasoning pathways to mitigate biases and resolve scientific disputes. • Experiments demonstrate that CAF outperforms conventional methods, with sentiment consistency gains reaching up to 19.47% and content consistency improving by as much as 12.95%. 2 Task Formulation Given a set of papers \mathcal{A}=\{A_{1},A_{2},\ldots,A_{V}\} , where each paper A_{i} is accompanied by a set of N peer reviews \mathcal{R}=\{R_{1},\ldots,R_{N}\} , and each review R_{i} consists of a numerical score s_{i} and textual comments c_{i} , the goal is to develop an automated system that generates both an editorial decision and a consensus-driven meta-review for each paper. This problem can be formalized as follows: (\mathcal{E},\mathcal{M})=\mathcal{F}\left(\{(s_{i},c_{i})\}_{i=1}^{N}\right), (1) where \mathcal{E}\in\{\text{Accept},\text{Reject}\} is the final editorial decision 1 1 1 In this paper, we focus on meta-reviews for conferences in the computer science domain. , which could either accept or reject the manuscript, \mathcal{M} is the generated meta-review text that synthesizes the key arguments from the reviews, \mathcal{F} is an LLM-based framework that processes the peer reviews to generate the editorial decision \mathcal{E} and meta-review \mathcal{M} . 3 Cognitive Biases in LLM-based Meta-Review Generation In this section, we will further analyze the existing challenges in current LLM-based meta-review generation systems, particularly focusing on how they may unintentionally introduce human cognitive biases: anchoring effect and conformity bias. 3.1 Anchoring Effect in Sequential Processing Figure 2: The impact of different reviewers (R1, R2, R3, R4, R5) on the final score, with higher y-values indicating greater weight. \Delta represents the difference in anchoring effect between GPT and human reviewers. The anchoring effect Mussweiler (2002) refers to the cognitive bias where individuals’ judgments are disproportionately influenced by an initial reference point, or “anchor” during decision-making processes. In meta-review generation, LLMs may exhibit this bias when they first process an initial set of comments or ratings, which can then disproportionately affect their final assessment. Measure. To assess the impact of different reviewers’ opinions on final decision, we model this process with logistic regression. Formally, the final score \mathcal{E} is modeled as a function of a weighted sum of the individual reviewer scores s_{i} , passed through a Sigmoid function to produce a probability: P(\mathcal{E}=1\mid s_{1},\dots,s_{N})=\sigma\left(\sum_{i=1}^{N}\xi_{i}\cdot s_{i}\right), (1) where \xi_{i} represents the anchoring coefficient for the i -th review, \sigma represents the sigmoid function, and s_{i} is the score provided by reviewer i . Ideally, the coefficients \xi_{i} should be uniform to ensure equal representation of all reviews. However, due to the anchoring effect, the coefficient \xi_{1} (for the first review) is often disproportionately emphasized, particularly in CoT methods. By comparing the \xi_{1} values across experiments, we can gauge

Read the complete text