OmniReview: A Large-scale Benchmark and LLM-enhanced Framework for Realistic Reviewer Recommendation Yehua Huang, Penglei Sun, Zebin Chen, Zhenheng Tang, Xiaowen Chu Abstract . Academic peer review remains the cornerstone of scholarly validation, yet the field faces some challenges in data and methods. From the data perspective , existing research is hindered by the scarcity of large-scale, verified benchmarks and oversimplified evaluation metrics that fail to reflect real-world editorial workflows. To bridge this gap, we present OmniReview , a comprehensive dataset constructed by integrating multi-source academic platforms encompassing comprehensive scholarly profiles through the disambiguation pipeline, yielding 202,756 202,756 verified review records. Based on this data, we introduce a three-tier hierarchical evaluation framework to assess recommendations from recall to precise expert identification. From the method perspective , existing embedding-based approaches suffer from the information bottleneck of semantic compression and limited interpretability. To resolve these method limitations, we propose Pro filing Scholars with M ulti-gate M ixture- o f- E xperts ( Pro-MMoE ), a novel framework that synergizes Large Language Models (LLMs) with Multi-task Learning. Specifically, it utilizes LLM-generated semantic profiles to preserve fine-grained expertise nuances and interpretability, while employing a Task-Adaptive MMoE architecture to dynamically balance conflicting evaluation goals. Comprehensive experiments demonstrate that Pro-MMoE achieves state-of-the-art performance across six of seven metrics, establishing a new benchmark for realistic reviewer recommendation. Our project can be seen in https://sites.google.com/view/omnireview-dataset . 1. Introduction Figure 1 . Comparison of existing datasets and our proposed OmniReview dataset. Conf. denotes score confidence for the recommendation of reviewers. Academic peer review is a standard practice for maintaining scientific quality ( Mulligan et al. 2013 ) . Given the continuous growth of manuscript submissions, appropriate reviewer assignment is necessary to uphold these standards. However, editors often face challenges in identifying experts with the specific domain knowledge and methodological background required for evaluation ( Liu et al. 2014 ) . Consequently, reviewer recommendation systems have been developed to support editorial decision-making, using machine learning to match submissions with suitable reviewers ( Lin et al. 2023 ) . The effectiveness of such systems depends on the availability of high-quality datasets that accurately capture the relationships among papers, reviewers, and their expertise. Well-curated datasets not only enable models to learn meaningful representations of reviewer expertise and research topics, but also serve as the foundation for fair and reliable evaluation of recommendation methods ( Mimno and McCallum 2007 ; Charlin and Zemel 2013 ; Zhu et al. 2022 ) . However, existing datasets exhibit three limitations in the real-world deployment, as shown in Figure 1 : (1) Insufficient Reviewer Profiling. Previous researches highlight the effectiveness of cross-platform scholar identities in research activities ( Haak et al. 2018 ; Cress 2019 ) . However, existing datasets lack comprehensive reviewer profiles. By relying solely on local metadata without integrating large-scale academic graphs, these datasets fail to capture the full scholarly context of researchers. For instance, existing benchmarks ( Mimno and McCallum 2007 ; Karimzadehgan et al. 2008 ; Zhang et al. 2025b ) are often confined to basic metadata from specific conferences. Similarly, other large-scale datasets ( Peng et al. 2025 ) remain limited to platform-specific information (e.g., within Frontiers), lacking the broader external context required for accurate expertise profiling. (2) Biased Candidate Labels. Current datasets contain deficiencies in reviewer candidate construction, leading to two primary biases. First, artificially created datasets by manual annotation or hypothesis introduce label noise. For instance, existing datasets ( Lin et al. 2023 ; Zhang et al. 2025b ) typically utilize researchers’ annotations to assign the relevance of potential reviewers, but this can easily introduce the subjective bias of the annotators. Additionally, exHarmony ( Ebrahimi et al. 2025 ) assumes authors as reviewers, but author lists often include junior researchers who may not yet possess the necessary qualifications for reviewing. Second, relying solely on historical assignment records restricts the candidate pool. Some datasets ( Peng et al. 2025 ; Zhang et al. 2024 ) are confined to a closed set of past reviewers. However, without explicitly verifying the expertise compatibility between candidates and papers, the recommendation task fails to align with real-world scenarios. (3) Simplistic Evaluation Metrics. Existing benchmarks ( Peng et al. 2025 ; Zhang et al. 2024 ) primarily focus on standard retrieval metrics based on historical assignments. However, given the inherent sparsity of ground-truth labels (i.e., historical reviewers are only a small subset of all scholars), these metrics suffer from the false negative bias. It unfairly penalizes models for recommending qualified candidates simply because they were not historically selected. Furthermore, current metrics fail to assess nuanced capabilities essential for practical deployment, such as confidence score, noise filtering, and the ability to discriminate between unassigned experts and truly unsuitable candidates. To address these limitations, we propose OmniReview , a comprehensive peer-review dataset and evaluation constructed based on peer-review records in Frontiers open-access platform 1 1 1 https://www.frontiersin.org/ . Our approach tackles the aforementioned limitations from three parts . First , to overcome data fragmentation, we integrate three authoritative sources: the Open Academic Graph (OAG) ( Tang et al. 2008 ; Zhang et al. 2019 ; Zhang et al. 2023 ; Sinha et al. 2015 ) , the Frontiers open-access platform, and the ORCID Public Data File 2 2 2 https://info.orcid.org/documentation/integration-guide/working-with-bulk-data/ . We employ a pipeline involving entity alignment, discipline taxonomy construction, and results verification to bridge information across these sources. This process successfully links reviews to researcher profiles, yielding 202,756 verified records and 150,287 identified reviewers. Second , to mitigate label bias and the false negative issue, we move beyond sparse historical assignments by constructing dense relevance labels. We organize publications and scholars into a hierarchical subject taxonomy and identify potential experts based on semantic matching. To ensure high quality, we further filter these candidates using h-index thresholds. This strategy allows us to recognize qualified but unassigned experts, thereby enriching the ground truth and making the recommendation system more robust. Third , to transcend simplistic evaluation metrics, we design a comprehensive evaluation framework consisting of three distinct tasks: (1) Task 1 (Recall): Retrieving historical ground-truth reviewers; (2) Task 2 (Discrimination): Filtering out hard-negatives candidates who appear relevant superficially but lack specific domain expertise; (3) Task 3 (Ranking): Fine-grained ranking of the best experts among the qualified candidates. This structured approach allows for a comprehensive diagnosis of the system’s capabilities in retrieval, noise filtering, and precise ranking. While existing embedding-based approaches have shown promise ( Singh et al. 2023 ; Charlin and Zemel 2013 ; Zhang et al. 2025b ) , they face inherent limitations when evaluated against the comprehensive requirements of OmniReview: First , these methods are constrained by the information bottleneck of semantic compression ( Pan et al. 2024 ; Weller et al. 2025 ) . By reducing complex publication histories to static vectors, they often oversmooth the fine-grained differences between sub-fields. This loss of granularity specifically hinders Task 2 (Discrimination), as the model struggles to distinguish true experts from hard negative candidates who share superficial keywords but lack deep domain expertise. Second , they suffer from limited interpretability ( Wang et al. 2018 ) . The implicit, high-dimensional nature of latent embeddings acts as a black box. This leaves decision-makers without the verifiable textual evidence required to validate recommendations, a critical gap for real-world adoption. To address these challenges, we propose Pro filing Scholars with M ulti-gate M ixture- o f- E xperts ( Pro-MMoE ), which resolves the aforementioned limitations through a two-stage pipeline. First, to simultaneously mitigate semantic compression and resolve the limited interpretability, we construct LLM-generated Semantic Profiles. Instead of compressing history into opaque vectors, we instruct an LLM to extract and summarize fine-grained research interests from publication records. This operation preserves the discipline-adjacent nuances needed for precise identification while naturally providing readable textual evidence to ground the model’s decisions. Second, to overcome the optimization conflict, we implement a Task-Adaptive MMoE architecture. By assigning specific expert networks to learn shared representations and distinct towers for Recall, Discrimination, and Ranking, our model dynamically balances the trade-off between retrieving broad candidates and filtering for absolute expertise within a unified framework. Extensive experiments on the OmniReview benchmark demonstrate the superiority of our approach. Pro-MMoE outperforms state-of-the-art baselines by margins of 1.02\% , 5.39\% , and 17.15\% across the three defined tasks, respectively. In conclusion, this paper makes the following contributions: \bullet We construct a large-scale, verified peer-review dataset OmniReview , comprising 202,756 papers and 150,287 reviewers with ground-truth review records. Based on this, we define an evaluation framework to assess reviewer recommendations across recall, discrimination, and ranking capabilities. \bullet We propose Pro-MMoE , a novel framework designed to overcome the limitations of implicit embeddings and single-objective optimization. By synergizing LLM-generated semantic profiles with a Task-Adaptive MMoE architecture, our approach simultaneously enhances representation granularity, improves interpretability, and resolves the conflicts between diverse evaluation goals. \bullet Experiments demonstrate that Pro-MMoE achieves state-of-the-art performance on the OmniReview benchmark. It surpasses existing baselines across six of seven metrics, showing improvements in both precise expert identification and reliable candidate ranking. Table 1 . Comparison between OmniReview with existing peer reviewer recommendation dataset. GT. refers to ground-truth. Dataset Papers Reviewers Reviewer Profiling Candidate Labels Evaluation Metrics Academic Graph ORCID GT. Reviewer Verified Candidate Recall Discrimination Ranking NIPS ( Lin et al. 2023 ) 34 190 ✗ ✗ ✗ ✓ ✓ ✗ ✗ KDD ( Zhang et al. 2025b ) 174 737 ✗ ✗ ✗ ✓ ✓ ✗ ✗ ACM-DL Library ( Zhang et al. 2020 ) 13k 22k ✗ ✗ ✗ ✓ ✓ ✗ ✗ exHarmony ( Ebrahimi et al. 2025 ) 8k 33k ✓ ✗ ✗ ✗ ✓ ✗ ✗ OAG-Bench ( Zhang et al. 2024 ) 40k 42k ✓ ✗ ✓ ✗ ✓ ✗ ✗ FRONTIER-RevRec ( Peng et al. 2025 ) 478k 178k ✗ ✗ ✓ ✗ ✓ ✗ ✗ OmniReview(Ours) 203k 150k ✓ ✓ ✓ ✓ ✓ ✓ ✓ 2. Related Work 2.1. Reviewer Recommendation Dataset Table 1 demonstrates that reviewer recommendation datasets have evolved significantly from small-scale benchmarks to large-scale, multi-disciplinary resources. The seminal NIPS dataset ( Lin et al. 2023 ) pioneered expert-annotated relevance judgments for 34 papers and 190 reviewers. Building upon this foundation, Zhang et al. ( Zhang et al. 2025b ) integrated review records from three existing datasets to create the KDD dataset, which marked relevance scores for 3,480 paper-reviewer pairs. However, these early datasets remained limited to specific domains. To address this constraint, OAG-Bench ( Zhang et al. 2024 ) expanded coverage to all academic subjects and linked dataset metadata to a comprehensive scholar graph, thereby enriching academic archives. The ACM-DL dataset ( Zhang et al. 2020 ) extracted information from the ACM Digital Library, framing reviewer-paper matching as a multi-label classification problem using 1,944 research field labels from the CCS taxonomy, while exHarmony ( Ebrahimi et al. 2025 ) leveraged the OpenAlex scholar graph under the hypothesis that paper authors represent potentially optimal reviewers for related submissions. The most comprehensive effort to date, FRONTIER-RevRec ( Peng et al. 2025 ) , further expands the data volume to 177,941 reviewers and 478,379 papers. In comparison, our dataset retains 202,756 review records and matches the metadata involved in the review records with the open academic graph. 2.2. Recommendation Methods Statistical-based recommendation historically employed TF-IDF weighting to represent items as sparse vectors to determine relevance ( Ramos et al. 2003 ) . Embedding-based methods leveraged neural representations for similarity computation. Zhang et al. ( Zhang et al. 2025b ) proposed the chain-of-factors matching that utilized factor-based annotations to capture nuanced matching criteria beyond simple text similarity. Graph-based approaches exploit structural relationships through link prediction. Relational graph convolutional network ( Schlichtkrull et al. 2018 ) extended GCNs to multi-relational graphs, modeling reviewer-paper interactions with relation-specific transformations. Li et al. ( Li et al. 2025 ) proposed a recommendation framework that demonstrates the potential of large language models for semantic understanding and the MMoE architecture for modeling diverse user interests, enabling more accurate and explainable recommendations. 3. Dataset Construction 3.1. Definition Let’s define our proposed OmniReview dataset as \mathcal{D}=(\mathcal{P},\mathcal{A},\mathcal{R}) , where \mathcal{P} represents the paper collection, \mathcal{A} denotes the scholar set, and \mathcal{R} defines the peer review records. The construction methodology comprises three sequential phases: (1) entity alignment, (2) discipline taxonomy construction, and (3) task formulation. The entity disambiguation phase resolves ambiguities in academic information across sources by leveraging metadata, such as titles and co-author networks. The discipline taxonomy construction phase builds a comprehensive academic taxonomy by organizing research domains into a hierarchical tree structure. Finally, the task formulation phase defines the specific evaluation tasks and selects unqualified candidates and potential candidates. 3.2. Entity Alignment Complementary academic data sources provided differentiated information. To ensure the reliability of scholar identification, the entity alignment process mainly follows four steps: (1) data cleaning; (2) publication matching; (3) scholar matching; (4) verification. Figure 2 . Disambiguation Flowchart 3.2.1. Data Cleaning In this process, we first check the uniqueness of scholars and the publications. For OAG, we identify and remove author records lacking unique persistent identifiers, as these entries cannot be reliably distinguished from other authors with similar names or affiliations. Regarding the Frontiers public metadata, we filter out reviewers who are not associated with any publication records, as these reviewers lack verifiable scholarly output necessary for disambiguation and assessing their expertise and domain knowledge. Similarly, for ORCID data, we exclude scholar profiles containing no publication information. The cleaning procedure ensures that the final integrated dataset maintains high data integrity, containing only researchers with identifiable academic identities and demonstrable publication histories. Algorithm 1 Publication Matching for Cross-Source Publications 1: Two publication titles t^{(1)} and t^{(2)} from publications P^{(1)} and P^{(2)} in different data sources S^{(1)} and S^{(2)} 2: Boolean match result indicating whether titles represent the same publication 3: function NormalizeTitle ( title ) 4: clean\_title\leftarrow\text{remove\_punctuation}(title) 5: normalized\leftarrow\text{to\_lowercase}(clean\_title) 6: return split_into_words ( normalized ) 7: end function 8: words^{(1)}\leftarrow\textsc{NormalizeTitle}(t^{(1)}) 9: words^{(2)}\leftarrow\textsc{NormalizeTitle}(t^{(2)}) 10: for k=1 to \text{length}(words^{(1)}) do 11: if words^{(1)}[k]\neq words^{(2)}[k] then 12: return false 13: end if 14: end for 15: return true 3.2.2. Publication Matching In this process, we match articles from different sources based on their titles. Typically, the matching challenges lie in the slight differences in the format of title records, including word capitalization or punctuation. Because of this, we adopt word-level title matching, shown in Algorithm 1 , to eliminate the difference and match the publications. Formally, let \mathcal{P}^{(1)}=\{p^{(1)}_{1},p^{(1)}_{2},\ldots,p^{(1)}_{n_{1}}\} and \mathcal{P}^{(2)}=\{p^{(2)}_{1},p^{(2)}_{2},\ldots,p^{(2)}_{n_{2}}\} represent the publication sets from two distinct data sources. For each publication p , we define a normalization function \mathcal{N}(p)=\{w_{1},w_{2},\cdots,w_{m}\} that converts all letters to lowercase, removes all punctuation marks and splits into words, and, following w^{(1)}_{1}=w^{(2)}_{1},\cdots,w^{(1)}_{n}=w^{(2)}_{n}\rightarrow p^{(1)}_{i}=p^{(2)}_{j} , every word in two possible publications should be strictly the same to match them. 3.2.3. Scholar Matching In this process, we establish cross-dataset author identity resolution through publication-based co-occurrence analysis. Similar to the challenges in publication matching, different data sources vary in the storage of special characters of scholars’ names, such as Roman numerals and pinyin characters. We first define a normalization function that transforms all special characters to their potential expressions in English. To match scholar a in data source S_{i} with publications set P_{i}^{(a)}=\{p_{1}^{(a)},p_{2}^{(a)},\cdots,p_{n}^{(a)}\} to the scholar in source S_{j} , the first step is to get the same publications P_{j}=\{p|p\in S_{j},p=p_{i}^{(a)},p_{i}^{(a)}\in P_{i}^{(a)}\} in source S_{j} . The next step is to find the potential author set A^{\prime} based on the intersection of the co-authors of all p_{i}^{(a)}\in P_{i}^{(a)} . The final matching decision between candidate scholars a and a^{\prime}\in A^{\prime} follows a sequential two-stage verification process. First, we assess lexical similarity by examining the number of identical normalized name tokens between the targeted scholar a and candidate a^{\prime}\in A^{\prime} . The candidate with the highest similarity represents the matched scholar a^{\prime} in source S_{j} . If no overlapping name tokens are found or there exist candidates sharing the same similarity, we apply a secondary verification criterion requiring exact matching of the first character of words in normalized names. Algorithm 2 demonstrates the scholar matching algorithm. Algorithm 2 Cross-Source Scholar Matching with Optimal Token Matching 1: Scholar a from source S_{i} with publication set P^{(a)}_{i} 2: Publication set P_{j} from source S_{j} containing publications with matching titles to P^{(a)}_{i} 3: Best matched scholar a^{\prime} from S_{j} or \varnothing if no match found 4: function Normalize ( name ) 5: clean\leftarrow\text{ConvertRomanNumerals}(name) 6: clean\leftarrow\text{TransliteratePinyin}(clean) 7: return clean 8: end function 9: \mathcal{A}^{\prime}=\bigcap_{p_{j}\in P_{j}}\text{GetAuthors}(p_{j}) 10: name_{a}\leftarrow\textsc{Normalize}(a.\text{name}) 11: words^{(a)}\leftarrow\text{split\_into\_words}(name_{a}) 12: best\_candidate\leftarrow\varnothing 13: max\_score\leftarrow 0 14: for each a^{\prime}\in\mathcal{A}^{\prime} do 15: name_{a^{\prime}}\leftarrow\textsc{Normalize}(a^{\prime}.\text{name}) 16: words^{(a^{\prime})}\leftarrow\text{split\_into\_words}(name_{a}^{\prime}) 17: if |words^{(a)}\cap words^{(a^{\prime})}|>max\_score then 18: max\_score\leftarrow|name_{a}\cap name_{a^{\prime}}| 19: best\_candidate\leftarrow append(best\_candidate,a^{\prime}) 20: end if 21: end for 22: if max\_score>0\And|best\_candidate|=1 then 23: return best\_candidate 24: end if 25: for each a^{\prime}\in best\_candidate do 26: name_{a^{\prime}}\leftarrow\textsc{Normalize}(a^{\prime}.\text{name}) 27: initials_{a}\leftarrow\{w[0]\mid w\in name_{a}\} 28: initials_{a^{\prime}}\leftarrow\{w[0]\mid w\in name_{a^{\prime}}\} 29: if initials_{a}=initials_{a^{\prime}} then 30: return a^{\prime} 31: end if 32: end for 33: return \varnothing 3.2.4. Verification For every candidate pair (a,a^{\prime}) identified through the matching process, we compare their complete publication records across all available data sources, and the verification requires that at least one publication matches across different data sources. Specifically, for author a from source \mathcal{S}_{i} against matched author a^{\prime} from source \mathcal{S}_{j} , a valid match is accepted only if there exists at least one publication pair (p,p^{\prime}) such that \mathcal{N}(p.\text{title})=\mathcal{N}(p^{\prime}.\text{title}) to confirm they represent the same scholarly work. 3.3. Construction of the Discipline Taxonomy Our dataset has a collection of 109,579,745 publications and 34,855,513 researcher profiles, presenting computational challenges for recommendation system development and benchmark construction due to its large scale. To address this complexity, we construct a hierarchical discipline taxonomy \mathcal{T}=\{L_{1},L_{2},L_{3}\} , where L_{1} captures the most general research domains and L_{3} represents the most specialized research topics. We employ the Qwen3-Embedding-4B model to generate semantic representations for each article based on its title and abstract, and articles are then classified to subject nodes through semantic similarity maximization. Additionally, each scholar a\in\mathcal{A} derives his comprehensive subject profile through the union of his publication assignments \hat{l}_{i}^{(a)}=\bigcup_{p\in P_{a}}\hat{l}_{i}^{(p)} , where P_{a} represents the publication set of scholar a , and \hat{l}_{p} denotes the assigned subject of publication p in level i . Through discipline taxonomy, candidates can be efficiently selected, significantly reducing the computational burden. 3.4. Task Statement As illustrated in Section 1 , we define three tasks to evaluate the performance of the recommendation system: (1) Retrieving historical ground-truth reviewers; (2) Filtering out hard-negative candidates who appear relevant superficially but lack specific domain expertise; (3) Fine-grained ranking of the best experts among the qualified candidates. 3.4.1. Ground-truth Reviewers Retrieval Existing datasets ( Ebrahimi et al. 2025 ; Zhang et al. 2024 ; Peng et al. 2025 ) evaluate the ability of reviewer recommendation systems to retrieve ground-truth reviewers using ranking-based metrics that contrast them against a pool of irrelevant candidate reviewers. In this task, we assess retrieval effectiveness by measuring the system’s capacity to assign higher confidence scores to ground-truth reviewers. For each paper p in our benchmark, the system must generate confidence scores f(p,r)\in[0,1] for all genuine reviewers, with the expectation that true reviewers obtained confidence scores closer to 1. The evaluation metric emphasizes the concentration of confidence scores on actual reviewers. 3.4.2. Elimination of Unqualified Candidates This task evaluates a recommendation system’s ability to identify and eliminate scholars who lack domain-specific expertise despite apparent disciplinary proximity. For a target paper p categorized under subject c_{p}\in L_{3} , we construct a challenging candidate pool by identifying the most semantically similar subject category c^{*}\in L_{3}\setminus\{c_{p}\} using precomputed category embeddings. From category c^{*} , we sample scholars who have substantial publication records within the category c^{*} but zero publications in the target category c_{p} or any of its descendant categories in the hierarchical taxonomy \mathcal{T} . These scholars represent the most deceptive negative examples that researchers with strong academic credentials in closely related fields who nevertheless lack the specific expertise required to evaluate papers in c_{p} . The recommendation system must assign significantly lower confidence scores to these unqualified candidates compared to domain experts. 3.4.3. Best Reviewers Selection This task evaluates a recommendation system’s ability to identify actual reviewers from a challenging pool of cross-disciplinary candidates who possess relevant expertise through publication records. For each target paper p categorized under subject c_{p}\in L_{3} , we first identify the most semantically similar category c^{*}\in L_{3}\setminus\{c_{p}\} using precomputed category embeddings. From category c^{*} , we sample scholars who have: (1) substantial publication records within the category c^{*} , and (2) at least one publication in the target category c_{p} , representing cross-disciplinary experts with verified domain knowledge. The recommendation system must rank the ground-truth reviewers higher within this challenging pool of domain-competent candidates. This task simulates realistic editorial scenarios in which editors must select reviewers from a pool of cross-disciplinary experts who share relevant domain knowledge through their publications. Figure 3 . (a) The overview of the Pro-MMoE architecture, comprising three main modules: Profiling Module, Multi-gate Mixture-of-Experts Module, and Task-specific Prediction Towers. Conf. refers to confidence and Rank. refers to ranking. (b) Detailed illustration of the review candidate embedding process. (c) Expert network structure within MoE, featuring multiple shared subnetworks. (d) Task-specific DNN towers for specific tasks. 4. Method 4.1. Method Overview Our Pro-MMoE method utilizes large language models and a Multi-gate Mixture-of-Experts (MMoE) architecture to jointly optimize two complementary objectives: predicting recommendation scores and estimating confidence. As depicted in Figure 3 , the framework comprises three core components: (1) a Profiling Module that utilizes large language models to summarize the paper and the scholar and apply an embedding model to embed the summaries; (2) a Multi-gate Mixture-of-Experts Module that jointly captures and handles the feature of the paper-candidate pair; and (3) Task-specific prediction towers that respectively predict confidence score, and recommendation scores for the paper-candidate pair. 4.2. Profiling Module The Profiling Module constructs rich vector representations for both the query paper and review candidates via a hierarchical LLM-based summarization framework, mitigating information loss inherent in direct embedding methods and enhancing the interpretability ( Jiang et al. 2024 ; Bonifacio et al. 2022 ; Wang et al. 2023 ) . For the query paper, the LLM is employed to distill the key contributions, research focus, and technical context from its title and abstract into a compact yet semantically complete summary. For review candidates, the LLM first extracts the key technical content of individual publications and then aggregates these summaries into an expertise-oriented profile. Finally, both the query paper summary and the candidate profile summary are encoded using the Qwen3 Embedding model to produce dense vector representations. The joint embedding is formulated as: (1) \begin{aligned} \mathbf{h}_{\text{joint}}=\left[\mathbf{E}_{\text{Qwen3}}(\text{paper summary}),\ \mathbf{E}_{\text{Qwen3}}(\text{reviewer summary})\right],\end{aligned} where \mathbf{E}_{\text{Qwen3}} denotes the Qwen3 embedding function, and the concatenation operation [\cdot,\cdot] combines paper and candidate representations. 4.3. Multi-gate Mixture-of-Experts Module The Multi-gate Mixture-of-Experts Module is designed for multi-task learning ( Ma et al. 2018 ) , and it processes the joint embedding through a shared set of expert networks, with task-specific gating mechanisms that dynamically weight expert contributions based on input characteristics. This module contains n expert networks \{E_{i}\}_{i=1}^{n} , each implemented as a feed-forward neural network, and two gating networks \{G^{(k)}\}_{k=1}^{2} corresponding to our two kinds of tasks (i.e., confidence prediction for Task 1,2 and ranking for Task 3 in Section 3.4 ). The gated expert mixture for task k is computed as: (2) \begin{aligned} \hat{h}^{(k)}=\sum_{i=1}^{n}g^{(k)}_{i}(\mathbf{h}_{\text{joint}})\cdot E_{i}(\mathbf{h}_{\text{joint}}),\end{aligned} where g^{(k)}_{i}(\mathbf{h}_{\text{joint}}) denotes the i -th element of the gating output vector G^{(k)}(\mathbf{h}_{\text{joint}}) for task k . 4.4. Task-specific Prediction Towers The task-specific prediction towers transform the gated expert representations into final predictions for each objective. Specifically, the Confidence Prediction Tower is designed for ground-truth reviewers retrieval and elimination of unqualified candidates to estimate the model’s certainty in the candidates, while the Ranking Tower is responsible for the best reviewers selection task and aims at assigning proper scores for candidates to find out the best reviewers. Each tower employs specialized loss functions optimized for each task objective. For the confidence prediction tower, we utilize a weighted binary cross-entropy loss with entropy regularization: (3) \begin{aligned} \mathcal{L}_{\text{conf}}=\mathcal{L}_{\text{BCE}}(y,\hat{y})+\lambda_{\text{entropy}}\cdot\mathcal{H}(G),\end{aligned} where \mathcal{L}_{\text{BCE}} applies inverse class frequency weighting to handle label imbalance, and \mathcal{H}(G)=-\mathbb{E}[\sum_{i}g_{i}\log g_{i}] maximizes the entropy of gating distributions to prevent expert specialization collapse, where g_{i} denotes the i -th element of the gating output vector G . For the recommendation tower, we implement a hybrid AUC-margin loss: (4) \begin{aligned} \mathcal{L}_{\text{auc}}&=\mathbb{E}\left[-\log\sigma(s^{+}-s^{-})\right],\\ \mathcal{L}_{\text{margin}}&=\mathbb{E}\left[\max(0,m-(s^{+}-s^{-}))\right],\\ \mathcal{L}_{\text{rec}}&=(1-\lambda_{\text{auc}})\cdot\mathcal{L}_{\text{margin}}+\lambda_{\text{auc}}\cdot\mathcal{L}_{\text{auc}}.\end{aligned} This loss function simultaneously optimizes pairwise ranking margins and global AUC maximization. The margin component ensures sufficient score separation between qualified and unqualified reviewers, while the AUC component optimizes the overall ranking quality with parameter \lambda_{\text{auc}} controlling their relative importance. This dual-loss formulation enables our model to achieve both precise confidence calibration for reviewer validation tasks and optimal ranking performance for reviewer recommendation tasks. 5. Experiment Table 2 . Reviewer Recommendation System Performance Evaluation Method Name Method Category Task 1 Task 2 Task 3 RRC \uparrow UCC \downarrow MAP \uparrow R-prec \uparrow Recip-rank \uparrow NDCG \uparrow Success@5 \uparrow TF-IDF ( Ramos et al. 2003 ) Statistic 0.8715 0.3523 0.7064 0.5746 0.7557 0.7946 0.9168 Dual-tower ( Huang et al. 2013 ) Bi-encoder 0.8004 0.4212 0.4163 0.2053 0.4163 0.5534 0.7115 RGCN ( Schlichtkrull et al. 2018 ) Graph 0.6826 0.3608 0.5330 0.3640 0.5517 0.6612 0.8779 SciBERT ( Beltagy et al. 2019 ) Embedding 0.7155 0.4523 0.5658 0.3950 0.6058 0.6898 0.8885 SPECTER2 ( Singh et al. 2023 ) Embedding 0.8430 0.4533 0.6875 0.5552 0.7341 0.7797 0.8956 BGE-M3 ( Chen et al. 2024 ) Embedding 0.8709 0.4891 0.6510 0.5153 0.6972 0.7531 0.9044 Qwen3-4B ( Zhang et al. 2025a ) Embedding 0.8864 0.4734 0.6745 0.5405 0.7223 0.7708 0.9097 CoF ( Zhang et al. 2025b ) Multi-Embedding 0.9092 0.3877 0.7194 0.5972 0.7586 0.8021 0.9239 LLM-as-Judge ( Lyu et al. 2024 ) LLM-based 0.4192 0.0355 0.6306 0.4931 0.6934 0.7406 0.8549 Pro-MMoE (ours) LLM-based+MMoE 0.9185 0.1900 0.8114 0.7000 0.8564 0.8724 0.9735 5.1. Experiment Settings 5.1.1. Dataset We constructed the training, validation, and test sets by stratified sampling in a ratio of 7:2:1. The original test dataset comprises 20,299 review records. Each review record contains the metadata of the paper, the metadata of the reviewers, the metadata of the editor, and the category in our subject tree, and the detailed example can be viewed in Appendix D . Considering the time-intensive nature of this process, we conducted stratified sampling based on the distribution of academic disciplines to create a representative subset for evaluation. This resulted in a curated test set of 565 review records, which maintains the disciplinary distribution of the original dataset while enabling efficient and consistent evaluation across all methods. Specifically, in the training process of our proposed Pro-MMoE method, we similarly applied stratified sampling to the training dataset according to academic disciplines, selecting 8,427 review records to ensure balanced representation across domains while maintaining computational feasibility during the training phase. 5.1.2. Evaluation Metrics We evaluate model performance across three distinct tasks mentioned in Section 3.4 : Task 1 measures the model’s ability to retrieve actual reviewers using the mean of real reviewer confidence (RRC), where higher values indicate better performance. Task 2 assesses elimination capacity using the mean of unqualified candidate confidence (UCC), where lower values are preferred. Task 3 evaluates the recommendation quality using five standard information retrieval metrics: Mean Average Precision (MAP), R-Precision (R-prec), Reciprocal Rank (Recip-rank), Normalized Discounted Cumulative Gain (NDCG), and success rate at position 5 (Success@5). In task 3, each metric score represents the average score across all review records in the test set. 5.1.3. Training Details Our complete pipeline integrates three key components: the Qwen3-30B-A3B for reviewer summarization, the Qwen3-Embedding-4B model for generating semantic embeddings, and an MMoE architecture with 3 experts for multi-task learning. During the reviewer summarization phase, we extract the most representative publications for each candidate by selecting their 5 most highly-cited articles and 5 most recently published articles from their academic portfolio. We trained the model for 100 epochs using a two-stage optimization strategy. In the first stage (epochs 1-50), we froze the recommendation tower and its associated gating network, training only the remaining architecture to establish robust feature representations. In the second stage (epochs 51-100), we only train the recommendation tower and its gating network, allowing fine-tuning of the recommendation-specific components while preserving the learned feature representations from the first stage. 5.1.4. Baseline Methods We comprehensively assess a diverse range of recommendation models spanning traditional information retrieval methods, dense embedding architectures, graph-based approaches, and our novel framework. Specifically, we include: (1) a TF-IDF classifier ( Ramos et al. 2003 ) as the statistical baseline for text-based matching; (2) a Dual-tower model ( Huang et al. 2013 ) representing the bi-encoder approach with separate encoders for paper and reviewer representations; (3) a Relational Graph Convolutional Network (RGCN) ( Schlichtkrull et al. 2018 ) that leverages heterogeneous scholarly graph structures; (4) Embedding models including SciBERT ( Beltagy et al. 2019 ) , SPECTER2 ( Singh et al. 2023 ) , BGE-M3 ( Chen et al. 2024 ) , Qwen3-Embedding-4B ( Zhang et al. 2025a ) , and Chain-of-Factors Paper-Reviewer Matching proposed by Zhang et al. ( Zhang et al. 2025b ) that generate representations of scholarly semantic, citation or topic content; (5) a LLM-as-Judge approach ( Lyu et al. 2024 ) that directly utilizes large language models for reviewer assessment. Details of the baseline methods can be viewed in Appendix C . 5.2. Results and Analysis Table 2 presents the comprehensive performance comparison of our proposed approach against baseline methods across three evaluation tasks. Our Pro-MMoE method achieves state-of-the-art performance on six out of seven metrics while demonstrating competitive performance on the remaining metric. 5.2.1. Overall Performance Analysis Our proposed method Pro-MMoE outperforms all baselines on Task 1 (RRC) with a score of 0.9185 , representing a relative improvement of 1.02% over the second-best method CoF ( Zhang et al. 2025b ) ( 0.9092 ) and 5.39% over the strong TF-IDF baseline (0.8715). For Task 2 (UCC), our model achieves the second-best result of 0.1900 , improving upon most existing review recommendation methods. In Task 3, our model achieves 0.8114 MAP , 0.7000 R-prec , 0.8564 Recip-rank , 0.8724 NDCG , and 0.9735 Success@5 , surpassing CoF by margins of 12.79% , 17.15% , 12.89% , 8.76% , and 5.37% respectively. 5.2.2. Methodological Category Comparison TF-IDF, as a lightweight statistical-based method, achieved a third-place result. This indicates that the keywords of the article play an important role in reviewers’ selections. The Dual-tower bi-encoder model and RGCN show relatively poor performance across all tasks, suggesting that simple models cannot effectively capture the review relationship. Recommendation based on semantic relevance is directly related to the strength of the embedding model. Earlier models, including SciBERT and SPECTER2, achieve relatively weak performance across all three tasks, while more recent BGE-M3 and Qwen3-Embedding-4B demonstrate stronger performance. The second-best method CoF indicates that utilizing three factors of semantic, topic, and citation can further enhance the capture of the review relationship. The LLM-as-Judge approach tends to give a low score for each paper-review pair, resulting in a poor score in Task 1. Our Pro-MMoE maintains competitive UCC performance while substantially improving all other metrics, confirming that the mixture-of-experts architecture based on LLM profiling successfully balances discriminative and generative capabilities. 5.3. Ablation Study 5.3.1. Ablation Study on Framework Components Table 3 . Ablation Study on Framework Components Method RRC \uparrow UCC \downarrow NDCG \uparrow Pro-MMoE 0.9185 0.1900 0.8724 w/o MG 0.8860 0.2086 0.9123 w/o MG & MoE 0.5090 0.4870 0.5698 w/o LLM Sum. 0.8571 0.1145 0.8537 w/o LLM Sum. & MG 0.8806 0.1390 0.8361 w/o LLM Sum. & MG & MoE 0.8864 0.4734 0.7708 The ablation study in Table 3 is structured in two parts: the upper part progressively removes the Multi-Gate (MG) mechanism and Mixture-of-Experts (MoE) components while retaining LLM-generated summaries, whereas the lower part first removes the LLM summarizer and then additionally eliminates MG and MoE. Without the MMoE, the recommendation falls back to comparing the semantic embedding similarity. The results demonstrate that LLM summaries provide critical advantages in both recommendation quality and confidence discrimination, and the combined use of LLM summarizer and MMoE architecture demonstrates a better and more balanced performance of achieving higher confidence discrimination capability and improving recommendation ranking simultaneously. 5.3.2. Impact of the Number of Experts We conduct an ablation study on the number of experts that investigates the optimal configuration of our Multi-gate Mixture-of-Experts (MMoE) architecture. Figure 4 presents that the confidence gap between RRC and UCC remains relatively stable across different expert configurations, indicating similar discrimination capability between qualified and unqualified reviewers regardless of expert count. Notably, the NDCG metric achieves its peak performance at exactly 3 experts, after which additional experts provide diminishing returns or even slight degradation in ranking quality. This suggests that increasing expert numbers beyond 3 cannot enhance the model’s ability to distinguish real from unqualified reviewers, and it also fails to improve the recommendation ranking quality, with the optimal configuration occurring at 3 experts for maximum NDCG performance. Figure 4 . Impact of the number of experts in MMoE. MMoE with 3 experts shows the optimal trade-off between recommendation quality and computational efficiency. 5.3.3. Impact of the Data Volume We conduct an ablation study by varying the training data ratio from 0.05 to 1.0 to investigate the impact of training data quantity on model performance. Figure 5 demonstrates the performance trends of our model under varying training data ratios. As the data quantity decreases, all three metrics exhibit moderate degradation. However, the model maintains robust performance even with significantly reduced training data. Notably, at the lowest data ratio (0.05), our model still outperforms most baseline methods, indicating that our approach achieves considerable data efficiency. This resilience to data reduction suggests that the model can effectively learn essential patterns from limited examples, making it practical for scenarios where large-scale annotated data may be unavailable or costly to obtain. The performance curves, particularly for NDCG, further confirm the model’s ability to maintain high-quality recommendations with minimal training data requirements. Figure 5 . Performance trends across different training data volume ratios. The model maintains robust performance even with reduced data volumes, showing only moderate degradation in RRC, UCC, and NDCG metrics as the data volume ratio decreases from 1.0 to 0.05. 6. Conclusion In this work, we try to design a realistic peer reviewer recommendation framework from both data and method perspectives. From the data perspective, we construct a large-scale dataset containing 202,756 verified review records and 150,287 ground-truth reviewers, disambiguate against multi-source databases like OAG and ORCID. To standardize the evaluation in this field, we introduced three-tier evaluation tasks that tests the model’s capability to distinguish genuine reviewers from candidates with varying degrees of domain overlap. From the method perspective, we propose Pro-MMoE, an LLM-enhanced multi-task learning framework. By synergizing LLM-generated semantic profiling with task-adaptive gating mechanisms, our approach effectively resolves the conflict between candidate ranking and confidence calibration. Extensive experiments confirm that our method establishes new state-of-the-art performance on six out of seven metrics. We hope OmniReview will serve as a foundational resource, fostering a more transparent, efficient, and equitable academic peer review ecosystem. Appendix A Ethical Use of Data Open Academic Graph (OAG) is released under the Open Data Commons Attribution (ODC-BY) license. ORCID provides its Public Data File under the CC0 1.0 Public Domain Dedication. Article metadata from the Frontiers open-access platform is distributed under Creative Commons CC0 terms. All data are used in accordance with their respective licenses. Appendix B Details of the Discipline Taxonomy Our discipline taxonomy is constructed based on the Classification and code of disciplines 3 3 3 https://openstd.samr.gov.cn/bzgk/gb/newGbInfo?hcno=4C13F521FD6ECB6E5EC026FCD779986E&refer=outter . (1) Mathematics (2) Information Science and System Science (a) Basic Disciplines of Information Science and System Science • Information Theory • Cybernetics • System Theory (b) Systemology • Chaos • General System Theory • Dissipative Structure Theory • Synergetics • Catastrophe Theory • Hypercyclic Theory (c) Control Theory • Large System Theory • System Identification • State Estimation • Robust Control (d) System Evaluation and Feasibility Analysis • System Evaluation and Feasibility Analysis (e) System Engineering Methodology • System Modeling (f) System Engineering • System Engineering (3) Mechanics (4) Physics (5) Chemistry (6) Astronomy (7) Earth Sciences (8) Biology (9) Agriculture (10) Forestry (11) Animal Husbandry and Veterinary Science (12) Fisheries Science (13) Chemistry (14) Basic Medicine (15) Clinical Medicine (16) Preventive Medicine and Hygiene (17) Military Medicine and Special Medicine (18) Pharmacy (19) Traditional Chinese Medicine and Chinese Materia Medica (20) Basic Disciplines of Engineering and Technical Sciences (21) Surveying and Mapping Science and Technology (22) Materials Science (23) Mining Engineering Technology (24) Metallurgical Engineering Technology (25) Mechanical Engineering (26) Power and Electrical Engineering (27) Energy Science and Technology (28) Nuclear Science and Technology (29) Electronics, Communication and Automatic Control Technology (30) Computer Science and Technology (31) Chemical Engineering (32) Textile Science and Technology (33) Food Science and Technology (34) Civil engineering and construction (35) Water Conservancy Project (36) Transportation Engineering (37) Aeronautics and Astronautics Science and Technology (38) Environmental Science and Technology (39) Safety Science and Technology (40) Management (41) Marxism (42) Philosophy (43) Religious Studies (44) Linguistics (45) Literature (46) Art Studies (47) History (48) Archaeology (49) Economics (50) Political Science (51) Jurisprudence (52) Military Science (53) Sociology (54) Ethnology (55) Journalism and Communication Studies (56) Library, Information and Documentation (57) Sports Science (58) Statistics Note: Due to space constraints, only a subset of the complete discipline tree is shown above. The full classification tree containing all disciplines is available in the code repository. ⬇ SYSTEM_MESSAGE = """ You are an expert in academic paper review . Your task is to assess whether a candidate is suitable for reviewing a target paper . """ USER_TEMPLATE = """ Target Paper : Title : {} Abstract : { target_paper []} The five representative papers of the candidate : Please provide a confidence score indicating the suitability of the candidate for reviewing the target paper , based on the candidate ’ s research field , methods , and the alignment of the topic with the target paper . Output format : Only output an integer between 0 and 10, representing the confidence level (10 indicates a perfect match , 0 indicates no match at all ). Do not output any other content . """ Figure 6 . Prompt for LLM-as-Judge Baseline Method Paper OAG ID 643e5d140746dc40e37132a4 Author OAG IDs 63b015c684ab04bd7fc3601a, 65c9ced7671a2719984cfa1d, 653291ef1ae766ad0180d649, 53f42ca2dabfaee02ac56175, 54409165dabfae805a6d2900 Author ORCIDs 0000-0002-5962-4008, 0000-0002-6151-6858, 0000-0002-7268-9041, 0000-0002-4355-8368, 0000-0003-3100-1092 (optional) Reviewer OAG IDs 53f4535fdabfaefedbb459b9, 62e495fcd9f204418d6ba64d Reviewer ORCIDs 0000-0002-1147-766X, 0000-0002-2845-8617 (optional) Unqualified Candidate 54593219dabfaeb0fe344e7a, 640113cc45b36d5cbc41df62, 6403a6817691d561fb1f957b, 56319e4745ceb49c5e2136d8, OAG IDs 542d5076dabfae11fc45b850, 63afa47a84ab04bd7fb9a239, 64ca59ff75f2d36822961435, 670a2b79f4d6c71db9073468 Unqualified Candidate None, 0000-0002-9899-7659, 0000-0003-1977-1545, 0000-0002-3807-3171, 0000-0003-2421-7290, None, ORCIDs (optional) 0000-0002-6742-1964, None Potential 64049d0b7691d561fb35e34f, 64902f4f58a6797656b4e24c, 53f3acd6dabfae4b34b0283b, 53f428c9dabfaeb1a7b607e0 Candidate OAG IDs 53f43a17dabfaeee229cd2d0, 53f488e4dabfaec09f2aec95, 53f44eb6dabfaec09f1e9159, 53f43daddabfaee2a1d1dd42 Potential Candidate 0009-0001-6077-3924, 0000-0003-2442-3246, 0000-0002-4524-993X, 0000-0003-3689-8433, None, None, None, ORCIDs (optional) 0000-0001-5659-6841 Editor OAG ID 53f43a11dabfaeecd6980dd1 Editor ORCID None (optional) L1 Category Computer Science and Technology L3 Category Computer Neural Networks Table 4 . Example record from the OmniReview dataset showing peer review metadata with cross-referenced identifiers from Open Academic Graph (OAG) and ORCID. The record includes paper identification, author and reviewer information with their respective OAG IDs and ORCIDs (where available), candidate reviewer pools (unqualified and potential), editor details, and subject categorization at both L1 and L3 levels. Paper OAG ID 643e5d140746dc40e37132a4 Paper Title Multilevel comparison of deep learning models for function quantification in cardiovascular magnetic resonance: On the redundancy of architectural variations Keywords cardiovascular magnetic resonance, MRI, artificial intelligence, deep learning, cardiac image segmentation Paper Abstract BackgroundCardiac function quantification in cardiovascular magnetic resonance requires precise contouring of the heart chambers. This time-consuming task is increasingly being addressed by a plethora of ever more complex deep learning methods. However, only a small fraction of these have made their way from academia into clinical practice. In the quality assessment and control of medical artificial intelligence, the opaque reasoning and associated distinctive errors of neural networks meet an extraordinarily low tolerance for failure.AimThe aim of this study is a multilevel analysis and comparison of the performance of three popular convolutional neural network (CNN) models for cardiac function quantification.MethodsU-Net, FCN, and MultiResUNet were trained for the segmentation of the left and right ventricles on short-axis cine images of 119 patients from clinical routine. The training pipeline and hyperparameters were kept constant to isolate the influence of network architecture. CNN performance was evaluated against expert segmentations for 29 test cases on contour level and in terms of quantitative clinical parameters. Multilevel analysis included breakdown of results by slice position, as well as visualization of segmentation deviations and linkage of volume differences to segmentation metrics via correlation plots for qualitative analysis.ResultsAll models showed strong correlation to the expert with respect to quantitative clinical parameters (rz’ = 0.978, 0.977, 0.978 for U-Net, FCN, MultiResUNet respectively). The MultiResUNet significantly underestimated ventricular volumes and left ventricular myocardial mass. Segmentation difficulties and failures clustered in basal and apical slices for all CNNs, with the largest volume differences in the basal slices (mean absolute error per slice: 4.2 ± 4.5 ml for basal, 0.9 ± 1.3 ml for midventricular, 0.9 ± 0.9 ml for apical slices). Results for the right ventricle had higher variance and more outliers compared to the left ventricle. Intraclass correlation for clinical parameters was excellent ( \geq 0.91) among the CNNs.ConclusionModifications to CNN architecture were not critical to the quality of error for our dataset. Despite good overall agreement with the expert, errors accumulated in basal and apical slices for all models. Year 2023 Venue FRONTIERS IN CARDIOVASCULAR MEDICINE Reference IDs 617b66765244ab9dcbb6a772, 5c0f8b00da562944ac9c4db1, 58d82fcbd649053542fd5e6e, 5d1c7ccb3a55ac8c230aa97c, 5c04965717c44a2c74707951, 5c0f8cfdda562944aca09225, 622861f15aee126c0f853993, 627cd7a35aee126c0f3ff10f, 62652e325aee126c0fcfa6be, 5de0ec7e3a55acaf316bc74d, 59cb0d990cf23f44bec69750, 5fb255c691e01186d3f5df69, 62f6034590e50fcafdf6fc48, 62f23b8890e50fcafdca585c, 60ffe0c35244ab9dcb2cdc91, 612a13c15244ab9dcb1362f3, 5dca89823a55ac77dcb0213a, 5db92bcf47c8f7664623343a, 621865e35aee126c0fa5818e, 53e9a76db7602d97030a5fa6, 5c61603ce1cd8eae1501d8f9, 5fd0a271d4150a363cdde009, 60cdcb2de4510cd7c8655ca3, 5c6fc844e1cd8e35d214587d, 5550417d45ce0a409eb3bc08, 61dd5fae5244ab9dcb8bcd38, 5f48dde79fced0a24beafa07, 5c0f8893da562944ac975086, 573698016e3b12023e6da477, 61c301b85244ab9dcb04a914, 5f954e8e9fced0a24b179a94, 55a537a165ceb7cb02e4c52f, 5c136c7fda56295a08a8fead, 5dca89783a55ac77dcb01e84, 56d8e6c8dabfae2eee365661, 605aa2a4e4510cd7c86be281 Table 5 . Example paper metadata from Open Academic Graph Figure 7 . Case study of summary for a query paper and 3 review candidates. Appendix C Details of Baseline Methods For the TF-IDF, SciBERT, SPECTER2, BGE-M3, and Qwen3-Embedding-4B models, the final similarity score for each paper-reviewer pair is computed through a two-step process. First, we calculate the cosine similarity between the query paper’s embedding and the review candidate’s embedding, where the candidate’s embedding is derived as the mean of embeddings from all their publications. Second, we collect the cosine similarity value distribution from actual paper-reviewer pairs alongside pairs consisting of papers and candidates who are semantically most distant within the discipline taxonomy. Based on this distribution, we train an isotonic regression model to map the cosine similarity values into the 0-1 range, thereby generating a confidence score prediction. Similarly, we follow the original pipeline of the Chain-of-Factors Paper-Reviewer Matching to get the score distribution of the positive and negative samples and trained a isotonic regression model to project the scores into the 0-1 range to predict the confidence score. For RGCN, we initialize the nodes with the Qwen3-Embedding-4B embeddings of papers and scholars and a DNN and softmax are used to convert the embedding output by the graph into scores. For LLM-as-judge method, we provide the titles and abstracts of query paper and 5 latest publications of the review candidates for LLM and the LLM need to give the match score from 0 to 10(0 indicates no match at all, while 10 indicates a perfect match). The prompt we use is shown in Figure 6 . Appendix D Case Study for OmniReview Table 4 presents an example of peer review records from our OmniReview dataset. By leveraging identifiers from Open Academic Graph (OAG) and ORCID, we can access these databases to retrieve comprehensive metadata. Table 6 showcases an example of scholar metadata from Open Academic Graph, while Table 5 illustrates the paper metadata structure within the same database. Appendix E Case Study for LLM Summaries in Pro-MMoE This case study, shown in Figure 7 , demonstrates how LLM-generated summaries efficiently differentiate peer review candidates for a neuroscience paper on oculomotor remapping by exposing critical disparities in domain alignment, technical accuracy, and contextual relevance. The real reviewer’s summary precisely aligns with the paper’s cognitive neuroscience focus—emphasizing visual attention, perceptual stability, and fMRI methodology, while directly addressing its specific limitations like statistical controls and behavioral validation gaps. In stark contrast, the unqualified candidate’s summary fixates on irrelevant domains such as machine learning applications and robotics, entirely overlooking the paper’s core scientific context, whereas the misleading candidate’s summary compounds inaccuracy by erroneously attributing diffusion MRI and tractography techniques to a study exclusively using fMRI phase-encoding. The LLM’s structured analysis thus objectively isolates the real reviewer’s unique suitability by filtering out superficial or fabricated expertise, proving its value in rapidly identifying reviewers whose domain-specific knowledge and methodological precision match a paper’s actual requirements. Scholar OAG ID 53f4535fdabfaefedbb459b9 Name Maria A. Zuluaga Org EURECOM ORCID 0000-0002-1147-766X Table 6 . Example scholar metadata from Open Academic Graph Appendix F Prompts for LLM Summarizer Figure 8 and Figure 9 demonstrate prompts we use for LLM summaries profiling. We firstly utilize LLM to generate summaries for query papers and reviewers’ selected papers based on title and abstract. Then the LLM provides reviewers’ summaries based on their selected papers. ⬇ SYSTEM_MESSAGE = ’’’ You are an expert academic reviewer assistant . Generate a concise , critical summary of the research paper based ONLY on the title and abstract provided . Your summary must help peer reviewers quickly assess the paper ’ s suitability for publication . ** Critical Requirements :** 1. ** STRICT WORD LIMIT **: 150-180 words maximum . Every word must add value . 2. ** SOURCE RESTRICTION **: Use ONLY the provided title and abstract . Never invent details , methods , results , or context not explicitly stated . 3. ** REVIEWER - FOCUSED CONTENT **: - Core research question and its significance - Methodology assessment ( rigor , appropriateness based on abstract description ) - Key findings and their reliability - Novel contributions vs . existing literature - Major limitations explicitly mentioned - Overall publication recommendation likelihood 4. ** TONE \& STYLE **: - Objective , professional academic tone - Critical but constructive evaluation - Avoid promotional language or excessive praise - Use precise terminology from the field 5. ** STRUCTURE **: Single coherent paragraph with logical flow : Problem Methods Findings Contribution Limitations Assessment . ** WARNING **: If the abstract lacks crucial details ( methods , limitations , contributions ), explicitly state these gaps as review concerns . Never compensate with external knowledge . ’’’ USER_TEMPLATE = """ Please provide a critical review summary for peer reviewers based on the following paper title and abstract . Strictly adhere to the word limit and guidelines . Title : "{}" Abstract : "{}" """ Figure 8 . Paper Summarization Prompt ⬇ SYSTEM_MESSAGE = ’’’ You are an expert academic analyst . Your task is to analyze article summaries written by a candidate to identify their research domain expertise patterns . Focus exclusively on their demonstrated understanding of different research fields , key problems , and domain - specific significance . ** Core Analysis Dimensions :** - ** Domain Coverage **: Range of research fields covered across the summaries - ** Field - Specific Insight **: Depth of understanding demonstrated in each domain ( terminology , key challenges , seminal works ) - ** Problem Recognition **: Ability to identify domain - specific research problems and their importance - ** Contribution Evaluation **: Skill in assessing what constitutes meaningful contribution within specific fields - ** Field Context Awareness **: Understanding of how research fits within broader domain landscape - ** Terminology Mastery **: Appropriate use of domain - specific concepts and jargon - ** Cross - Domain Connections **: Recognition of relationships between different research areas ** Output Requirements :** 1. ** Primary Research Domains ** ( bullet points ): 2-3 fields where candidate shows strongest expertise 2. ** Secondary Domains ** ( bullet points ): Additional fields with demonstrated understanding 3. ** Domain Expertise Pattern ** ( concise paragraph ): How they approach and evaluate different research fields 4. ** Field - Specific Strengths ** (2 bullet points ): Most notable domain understanding capabilities ** Critical Guidelines :** - Base analysis ONLY on observable domain knowledge from the summaries - Focus on field - specific understanding rather than general analytical skills - Prioritize research domain expertise over writing style or structural preferences - Avoid inferring methodological assessment abilities from summary content - Maintain descriptive neutrality - characterize domain expertise without judgment - Word limit : 200 words maximum ’’’ USER_TEMPLATE = ’’’ Please analyze the following article summaries and provide a comprehensive characterization of their distinctive traits and working patterns . Focus on identifying observable characteristics rather than making suitability judgments . ** Candidate ’ s Articles Summaries :** {} ’’’ Figure 9 . Review Candidate Summarization Prompt