FLeW: Facet-Level and Adaptive Weighted Representation Learning of Scientific Documents
Zheng Dou 1, Deqing Wang 1,3(✉), Fuzhen Zhuang 2,3, Jian Ren 1, Yanlin Hu 4 · 2025
Read FLeW: Facet-Level and Adaptive Weighted Representation Learning of Scientific Documents by Zheng Dou 1, Deqing Wang 1,3(✉), Fuzhen Zhuang 2,3, Jian Ren 1, Yanlin Hu 4 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07531.txt.
Introduction
Zheng Dou and 4 co-authors present FLeW in 2025, a method for representing scientific documents as embeddings that respect their facets. An embedding is a numeric summary used for search and recommendation. Existing methods either underuse citation signals and give one flat vector, split documents finely at high integration cost, or need hand-made task categories to adapt.
FLeW unifies the three approaches. A new triplet sampling method uses citation intent, whether a citation gives background, method, or results, plus citation frequency to strengthen training signals. Those intents partition each document into facet-level embeddings aligned with how scientific writing is structured. A simple weight search then blends the facet embeddings into a task-specific summary without extra task modules. The design targets better scholarly search and recommendation across fields.
Catalog record
| CREATOR | Zheng Dou 1, Deqing Wang 1,3(✉), Fuzhen Zhuang 2,3, Jian Ren 1, Yanlin Hu 4 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-07531
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FLeW: Facet-Level and Adaptive Weighted Representation Learning of Scientific Documents Zheng Dou 1, Deqing Wang 1,3(✉), Fuzhen Zhuang 2,3, Jian Ren 1, Yanlin Hu 4 Abstract Scientific document representation learning provides powerful embeddings for various tasks, while current methods face challenges across three approaches. 1) Contrastive training with citation-structural signals underutilizes citation information and still generates single-vector representations. 2) Fine-grained representation learning, which generates multiple vectors at the sentence or aspect level, requires costly integration and lacks domain generalization. 3) Task-aware learning depends on manually predefined task categorization, overlooking nuanced task distinctions and requiring extra training data for task-specific modules. To address these problems, we propose a new method that unifies the three approaches for better representations, namely FLeW. Specifically, we introduce a novel triplet sampling method that leverages citation intent and frequency to enhance citation-structural signals for training. Citation intents (background, method, result), aligned with the general structure of scientific writing, facilitate a domain-generalized facet partition for fine-grained representation learning. Then, we adopt a simple weight search to adaptively integrate three facet-level embeddings into a task-specific document embedding without task-aware fine-tuning. Experiments show the applicability and robustness of FLeW across multiple scientific tasks and fields, compared to prior models. 1 Introduction The rapid growth of scientific publications across diverse fields has spurred the need for high-quality document representations to support downstream tasks like classification, retrieval, and search [ 4 , 15 , 1 ] . Compared to general-purpose text, scientific documents exhibit unique relational structures and encapsulate more concentrated knowledge. Current approaches focus on these features of scientific documents for better representations while still facing challenges. First, contrastive training with citation-structural signals has shown effectiveness in scientific document representation learning [ 4 , 13 ] . However, they underutilize richer information from citation edges and generate single-vector representations, limiting their ability to capture fine-grained information. Second, fine-grained representation learning generates multiple vectors at the sentence or aspect level to capture detailed knowledge and information [ 12 , 14 ] . However, these methods either require costly integration or lack generalization across different fields. Third, task-aware learning also generates multi-vector representations, with each vector tailored to a specific task category [ 18 , 15 ] . But they need manually predefined task categorization and fail to capture nuanced differences within the same category. Additionally, such methods often require extra modules and additional training data to learn task-specific parameters. In this work, we propose F acet- L evel and Adaptiv e W eighted representation learning of scientific documents, named FLeW , which unifies citation-structural contrastive training, fine-grained multi-vector representation, and task-aware learning into a single framework and address their challenges. Specifically, FLeW uses a novel triplet sampling method to utilize citation intent (background, method, result) and citation frequency for more informative contrastive training. The classification of citation intent aligns with the general structure of scientific writing, serving as a generalized facet partition for fine-grained multi-vector learning with more applicability across different fields. FLeW adopts a simple weight search to use the weighted sum of facet-level representations as the final document representation adaptive to multiple tasks instead of task-aware fine-tuning. By integrating task information into facet weights, FLeW effectively captures task-specific differences and improves the applicability across tasks. In summary, the contributions of this work are as follows: • We propose a structural sampling method that uses citation intent and frequency information for enhanced citation-structural learning, along with a textual splitting method to divide triplet texts into faceted parts for better fine-grained textual representation. • We propose to adopt a simple weight search and use the weighted sum of facet-level representations as the final document representation adaptive to multiple tasks instead of task-aware fine-tuning. • Extensive experiments on a large-scale benchmark comprising 19 tasks and a citation recommendation dataset spanning 19 fields demonstrate that FLeW achieves superior performance compared to prior methods across multiple scientific tasks and fields. 2 Background Document representation learning is the process of encoding a given input document \mathcal{D} into a low-dimensional dense vector \boldsymbol{V} by an {Encoder} model. The generated vector, also known as embedding or representation, contains rich semantic and context information of the document and can be used for vector calculation in downstream tasks. BERT [ 5 ] is a pre-trained model with encoder-only architecture consisting of multiple layers of Transformer [ 16 ] to encode the tokens in a given input sequence. The final hidden state of special {[CLS]} token is commonly used as an aggregate representation of the input sequence. For a scientific document, the title provides a thorough overview of the paper’s topic and the abstract is a comprehensive summary of the entire text. Following prior work [ 4 , 13 , 15 ] , using the {title} and {abstract} separated by the {[SEP]} token as input effectively balances input length constraints with high-quality representation. Since the great success of BERT on various NLP tasks especially text understanding, we use BERT-based model as the {Encoder} and also take the final hidden state of the {[CLS]} token as output representation: \displaystyle\boldsymbol{V}=Encoder(title[SEP]abstract)_{[CLS]} (1) Our goal is to obtain such pre-trained {Encoder(s)} which can generate informative representation of any input scientific document (title and abstract) directly utilized for downstream tasks. Figure 1: Approach overview of FLeW. 3 Methodology 3.1 Structural Sampling A citation from a citing paper consists of one or several sentences referring to a specific aspect of the cited paper. This aspect, commonly referred to as citation intent, can be classified into three distinct categories: background, method, and result. Furthermore, the influence of cited papers within the same citing paper can vary significantly. An intuitive criterion is that the more frequently a cited paper is referenced within the citation contexts of a citing paper, the greater its influence on the citing paper. Hence, citation edges can both be classified into three intents and be influence-weighted based on citation frequency. As shown in Fig. 1 (a), we propose a novel triplet sampling method utilizing this information to sample facet-structural triplets from citation graph for enhanced citation-structural contrastive learning. Weighted Facet Subgraphs. We construct the original citation graph \mathcal{G}=(\mathcal{V},\mathcal{E}) , where \mathcal{V} represents the set of paper nodes, and \mathcal{E} denotes the set of citation edges between citing and cited papers. Each edge e\in\mathcal{E} is associated with citation contexts and annotated with citation intent labels ( \mathcal{T}=\{background\,(bg),method\,(mt),result\,(rs)\} ). Based on the intent labels, we extract three facet-specific subgraphs, \mathcal{G}_{bg},\mathcal{G}_{mt},\mathcal{G}_{rs} , where each subgraph captures citation edges corresponding to a specific intent. Formally, the facet subgraphs are defined as follows: \mathcal{G}_{x}=(\mathcal{V},\mathcal{E}_{x}),\quad\mathcal{E}_{x}=\{e\in\mathcal{E}\mid e^{\mathcal{T}=x}\},\quad x\in\{bg,mt,rs\}. (2) To further incorporate the influence of citation frequency, we assign weights to the edges in each facet subgraph. According to our proposed criterion of influence, for the same citing paper, the cited paper with more citations is more influential than those with less citations. The weight w(e) of an edge e\in\mathcal{E}_{x} is determined by the number of citation contexts |\mathcal{C}_{x}(e)| associated with the same intent x between citing and cited nodes. The resulting weighted facet subgraphs are formalized as: \mathcal{G}_{x}^{W}=(\mathcal{V},\mathcal{E}_{x},\mathcal{W}_{x}),\quad\mathcal{W}_{x}(e)=|\mathcal{C}_{x}(e)|,\quad x\in\{bg,mt,rs\}. (3) Subgraph Embedding and Neighborhood Sampling. We utilize PyTorch-BigGraph (PBG) [ 9 ] to train and generate subgraph embeddings in the latent space derived from the citation-structural information. As PBG cannot