Automatic Detection of Research Values from Scientific Abstracts Across Computer Science Subfields

Hang Jiang, Tal August, Luca Soldaini, Kyle Lo, Maria Antoniak · 2025

Read Automatic Detection of Research Values from Scientific Abstracts Across Computer Science Subfields by Hang Jiang, Tal August, Luca Soldaini, Kyle Lo, Maria Antoniak (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16390.txt.

Introduction

Hang Jiang and co-authors studied the values that computer science papers express, and built tools to detect them at scale in 2025. Values are the basic beliefs that guide what researchers do, and the field's fast growth has reshaped its many subfields.

Past work read values by hand in a small set of machine learning papers. The new team wrote a detailed scheme with ten values, then trained classifiers to apply it widely. They ran the tools over more than 226,000 abstracts from thirty-two subfields and eighty-six venues across ten years. The study maps which values each community promotes, and how those emphases may shift over time.

Catalog record

CREATORHang Jiang, Tal August, Luca Soldaini, Kyle Lo, Maria Antoniak
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-16390

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automatic Detection of Research Values from Scientific Abstracts Across Computer Science Subfields Hang Jiang, Tal August, Luca Soldaini, Kyle Lo, Maria Antoniak Abstract Keywords: Research Values, Innovation, Interdisciplinary Studies, Text Mining, LLM 1 Background The field of computer science (CS) has rapidly evolved over the past few decades, providing computational tools and methodologies to different fields and forming new interdisciplinary communities. This growth in CS has significantly impacted institutional practices and relevant research communities. Therefore, it is crucial to explore what specific research values , known as basic and fundamental beliefs that guide or motivate research attitudes or actions , CS-related research communities promote. We are inspired by prior research that has manually analyzed research values from a small sample of machine learning papers ( Birhane et al., 2022 ) . We extend that study by annotating additional data and testing automatic methods for value detection, as no prior work has studied the automatic detection of research values from large-scale scientific texts across different research subfields. This paper introduces a detailed annotation scheme featuring ten research values that guide CS-related research 1 1 1 Data and code are available at: https://github.com/hjian42/CSValues . Based on the scheme, we build value classifiers to scale up the analysis and present an analysis of 226,600 paper abstracts from 32 CS-related subfields and 86 popular publishing venues over ten years. 2 Research Value Scheme Building on Birhane et al., 2022 , we define that a sentence encodes research value(s) if there is any expression of why something is desirable or undesirable. Through qualitative coding, three authors of the paper manually identified sentences discussing research values from 100 sampled abstracts and arrived at ten popular research values across different CS-related subfields. These values include (1) Performance, (2) Novelty, (3) Efficiency, (4) Generalizability, (5) Openness, Reproducibility, Collaboration, & Future Work (Openness), (6) Simplicity, (7) Phenomenon Understanding & Theoretical Grounding (Understanding), (8) Fairness, Bias, Privacy & Ethics (Integrity), (9) Societal Implications (Society), (10) Usability. Each research value is defined in a codebook with representative examples provided. 3 Data Sampling & Annotation We first curate a list of 86 popular CS-related research venues by consulting csrankings.org and sending questionnaires to 17 doctoral researchers from different communities. We then downloaded 226,600 papers from 2013 to 2022 with the internal PyS2 library. We randomly sampled 12 abstracts from each venue and randomly picked one sentence from each abstract for the annotation task, resulting in 1,032 sentences for annotation. Two authors of the paper double-annotated these sentences and held rounds of discussions to determine a single “consensus label” for whether each sentence contains a given research value. Due to the challenging nature of the value detection task, two authors annotate one value at a time. We split the annotated data into train (40%), validation (30%), and test (30%) sets. 4 Task Formulation & Method We formulate the research value detection task as follows: given a sentence s from an abstract, our target is to identify if a specific value v_{i} from a set of values V is encoded in the sentence. We build lexicon-based value classifiers and report their performance on the test set. We find that lexicon-based methods achieve decent accuracy and F1 scores, with F1 ranging from 0.71 to 0.88 across ten research values. We also experiment with LLM prompting techniques ( Ziems et al., 2024 ) for value detection. Surprisingly, we find that lexicon-based methods outperform LLM-based few-shot prompting methods on at least five values. LLM-based methods slightly perform better in Novelty and Simplicity. We therefore rely on the lexicon-based methods for our subsequent analysis. 5 Analysis & Findings By running value classifiers on all the abstracts, we are able to characterize individual research communities by their most frequent and most distinctive values. First, we find that researchers in different CS subfields tend to emphasize different research values in abstracts. Traditional fields such as Computer Architecture and Operating Systems are known to emphasize Efficiency and Performance. On the other hand, AI-related subfields such as Computer Vision highlight Performance, Generalizability, and Novelty in their abstracts. Furthermore, we have noted that interdisciplinary CS+X fields such as Computational Biology & Bioinformatics, Ethics of Computer Science, and Human-Computer Interaction (HCI), compared to traditional CS domains, are more likely to mention Openness and less likely to discuss Simplicity, Efficiency, and Novelty. The variation in these values demonstrates the diversity within CS communities in terms of their research practices. Secondly, we found that specific values are experiencing increased emphasis in abstracts over time. Between 2013 and 2022, an uptick in the discussion around seven values was observed in the abstracts across AI-related subfields, which include Robotics, Computer Vision, Natural Language Processing, and Machine Learning. This implies that AI-related researchers have adapted their practices to more clearly articulate the research values within their abstracts over time. However, this trend isn’t consistent across all CS subfields. Intriguingly, interdisciplinary CS+X subfields have shown an increasing emphasis on values such as Openness, Performance, and Understanding in their abstracts over time. On the other hand, subfields in Computer Theory have demonstrated a decreasing pattern in discussions about Simplicity over time, suggesting that theories may have become more advanced and complex. Areas connected with traditional CS Systems have not shown any notable trends with regards to research values over time, showing that there is an established writing practice in these subfields. Third, we examine the linguistic patterns used to communicate research values in abstracts. We find that these values are occasionally expressed directly with phrases like “novel solution”, “simple framework”, or “generalizable method”. However, more often they are subtly represented through conventional linguistic structures such as “propose a technique” or “create a dataset” to imply novelty, and “works in a variety of tasks” or “adapts to unseen tasks” to suggest generalizability. At last, we study the pairwise co-occurrence patterns of values through pointwise mutual information (PMI) and find that Openness and Simplicity tend to co-occur most frequently with the other five values. In addition, Novelty and Efficiency tend to co-occur with each other in abstracts. 6 Conclusion & Future Directions Our paper presents a detailed value scheme and conducts a study to uncover research values from scientific abstracts across different CS subfields over time. In the future, we plan to expand the study by (1) studying if these values co-occur and are correlated with citations, funding sources, author seniority, and co-authoring patterns, (2) extending the current analysis beyond abstracts such as introduction and even full papers, (3) extending the value scheme to non-CS fields to understand how values are expressed across disciplines. Datasheet Documentation This following section documents how we created the ResearchValues dataset following the datasheets for datasets guideline ( Gebru et al., 2021 ) . 1. For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description. The purpose of this dataset and study was to explore and analyze the research values promoted across different computer science (CS) subfields. We developed a detailed annotation scheme for 10 key CS research values and analyzed the distribution of these values in a large corpus of 226,600 CS paper abstracts. The goal was to characterize

Read the complete text