Research Borderlands: Analysing Writing Across Research Cultures

Shaily Bhatt [email protected], Tal August [email protected], Maria Antoniak [email protected] · 2025

Read Research Borderlands: Analysing Writing Across Research Cultures by Shaily Bhatt [email protected], Tal August [email protected], Maria Antoniak [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00784.txt.

Introduction

Shaily Bhatt, Tal August, and Maria Antoniak study how writing norms differ across research cultures, in a research paper from 2025. Each field has its own habits of structure, style, argument, and citation. Scholars who move between fields learn those norms the hard way, and language models often flatten them.

The team interviewed interdisciplinary researchers and built a framework of the norms that vary. They turned the framework into computer measures, then applied them to human-written papers at scale. The measures surfaced hidden norms in real writing. Applied to model output, they showed that language models tend to homogenize style across fields.

Catalog record

CREATORShaily Bhatt [email protected], Tal August [email protected], Maria Antoniak [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE102 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00784

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Research Borderlands: Analysing Writing Across Research Cultures Shaily Bhatt [email protected], Tal August [email protected], Maria Antoniak [email protected] Abstract Improving cultural competence of language technologies is important. However most recent works rarely engage with the communities they study, and instead rely on synthetic setups and imperfect proxies of culture. In this work, we take a human-centered approach to discover and measure language-based cultural norms, and cultural competence of LLMs. We focus on a single kind of culture, research cultures , and a single task, adapting writing across research cultures . Through a set of interviews with interdisciplinary researchers, who are experts at moving between cultures, we create a framework of structural, stylistic, rhetorical, and citational norms that vary across research cultures. We operationalise these features with a suite of computational metrics and use them for (a) surfacing latent cultural norms in human-written research papers at scale; and (b) highlighting the lack of cultural competence of LLMs, and their tendency to homogenise writing. Overall, our work illustrates the efficacy of a human-centered approach to measuring cultural norms in human-written and LLM-generated texts. *[inlinelist,1]label=(), 1 Introduction What makes an NLP paper an NLP paper? Is it when a paper focuses on language technologies? Is it that the authors are NLP experts or that they use NLP methods? What if it discusses user perceptions of an NLP technology — does that make it an HCI paper? Maybe it is the descriptive “Figure 1” or is it the two-column ACL format? Figure 1: We survey and interview interdisciplinary researchers (§ 3 ) to develop a framework of writing norms that vary across research cultures (§ 4 ) and operationalise them using computational metrics (§ 5 ). We then use this evaluation suite for two large-scale quantitative analyses: (a) surfacing variations in writing across 11 communities (§ 6 ); (b) evaluating the cultural competence of LLMs when adapting writing from one community to another (§ 7 ). P5 on adapting writing: There’s a way to write … that makes it way more likely a paper with the same results gets accepted or not. P2 on tacit norms: All these norms because they’re not stated, you can only speculate based on observation. P6 on framing: You can have findings that absolutely blow people away that will get lost if your introduction is not framed in the right way. Table 1: Quotes from our interviewees (senior interdisciplinary scholars) on writing across research cultures. These factors are examples of the norms, expectations, and values that characterize communities ( Kroeber and Kluckhohn 1952 ) and manifest in a community’s communication Deardorff 2009 . Evaluating and aligning large language models (LLMs) to such cultural norms, which are often undocumented, is a difficult but urgent task Hovy and Yang 2021 ; Sorensen et al. 2024 . However, in most recent works, the concept of “culture” remains vaguely defined, if at all Adilazuarda et al. 2024 . Many evaluations rely on synthetic setups that lack grounding in specific tasks or cultural contexts, and operationalise culture through easy-to-use and imperfect proxies like language or nationality ( Zhou et al. 2025 ; Qadri et al. 2025 ) . We, instead, take a human-centered approach. We tackle the definition of the simultaneously vast and highly contextual question of defining and evaluating culture by zooming in on one specific instantiation of culture, research cultures , focusing on a specific task, adapting writing in research papers , and centering the community members , interdisciplinary researchers who are experts in writing for specific research communities. Using mixed-methods, we develop and operationalise a framework of cultural norms that holistically characterize the writing from different scientific communities, and we measure LLMs’ competence in adhering to these research cultural norms. Figure 1 shows a complete overview of our study. 1 1 1 In the rest of the paper, we use “scientific communities” and “research communities” interchangeably. We use “research cultures” to refer to the culture (encompassing norms, values, and expectations) of research communities. We rely on the definition of cultural competence provided by Deardorff 2009 ; which, in context of our task, loosely translates to the ability of an LLM to adhere to a set of cultural norms. We survey (N=78) and interview (N=10) interdisciplinary researchers to understand the differences in cultural norms in writing across research communities. We choose interdisciplinary researchers as they are experts at navigating between multiple communities. The survey (§ 3.1 ) confirms the ecological validity of our writing adaptation task and guides our choice of proxy to operationalise culture. Our interviews (§ 3.2 ) elicit researchers’ perceived differences in writing norms across communities. Through qualitative analysis of the interviews, we develop a framework of language-based writing norms that vary across research cultures (§ 4 ). We operationalise this framework using computational metrics and models into an evaluation suite (§ 5 ) which we use in two large-scale quantitative analyses of human-written and LLM-generated scientific writing. First, we analyse a corpora of research papers from different scientific communities, surfacing their variations (§ 6 ). We find that our metrics recover differences in writing across research cultures at scale, including anecdotal observations of our interviewees. Second, reflecting on the growing use of LLMs in scientific writing ( Liang et al. 2024 ) , we evaluate the research cultural competence of LLMs (§ 7 ). We find that current LLMs struggle to adhere to cultural norms and tend to homogenise writing across communities. We contribute to work in evaluating cultural competence by illustrating an alternative, human-centered method of eliciting and measuring cultural norms in human-written and LLM-generated text. We also open-source 2 2 2 github.com/shaily99/research_borderlands our evaluation suite for future research in science-of-science and evaluation of cultural competence of scientific writing tools. 2 Related Work 2.1 Understanding Research Communities Most prior works in understanding research communities have either focused on a single differentiating feature across many communities, or on understanding a specific aspect in one community. For example, Lucy et al. 2023 analyse the lexical choices and specialized jargon used in different research communities, computationally and at a large scale to measure the specificity of a community’s written work. Many prior works have also focused on deep explorations of citational practices, within and across fields ( Chen et al. 2025 ; Jurgens et al. 2018 ; Leydesdorff et al. 2019 ) . On the more qualitative side, Birhane et al. 2022 and Jiang et al. 2025 study the values encoded in machine learning research, Michael et al. 2023 survey the values and beliefs of the NLP research community, Gururaja et al. 2023 study the factors that have shaped NLP as a field, and Linxen et al. 2021 study the demographic biases in studies at the CHI. Some works have also focused on the framing of specific terms within a research community, such as “democratization” ( Subramonian et al. 2024 ) and “bias” ( Blodgett et al. 2020 ) , or “intersectionality” ( Ovalle et al. 2023 ) . We add to this body of work by eliciting and analysing a wide variety of writing norms that vary across research communities using mixed-methods. 2.2 LLM-tools for Scientific Research Recently, there has been a rapid rise in building tools to assist researchers in

Read the complete text