Research Community Perspectives on “Intelligence” and Large Language Models
Bertram Højer, Terne Sasha Thorn Jakobsen, Anna Rogers, Stefan Heinrich · 2025
Read Research Community Perspectives on “Intelligence” and Large Language Models by Bertram Højer, Terne Sasha Thorn Jakobsen, Anna Rogers, Stefan Heinrich (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20959.txt.
Introduction
Bertram Højer, Terne Jakobsen, Anna Rogers, and Stefan Heinrich survey researchers on what intelligence means. The word frames much language tech work, yet its sense stays vague. Their survey drew full replies from 303 researchers across five fields. Fields spanned language tech, machine learning, mind science, linguistics, and brain science.
Three marks of intelligence won broad assent: generalization, adaptability, and reasoning. Only twenty-nine percent called current language systems intelligent, a minority view. A still smaller share, about sixteen percent, named building intelligent systems as a goal. Those who did hold that goal were more apt to call current systems intelligent. The authors map a field split on its core term.
Catalog record
| CREATOR | Bertram Højer, Terne Sasha Thorn Jakobsen, Anna Rogers, Stefan Heinrich |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20959
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Research Community Perspectives on “Intelligence” and Large Language Models Bertram Højer, Terne Sasha Thorn Jakobsen, Anna Rogers, Stefan Heinrich Abstract Despite the widespread use of “artificial intelligence” (AI) framing in Natural Language Processing (NLP) research, it is not clear what researchers mean by “intelligence”. To that end, we present the results of a survey on the notion of “intelligence” among researchers and its role in the research agenda. The survey elicited complete responses from 303 researchers from a variety of fields including NLP, Machine Learning (ML), Cognitive Science, Linguistics, and Neuroscience. We identify 3 3 criteria of intelligence that the community agrees on the most: generalization, adaptability, & reasoning . Our results suggests that the perception of the current NLP systems as “intelligent” is a minority position (29%). Furthermore, only 16.2% of the respondents see developing intelligent systems as a research goal, and these respondents are more likely to consider the current systems intelligent. 1 Introduction Whether machines can be considered intelligent has long been a question of interest in both academia and industry. Researchers have a rich tradition of devising intelligence tests for machines, starting from the Turing test Turing (1950) , and the industry increasingly markets ‘‘smart’’ and ‘‘intelligent’’ systems. A recent iteration hereof are LLM-based systems, also referred to as ‘‘generative AI’’, 1 1 1 We are only concerned with systems based on Large Language Models (LLMs), i.e., models trained on large text corpora that are used for transfer learning Rogers and Luccioni (2024) . We use the term “AI” in line with its common use in the field, but we note that it entails that the term “intelligence” is applicable, which, as our results suggest, is not accepted by the majority of the respondents. which have been suggested to show “sparks of artificial general intelligence” Bubeck et al. (2023) . Figure 1: Correlation between criteria that the survey respondents selected as relevant for their notion of “intelligence”. Darker edges indicate stronger correlations, larger nodes indicate higher relevance. Only edges with \phi>|0.1| are shown. Irrespective of one’s position on whether systems based on current ML technology can be considered intelligent, there is undeniably an increase in the phenomenon we refer to as seeming intelligence : the user’s perception that a system is intelligent, based on their interaction with that system. This perception arises from the Eliza effect: humans have a strong tendency to attribute to their conversational partners, even simple chatbots, “all sorts of background knowledge, insights and reasoning ability” Weizenbaum (1966) , and the perception of intelligence is a driver of anthropomorphization, irrespective of whether a system is represented via an avatar or not Kim and Im (2023) . The tendency to anthropomorphize AI systems is not limited to the lay public or marketing by tech companies: 2 2 2 OpenAI describe their base class of models as ” (…) models without instruction following that can understand (…) natural language and code ”, GPT-4 is better ” … thanks to its broader general knowledge and advanced reasoning capabilities.” and the o1 class of models ” think before they answer ”. Mitchell (2021) points out that even researchers mislead themselves by using terms like “understanding”, which have long been criticized as “wishful mnemonics” McDermott (1976) . In particular, we have a longstanding tradition of evaluating “understanding” ( Wang et al., 2018 ; Wang et al., 2020 , e.g.) , “reasoning” ( Vilares and Gómez-Rodríguez, 2019 ; Srivastava et al., 2023 , e.g.) , “behaviors” ( Chang and Bergen, 2023 , e.g.) and “capabilities” ( Chen et al., 2024 ; Srivastava et al., 2023 , e.g.) . Given the above tendency towards anthropomorphization and the long history of “wishful mnemonics” in the academic literature, the goal of this study is to provide a snapshot of how researchers think about “intelligence” two years after the release of ChatGPT OpenAI (2022) . We pose the following research questions: RQ1 What are key criteria for a system to be considered “intelligent”, and to what degree do researchers in different fields, occupations, and career stages agree? RQ2 What is the perception of the “intelligence” of current LLM-based systems such as ChatGPT? RQ3 What role, if any, does the research agenda of respondents play in the notion of “intelligence”? Our results indicate a high level of coherence across groups in terms of the criteria that are relevant for intelligence. Independent of one’s field of research, occupation, and career stage, most researchers believe that generalization , adaptability , and reasoning are key aspects of intelligence ( RQ1 , illustrated in Figure 1 ). We additionally observe a general skepticism regarding the attribution of intelligence to LLM-based systems, although it decreases for “future systems based on a similar technology” ( RQ2 ). For RQ3 , we find that only a small subset of respondents consider it their goal to build intelligent technology, with the majority rather aiming to add to the scientific knowledge. Interestingly, the survey shows that those who consider ‘creating intelligent systems’ as their research goal are also more likely to attribute intelligence to the current systems ( RQ2 , RQ3 ). 2 Background and Related Work 2.1 Survey methodology When beliefs are implicit rather than clearly expressed in words or actions, administering surveys is often the only way of studying the attitudes of a large set of people Robson (1999) . This methodology is by no means perfect: the challenges of online self-administered surveys include low response rates and the possibility of misunderstandings, inaccurate self-reporting, and responses not made in good faith. Still, online surveys are a widely used methodology that allows us to systematically capture the beliefs held by a broad set of researchers, in different fields, occupations, and career stages, and analyze responses statistically Phillips (2017) . Among the prior surveys in the NLP community are the meta-survey of researcher beliefs Michael et al. (2023) and the survey of views on the factors important for paper-reviewer matching Jakobsen and Rogers (2022) . These surveys were distributed similarly to ours and received 327 and 180 valid responses respectively. Surveys have also been deployed more broadly to study researchers’ beliefs about the current state of AI, but these have mostly focused on impact predictions rather than beliefs about the actual technology. For example, the recent Pew survey on how US adults and experts view AI Pasquini et al. (2025) does not elicit how respondents define or use that term. Neither are definitions the focus of ‘timeline to AGI’ surveys. 3 3 3 https://wiki.aiimpacts.org/ai_timelines/predictions_of_human-level_ai_timelines/ai_timeline_surveys/ai_timeline_surveys Some of the aspects that we consider in this work were also in the AAAI 2025 presidential panel Rossi (2025) . 2.2 Defining “Intelligence” Criterion Survey definition Sources Prob. solving solving specific problems in a familiar domain Turing (1950) ; Mitchell (2021) Knowledge acqusition learning, understanding or gaining knowledge and skills Piaget (1952) ; Chollet (2019) Adaptability making sense of new environments and/or handling novel tasks Legg et al. (2007) ; Pfeifer (2006) Generalization successfully handling new types of data and situations Marcus (2003) ; Tenenbaum et al. (2011) Goal achievement accomplishing defined objectives and optimizing for specific tasks Piaget (1952) ; Van Gerven (2017) Reasoning logical inference - deductive, abductive etc. Legg et al. (2007) ; Lake et