Is a Peeled Apple Still Red? Evaluating LLMs’ Ability for Conceptual Combination with Property Type

Seokwon Song, Taehyun Lee, Jaewoo Ahn, Jae Hyuk Sung, Gunhee Kim · 2025

Read Is a Peeled Apple Still Red? Evaluating LLMs’ Ability for Conceptual Combination with Property Type by Seokwon Song, Taehyun Lee, Jaewoo Ahn, Jae Hyuk Sung, Gunhee Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-06086.txt.

Introduction

Song and co-authors test whether language models understand how combined concepts gain and lose properties, in a paper published in 2025. Peel an apple and it stays sweet but stops being red on the outside. Humans juggle such inherited, emerging, and canceled traits without effort. Prior tests covered few properties and never watched models generate the combinations. The authors built a dataset of 12,300 annotated items linking noun phrases, properties, and property types.

Three task formats probe the models: judging, choosing, and generating. Their automatic grading metric tracks human judgment well. Results show current models stumble in systematic ways, especially on properties that combination cancels or creates. The benchmark gives the field a sharper tool for a basic thinking skill. Word blending sits underneath everyday reasoning about new things.

Catalog record

CREATORSeokwon Song, Taehyun Lee, Jaewoo Ahn, Jae Hyuk Sung, Gunhee Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-06086

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Is a Peeled Apple Still Red? Evaluating LLMs’ Ability for Conceptual Combination with Property Type Seokwon Song, Taehyun Lee, Jaewoo Ahn, Jae Hyuk Sung, Gunhee Kim Abstract † † footnotetext: ⋆ \star Authors equally contributed. † † footnotetext: ♯ \sharp Work done during internship at Seoul National University. Conceptual combination is a cognitive process that merges basic concepts, enabling the creation of complex expressions. During this process, the properties of combination (e.g., the whiteness of a peeled apple) can be inherited from basic concepts, newly emerge, or be canceled. However, previous studies have evaluated a limited set of properties and have not examined the generative process. To address this gap, we introduce the Conceptual Combination with Property Type dataset ( CCPT ), which consists of 12.3K annotated triplets of noun phrases, properties, and property types. Using CCPT , we establish three types of tasks to evaluate LLMs for conceptual combination thoroughly. Our key findings are threefold: (1) Our automatic metric grading property emergence and cancellation closely corresponds with human judgments. (2) LLMs, including OpenAI’s o1, struggle to generate noun phrases which possess given emergent properties. (3) Our proposed method, inspired by cognitive psychology model that explains how relationships between concepts are formed, improves performances in all generative tasks. The dataset and experimental code are available at https://github.com/seokwon99/CCPT.git . \DeclareCaptionType equ[][] 1 Introduction The conceptual combination is a fundamental cognitive process that synthesizes multiple basic concepts into a novel concept Wisniewski 1997 ; Thagard 1984 ; Coutanche et al. 2019 ; Ward 2001 . Combining concepts is potentially limitless and plays an important role in various fields such as engineering, science Hampton 1997 , figurative language and literature Ward 2001 . For example, ‘a wilted flower’ evokes our sadness because we internally compare a flower’s former beauty and current feeble state. A skilled writer knows it and kindles diverse imagery through conceptual combinations. The potential for such conceptual combinations is vast, so large language models (LLMs) should be able to interpret or generate meaningful insights even for rare or novel combinations. Figure 1: Three types of properties derived from conceptual combination with an example of “apple”. Different concepts are formed by adding other concepts to “apple”. The green properties are component properties of the basic concept “apple”. The blue and orange are emergent and canceled properties , respectively. Several works have explored concept knowledge of language models, as outlined in Table 1 . Basic Concept Probing examines whether language models understand individual concepts at the entity (e.g., “horses are mammals”) and event levels (e.g., “drinking tea leads to feeling refreshed”). However, this approach mainly focuses on single concepts rather than combinations. Noun Compound Interpretation studies how models generate plausible meanings for combinations of nouns (e.g., “dog house” means “a house for a dog”), but does not explore deeper properties of combinations. In conceptual combination works, Big-Bench-CC investigates emergent properties. FakeReef delves into a special type of canceled property (membership inference). Still, none of them addresses all of emergent, component, or canceled properties nor generative tasks related to creating combinations or property. It is essential to understand not only properties that emerge from a combination, but also whether existing properties maintain or disappear Springer and Murphy 1992 ; Gagné et al. 2017 ; Estes and Glucksberg 2020 . Following Springer and Murphy 1992 , we identify three types of properties in conceptual combination, which are illustrated using the example of an “apple” in Figure 1 : (1) Component property refers to a property inherent to individual concepts (e.g., “red” for “apple on a toothpick”), (2) Emergent property is newly from the combination of concepts (e.g., “unstable” for “apple on a toothpick”), and (3) Canceled property is negated due to the combination (e.g., “crisp” for “rotten apple”). These three types form a complete set of properties related to conceptual combination since they indicate already present , newly created , and newly disappeared . dataset # Q Property MC com eme can Basic Concept Probing BERT-IS-A ( Lin and Ng 2022 ) 11,868 ✓ ✗ ✗ ✗ COPEN ( Peng et al. 2022 ) 10,624 ✓ ✗ ✗ ✗ PLM-Ontology ( Wu et al. 2023 ) 9,687 ✓ ✗ ✗ ✗ Noun Compound Interpretation SemEval-2013-Task4 ( Hendrickx et al. 2013 ) 355 ✗ ✗ ✗ ✓ ChocolateBunny ( Coil and Shwartz 2023 ) 105 ✗ ✗ ✗ ✓ COMPUN ( Rambelli et al. 2024 ) 918 ✗ ✗ ✗ ✓ Conceptual Combination BIG-bench-CC ( BIG-bench authors 2023 ) 103 \Delta ✓ ✗ ✓ FakeReef ( Ross et al. 2024 ) 854 ✓ ✗ ✓ ✓ CCPT (ours) 12,315 ✓ ✓ ✓ ✓ Table 1: Comparison of our dataset with previous concept property probing and noun compound interpretation datasets. Each column denotes as follows. # Q: the number of questions with gold answers. com , eme , and can denotes component, emergent and canceled property, respectively. MC : whether the combinations of multiple concepts are considered. In Big-bench-CC, \Delta indicates that component properties are present in a dataset as wrong options, but it does not specify what properties are component properties. In this study, we aim to fill this gap by proposing the C onceptual C ombination with P roperty T ype ( CCPT ) dataset. Our dataset contains 12.3K annotated triplets of noun phrase comprised of two basic concepts, property and property type. The basic concepts in noun phrases are comprised of 9K unique uni-gram concepts from ConceptNet Speer et al. 2017 . To evaluate LLMs’ ability on conceptual combination, we introduce two generative and one classification tasks, detailed in Table 2 and § 3.1 . Based on the prior works in cognitive psychology and linguistics, each task is designed to evaluate cognitive processes related to conceptual combination as follows. (1) Property induction : imagine the property of noun phrase Wilkenfeld and Ward 2001 ; Estes and Ward 2002 , (2) Noun phrase completion : create a noun phrase to represent certain property, highlighted as a critical point of creativity Kohn et al. 2011 , and (3) Property type prediction : predict a type of property to identify the origin of property Gagné et al. 2017 . Extensive experiments show that: (1) our automatic evaluation metric, based on the LLM-as-a-judge Zheng et al. 2023 method to detect property emergence and cancellation scores, closely matches human evaluations; (2) current LLMs, including o1 OpenAI 2024c , struggle to create noun phrases with truly emergent properties, often defaulting to results where one component already has the target property; and (3) our proposed method, motivated by the spread activation model Anderson 1983 ; Collins and Loftus 1975 in cognitive psychology, improves performance across all generative tasks, indicating that considering diverse relationships between concepts enhances conceptual combination. 2 Related Work 2.1 Conceptual Combination Conceptual combination is a mental process of combining concepts. Unique features often arise from the combination, especially when the combinations of elements are unusual Estes and Ward 2002 ; Ward 2007 . People come up with novel ideas to merge conflicts; thus, it is a prominent part of creativity Hampton 1997 . In scientific discovery, conceptual combination creates new scientific concepts Thagard 1984 , such as “light wave.” Even if nobody could observe light waves, the

Read the complete text