CresOWLve : Benchmarking Creative Problem-Solving Over Real-World Knowledge
Mete Ismayilzada Renqing Cuomao Daniil Yurshevich Anna Sotnikova, Lonneke van der Plas Antoine Bosselut
Abstract
Creative problem-solving requires combining multiple cognitive abilities, including logical reasoning, lateral thinking, analogy-making, and commonsense knowledge, to discover insights that connect seemingly unrelated pieces of information. However, most existing benchmarks for large language models (LLMs) evaluate only specific components of this process. Moreover, many creativity-oriented benchmarks rely on artificially constructed brainteasers or contrived scenarios that do not reflect how creative problem-solving occurs in real-world settings. To address this gap, we introduce CresOWLve , a benchmark for evaluating creative problem-solving using puzzles grounded in real-world knowledge. Problems in CresOWLve require employing multiple creative thinking strategies, retrieving facts from diverse domains, and creatively combining them to arrive at a solution. Evaluating several frontier non-thinking and thinking LLMs, we show that CresOWLve remains highly challenging. Our analysis reveals a consistent performance gap: models perform substantially better on factual questions than on creative ones (up to − 17 % -17\% drop). While models can often retrieve the relevant knowledge, they struggle to form the non-obvious creative connections required to integrate the knowledge and arrive at the correct answer.
\textsuperscript{*}
\textsuperscript{*}
footnotetext:
Equal contribution
1
Introduction
Creative problem-solving is a central component of human intelligence and plays a crucial role in scientific discovery, innovation, and everyday reasoning
(
13
)
. Solving problems creatively often requires a combination of several cognitive processes
(
33
)
.
Vertical thinking
refers to systematic, step-by-step reasoning that derives conclusions through logical progression, while
lateral thinking
involves restructuring a problem and identifying non-obvious connections between seemingly unrelated ideas
(
12
;
64
)
. Similarly, creativity research distinguishes between
divergent thinking
, which emphasizes generating multiple possible ideas or solutions, and
convergent thinking
, which focuses on identifying a single insightful solution by integrating different pieces of information
(
23
)
. Creative problem solving also relies on additional abilities such as abstraction and analogy making
(
28
)
, which allow individuals to transfer knowledge across domains, as well as commonsense reasoning
(
11
)
, which provides background knowledge about how the world works.
Figure 1:
An example from
CresOWLve
annotated with the real-world knowledge and creative thinking strategy.
Despite rapid advances in large language models (LLMs)
(
71
)
, most existing benchmarks evaluate only specific aspects of creative problem solving. Some benchmarks emphasize vertical reasoning through tasks such as factual reasoning
(
26
;
54
)
, mathematical/logical inference
(
27
;
56
)
, and commonsense reasoning
(
55
;
59
;
42
)
, while others focus on lateral thinking using brainteasers and situational puzzles
(
24
;
8
;
30
;
36
)
.
Other works have proposed benchmarks to measure convergent/divergent thinking using psychometric tests
(
57
;
17
;
5
)
or puzzles and games
(
61
;
3
)
.
Although these benchmarks capture important individual capabilities, real-world creative problem-solving typically requires a combination of these skills. Furthermore, many existing evaluations rely on artificially constructed brainteasers (e.g.
“What type of cheese is made backwards?”
) or contrived situational scenarios (e.g.
“Two men are beating each other up and both of them suddenly fall to the floor. How?”
). In practice, however, creative insights often emerge from retrieving relevant knowledge across diverse domains and reasoning about it in novel ways. As a result, these benchmarks provide only a partial picture of the real-world creative problem-solving abilities of models.
To address this gap, we introduce
CresOWLve
, a benchmark designed to evaluate creative problem-solving in LLMs through single-answer puzzles that require the combination of multiple cognitive abilities, such as vertical/lateral, convergent/divergent thinking, analogy-making, and are grounded in real-world knowledge (Appendix Table
3
provides a full comparison to past relevant benchmarks and Figure
1
illustrates an example puzzle from our benchmark).
CresOWLve
is constructed from questions drawn from the renowned Russian intellectual game ‘‘What? Where? When?’’
1
1
1
https://en.wikipedia.org/wiki/What%3F_Where%3F_When%3F
, in which expert human participants solve carefully crafted problems that require not only broad world knowledge but also creative insight to combine disparate facts in non-obvious ways.
To ensure accessibility and relevance, we design a multi-stage benchmark construction pipeline that filters unsuitable and non-creative questions and translates the remaining puzzles into English with manual validation. The resulting dataset provides a diverse and high-quality benchmark for evaluating creative problem-solving grounded in real-world knowledge
2
2
2
We release the benchmark at https://huggingface.co/datasets/mismayil/cresowlve
. We summarize our main contributions as follows:
•
We present
CresOWLve
, a bilingual benchmark for creative problem-solving grounded in real-world knowledge and solvable by human experts.
CresOWLve
spans a diverse range of knowledge and creative domains, varies in difficulty, requires multiple creative thinking strategies and is manually validated to ensure quality.
•
We evaluate several frontier open-weight and proprietary LLMs varying in their size and reasoning mode, on
CresOWLve
and demonstrate that, despite recent advances, the benchmark remains highly challenging, revealing substantial gaps in model performance on creative reasoning over real-world knowledge. Notably, thinking models substantially outperform their non-thinking counterparts, demonstrating that extended reasoning is a key enabler for creative problem-solving.
•
We conduct a comprehensive analysis of model performance across multiple dimensions, including question difficulty, domains of knowledge and creativity, and show that models consistently underperform on creative questions relative to factual ones. Our error analysis also shows that while models can often retrieve the relevant knowledge, they frequently fail to make the creative connections among facts necessary to arrive at the correct answer.
2
Related Work
Creativity Evaluation
Evaluating creativity is inherently challenging due to its subjective nature, and traditional assessments often rely on human judgment using techniques such as the Consensual Assessment Technique
(
33
;
4
)
or psychometric tests
(
57
;
17
;
23
;
46
)
. To complement these approaches, several computational metrics and corresponding benchmarks have been proposed that capture different dimensions of creativity, including novelty
(
70
;
44
;
51
;
37
)
, diversity
(
52
)
, surprise
(
7
;
38
;
35
)
, and quality
(
15
;
14
)
. Other works have introduced multi-task creativity evaluation datasets
(
32
;
29
;
66
)
where problems don’t have a single correct answer, but rather many subjective answers, and the evaluation focuses on measuring different dimensions of output creativity. In creative problem-solving, however, there are often single (or a few) correct answers, and the emphasis is on the type of creative thinking process involved to get to the final answer. Our work similarly targets creative problem solving with a single correct answer, hence our analysis focuses more on the evaluation of different creative thinking abilities employed by the model to solve our benchmark puzzles.
Creative Problem-Solving Benchmarks
While most works have focused on measuring vertical thinking capabilities of LLMs through tasks such as factual reasoning
(
26
;
54
)
, mathematical/logical inference
(
27
;
56
)
, and commonsense reasoning
(
55
;
59
;
6
;
34
;
42
)
, several benchmarks have also been proposed to evaluate other aspects of problem-solving that requires more creativity
(
33
)
. More specifically, past work has evaluated lateral thinking using mostly brainteasers and situational puzzles
(
24
;
8
;
30
;
36
;
39
;
62
)
, convergent/divergent thinking using psychometric tests
(
57
;
17
;
5
)
or puzzles and games
(
61
;
3
;
63
)
and abstraction/analogy-making through synthetic puzzles
(
2
;
9
;
48
;
40
)
. Other works have focused on specific domains such as coding
(
45
)
, and mathematics
(
68
;
58
)
. Our benchmark on the other hand requires models to employ multiple creative thinking abilities and reason creatively over real-world knowledge drawn from diverse domains. We also note that while the questions from the “What?Where?When” game has been used for LLM evaluation in the past by
41
, our work significantly differs in several aspects: 1)
41
only considers questions with more factuality and shorter reasoning chains while we focus on more creative questions 2)
41
does not perform any data filtering to remove unanswerable or Russian-culture specific questions 3)
41
evaluates only one LLM (namely,
LlaMa3-405B
) and provides limited analysis while we benchmark several frontier LLMs and provide extensive analysis on question difficulty, reasoning types and error categories. 4)
41
considers Russian-only evaluation, while we prepare and evaluate models on a human-validated English version of the dataset as well.
3
CresOWLve
3.1
Benchmark Construction
Data Collection
We collect
3,789
questions from the public database at
db.chgk.info
, which contains questions from the popular Russian intellectual game “What?Where?When?” spanning over 50 years. Questions in this game have been manually crafted by humans and often require a combination of skills such as logical thinking, intuition, and creative insight. Each question is annotated with a short answer, an answer explanation, and a difficulty rating (1 to 5) manually assigned by the game organizers. To ensure diversity in benchmark difficulty, we collect
\approx 700
questions per difficulty rating. We perform several filtering and annotation stages using
GPT-4o
(
31
)
due to its strong performance
(
25
;
16
)
. Prompts can be found in Appendix Tables
5
,
7
,
6
,
8
,
9
and
11
. Example questions for each stage can be found in Appendix Table
4
.
Data Filtering
Since the original game is in Russian and also in-person interactive, some questions rely heavily on linguistic and cultural knowledge specific to Russia, and some require physical inspection of external material, such as images or handouts. Hence, these questions are either extremely hard to answer or unanswerable completely. Therefore, we perform several filtering steps to ensure the validity and wide accessibility of the questions.
We first filter out unanswerable questions by annotating each question on whether they require physical external materials to be solved. This step removes
295
samples.
Next, we filter out questions that depend heavily on knowing specific facts rooted deeply in the Russian language and culture, rendering them extremely hard to answer or untranslatable into English. This step removes
799
samples.
Translation
Our goal in this work is to design a benchmark that is relevant and accessible for measuring creative problem-solving in frontier LLMs. However, since most LLMs are more proficient reasoners in English than in other languages, we also prepare an English version of our benchmark by translating each question-answer pair with all its metadata.
Human Validation
While LLMs have recently become remarkably effective in data annotation tasks, they can still exhibit hallucination and reasoning failures
(
60
)
. Therefore, we conduct a final human validation on the entire benchmark (including the factual reasoning questions) to ensure high quality. More specifically, three authors of this paper reviewed all remaining questions from the last filtering step to validate that they are answerable, not Russian-specific, and that the translations are correct. This step further removed
282
samples from the benchmark, leaving
2,413
samples for final evaluation.
Creative vs. Factual Reasoning
As noted by
41
, some questions can be answered with purely factual retrieval and reasoning involving no to minimal creative leap-of-thought. To further distinguish this dimension, we automatically categorize questions as either
factual
or
creative
, where creative questions require the model to combine and apply knowledge in ways that go beyond direct retrieval. This annotation yields
352
factual and
2,061
creative samples. We later use this factual subset to analyze the performance on creative vs. factual reasoning questions.
3.2
Final Benchmark
Our final benchmark contains
2,061
samples (creative subset) both in Russian and English (referred to as
CresOWLve
-Ru and
CresOWLve
-En respectively). Data statistics about the benchmark can be found in Appendix Table
2
.
Knowledge Domains
As noted earlier, one of the challenges of this benchmark is to creatively reason over real-world knowledge across domains. To visualize the domain diversity, we automatically annotate each benchmark sample with the knowledge domains required to solve it. The initial annotation yields
541
domains, which are too fine-grained and contain substantial overlap. We then consolidate them into broader subject domains, resulting in
34
coarse-grained, largely non-overlapping categories. Figure
2
illustrates the coarse domain breakdown. We can see that while the benchmark covers a wide range of topics, notably, Literature, History, Film & Media Studies, and Languages & Linguistics are dominating subjects. We also note that each question is annotated with multiple topics, and most questions involve at least two to four topics (See Appendix Figure
6
).
(a)
Distribution of knowledge domains.
(b)
Distribution of creative domains.
(c)
Distribution of cultures/demographics.
Figure 2:
Diversity of real-world knowledge, creative language, and cultures.
Creative Language & Thinking
In addition to the knowledge domains, our benchmark questions also require reasoning about creative language and employing multiple creative thinking strategies such as lateral thinking, abstraction, and analogy-making. To quantify them, we perform an automatic annotation and report the breakdown in Figure
2
. We see that most of the questions require lateral thinking, an ability to identify non-obvious associations, and a substantial number of questions involve abstraction and analogy-making. Many questions also require reasoning about jokes, puns, or metaphors, as well as commonsense knowledge, and most questions involve two creative domains (Appendix Figure
6
).
Cultures & Demographics
While the original game is in the Russian language, its questions often involve knowledge about entities and people from other cultures, too. To quantify the diversity of cultures and demographics in our benchmark, we automatically annotate each sample in the benchmark with the cultures involved in solving the question. We report the breakdown in Figure
2
. We note that while English and Russian cultures dominate, our benchmark also contains a substantial number of questions on other cultures, such as French, German, Italian, and Greek. Similar to knowledge and creative domains, most questions often involve knowledge about more than one culture (Appendix Figure
6
).
4
Experimental Setup
†
†
nicematrix-placeholder:
NiceTabular (nicematrix)
Table 1:
Overall performance results.
Models
We assess a broad set of LLMs on our benchmark, varying in reasoning mode, architecture, size, and training paradigm. Since our benchmark requires complex reasoning, we particularly distinguish between
non-thinking
models, which generate responses directly without explicit intermediate reasoning (unless instructed to via Chain-of-Thought prompting
(
65
)
) and
thinking
models, which are trained to explicitly think with a certain amount of effort before producing a final answer. We consider the following non-thinking models:
GPT-4.1-mini
,
GPT-4.1
(
1
)
,
OLMo-2-32B-Instruct
(
49
)
,
Qwen3-235B-A22B-Instruct
(
67
)
,
C4AI-Command-A
(
10
)
,
Mistral-Large-3-675B-Instruct
(
47
)
and
Llama-3.3-70B-Instruct
(
21
)
and following thinking models:
Qwen3.5-397B-A17B
(
53
)
,
Qwen3-235B-A22B-Thinking
(
67
)
,
DeepSeek-V3.2
(
43
)
,
GLM-5
(
69
)
,
GPT-5.4
(
50
)
,
Gemini-3-Flash
(
19
)
, and
Gemini-3.1-Pro
(
20
)
.
Task & Evaluation Methods
We formulate our problem as an open-ended question answering task. Since our benchmark requires substantial reasoning over real-world knowledge, we evaluate non-thinking models using Chain-of-thought prompting and thinking models using a standard prompt with varying levels of thinking effort.
We evaluate all models using their default recommended decoding setup. Evaluation prompts can be found in Appendix Table
12
.
Evaluation Metrics
We measure the model performance using two metrics:
Exact Match Accuracy
, where we match the reference answer with the model response after light normalization (i.e., lowercase, remove punctuation, and unicode normalization for Russian), and
LLM-as-a-judge Accuracy
, where we employ
GPT-4o
to judge the correctness of the model response given the reference answer
(
22
)
. To ensure high-quality LLM judgment, we instruct it to ignore typos, articles, or formatting differences and provide it with additional context, including the human answer explanations and other acceptable answers, if any. LLM-as-a-judge prompt can be found in Appendix Table
13
.
5
Results
Figure 3:
LLM Judge results by difficulty level (Exact Match, Appendix Figure
13
).
Overall Performance
Table
1
reports overall performance on
CresOWLve
-En and
CresOWLve
-Ru. Results vary widely across models and metrics, ranging from below
10\%
to above
80\%
. Non-thinking models all remain under
30\%
accuracy, with
GPT-4.1
being the strongest in this group. Thinking models generally outperform their non-thinking counterparts, and greater thinking effort consistently yields higher accuracy. This effect is particularly pronounced for
GPT-5.4
, whose performance nearly doubles when moving from no thinking to medium effort. Among thinking models, closed-source models lead overall; the
Gemini
series in particular achieves top performance even at minimal thinking effort, surpassing all open-weight models.
Regarding cross-lingual transfer, we observe mixed trends. Most non-thinking models suffer a notable performance drop on the Russian benchmark (up to
-5\%
LLM Judge and
-3\%
Exact Match), with
Qwen3-235B-A22B-Instruct
and
GPT-4.1
being the only exceptions. Thinking models, however, show the opposite trend: closed-source thinking models in particular achieve substantially higher scores on the Russian benchmark (up to
+15\%
LLM Judge and
+18\%
Exact Match).
This surprising advantage on
CresOWLve
-Ru may reflect stronger multilingual reasoning capabilities at higher levels of thinking, or potential data contamination (for which we find some evidence in Appendix Section
A
). Regardless, the consistently low performance of open-weight (and some proprietary) models across both languages confirms that
CresOWLve
poses a genuine challenge and serves as a robust testbed for benchmarking creative problem-solving in LLMs.
Performance by difficulty level
Figure
3
illustrates model performance stratified by difficulty level for both
CresOWLve
-En and
CresOWLve
-Ru under LLM Judge evaluation. As expected, accuracy decreases monotonically with difficulty for all models, confirming that the benchmark difficulty levels capture the underlying complexity of the questions. Across both benchmarks, the performance degradation from Very Simple to Very Hard is substantial — even the strongest thinking models lose over
20\%
accuracy. This highlights a key challenge posed by
CresOWLve
: while average accuracy figures may appear moderate, the benchmark contains a non-trivial proportion of questions that remain genuinely difficult even for state-of-the-art models, underscoring the headroom for future progress.
Performance by domain, creativity, and cultures
Detailed performance breakdowns by knowledge and creative domains, and culture groups are provided in Appendix Figures
7
,
8
,
9
and
10
,
11
,
12
for the
CresOWLve
-En and
CresOWLve
-Ru benchmarks respectively. Across benchmarks, performance is largely uniform across all three stratifications, though several notable trends emerge. Across domains, most models tend to perform better on Earth & Environmental Sciences questions, with
Qwen3-235B-A22B-Thinking
showing a particular advantage on Astronomy and
DeepSeek-V3.2
on Mathematics. Regarding creative language concepts, models generally perform worse on questions involving poems and metaphors, suggesting that figurative and poetic language remains a particular challenge. On the culture dimension,
Gemini-3.1-Pro
performs notably well on questions involving American cultural references,
GPT-4.1
on Japanese, and
DeepSeek-V3.2
on Latin.
Human Performance
We do not conduct a formal human evaluation, as the puzzles in
CresOWLve
are specifically designed to be solved by expert players of the original game and indeed, all questions in our benchmark have been correctly answered in past game episodes. This provides an implicit upper bound: the benchmark is solvable, but requires a combination of broad knowledge and creative reasoning. Interestingly, human experts in this setting operate under conditions that are in some respects more constrained than LLMs: they have been exposed to far less data and cannot rely on exhaustive factual recall. Instead, they compensate through intuition and creativity, using partial knowledge to make imaginative leaps toward the correct answer. LLMs, by contrast, have been trained on virtually all publicly available text and should in principle, have access to all the factual knowledge required to solve every question. The fact that they nonetheless fall short seems to highlight that the bottleneck is not knowledge retrieval, but the creative reasoning needed to connect that knowledge, precisely the capability that
CresOWLve
is designed to probe.
6
Analysis
Creative vs. Factual Reasoning
To investigate whether models struggle more with creative reasoning than factual retrieval, we evaluate on the factual subset of
CresOWLve
as discussed in Section
3.1
. Figure
4
reveals a consistent and substantial performance drop from factual to creative questions across all models and languages.
On
CresOWLve
-En, performance drop ranges from
-6.08\%
for
Gemini-3.1-Pro (high)
to
-17.21\%
for
Mistral-Large-3-675B-Instruct
. On
CresOWLve
-Ru, the same trend holds, though the absolute drop is smaller for
Gemini-3.1-Pro
models (
-2.14\%
and
-2.73\%
). For a fair comparison, we match the sample sizes across both categories and perform bootstrap sampling with 1000 iterations. In addition, we balance the number of samples across difficulty levels for both creative and factual questions (Appendix Figure
15
). These results confirm that creative questions pose a fundamentally harder challenge than factual ones.
Figure 4:
LLM Judge results by reasoning category (Exact Match, Appendix Figure
14
).
Difficulty vs. Complexity
To better understand the sources of question difficulty in
CresOWLve
, we investigate whether difficulty correlates with measurable complexity features of the questions. Specifically, we consider three proxy measures of question complexity: the number of domains involved, the maximum semantic distance between domains for a given question, and the number of atomic facts required to resolve it. The latter is estimated by prompting
GPT-4o
to decompose each question into a list of constituent factual sub-questions (prompt provided in Appendix Table
10
). Despite the intuitive appeal of these features as predictors of difficulty, we find no significant correlation between any of them and either the assigned difficulty level or model performance (Appendix Figures
16
,
17
). These findings strongly suggest that the difficulty of questions in
CresOWLve
doesn’t stem from the number of involved domains, the challenge of bridging semantically distant domains, or the volume of facts one needs to recall, but rather from the need to connect knowledge in creative ways.
Figure 5:
Error category distribution for best performing models.
Error Analysis
We conduct a manual analysis of errors produced by Gemini-3-Flash (medium). From a total of 1,087 error examples, we randomly sample 150 instances with the model’s reasoning traces and obtain annotations from three authors of the paper, who are also domain experts. Following an iterative error analysis process, we identify seven error categories that are most prominent in our data:
•
Missing creative connection:
The model retrieves relevant knowledge, but fails to recognize the intended associative or metaphorical link between clues, preventing it from making the key conceptual leap needed to reach the correct answer.
•
Overthinking:
The model identifies the correct concept during reasoning, but later replaces it with another answer due to reinterpretation or over-generalization.
•
Hallucination or unsupported fabrication:
The model invents facts, explanations, or source details that are not supported by the question or reliable knowledge.
•
Incorrect concept anchoring:
The model locks onto an incorrect concept early and builds reasoning around it, either through associative drift or by reinterpreting the clues to fit that concept.
•
Wrong hypothesis:
The model infers an incorrect rule or shared property from the clues and applies it consistently to produce an answer.
•
Wrong reference:
The model retrieves and reasons about an incorrect work, person, event, or source that superficially matches the clues but is not the intended reference.
•
Clue misinterpretation:
The model identifies the general source, context, or line of reasoning, but selects the wrong specific element required by the clue.
For each category, we provide representative examples in Appendix
B
. We then use these definitions to automatically label all wrong examples at scale.
The resulting error distribution for the best performing models is shown in Figure
5
. The most frequent error categories across models are
missing creative connection
and
clue misinterpretation
, further confirming our hypothesis about the difficulty of questions. A notable exception is
Gemini-3.1-Flash
, which shows significantly fewer missing creative connection errors but a substantially higher rate of
wrong reference
errors.
GPT-5.4
stands out for its disproportionately high rate of
incorrect concept anchoring
. Finally, low hallucination and wrong reference rates across most models confirm that failures stem not from missing knowledge but from an inability to make the creative connections between facts necessary to reach the correct answer.
7
Conclusion
We introduced
CresOWLve
, a bilingual benchmark for evaluating creative problem-solving in LLMs, constructed from real-world puzzles sourced from the What? Where? When? intellectual game. Our evaluation of several frontier LLMs reveals that the benchmark remains highly challenging, with even the strong thinking models falling considerably short on creative questions. Analysis shows that this difficulty does not stem from surface-level complexity features, but from the need to forge creative connections between knowledge from different domains. We hope
CresOWLve
serves as a challenging testbed to drive progress in creative reasoning.
Ethics Statement
This work introduces
CresOWLve
, a benchmark for evaluating creative problem-solving in large language models using puzzles derived from the intellectual game “What? Where? When?”. While the goal is to advance the evaluation of creative reasoning, several ethical considerations arise.
The benchmark is constructed from publicly available questions created by human authors over several decades. These questions reflect the intellectual contributions of their original creators, and we do not claim authorship of the underlying content. The dataset is used solely for research purposes, and proper attribution should be maintained where applicable.
Because the questions are designed to test non-obvious associative and creative connections, they may reflect the cultural context, assumptions, and potential biases of their authors. In particular, as the source material originates in Russian, some questions may encode culturally specific knowledge or perspectives. To mitigate this, we annotate questions based on their regional content; however, one should account for potential biases.
Given that the source material is publicly available and widely distributed, there is a risk that some models may have been exposed to similar questions during training. As discussed in our analysis, this potential data contamination may affect performance and should be considered when interpreting results.
Finally, although the benchmark is designed to measure creative problem-solving, creativity is inherently difficult to define and evaluate. Our categorization into factual and creative questions, as well as the proposed error taxonomy, involves subjective judgments and may not capture all aspects of creative reasoning.
Appendix A
Data Contamination Analysis
To investigate whether the performance advantage of thinking models on
CresOWLve
-Ru over
CresOWLve
-En could be attributed to memorization and potential data contamination, we conduct a targeted contamination analysis. To isolate other confounding factors, we restrict our analysis to the 580 questions that
Gemini-3.1-Pro
answered correctly in Russian but not in English. We focus on this model as the best-performing one in our evaluation, and deliberately use low thinking effort to ensure the model relies more on memorization than active reasoning. Following
18
, we first generate three unique word-level perturbations for each question using
GPT-4o
(prompt in Table X). We then run two contamination quizzes sharing the same prompt structure (prompt Table X) but with different answer choices. In the first quiz, we present the model with a multiple-choice question where all options are perturbed versions of the original question and an additional “none of the above” option is included; crucially, no correct answer is present. This allows us to identify positional biases by analyzing how frequently the model selects each option. We identify non-preferred positions as those chosen less frequently than random chance (less than
k/4=145
with
k=580
and 4 options), which yields options B and C as non-preferred(full distribution is ’A’: 187, ’B’: 60, ’C’: 47, ’D’: 273 with an additional 13 invalid answers). Notably, the model correctly prefers option D (“none of the above”) most of the time. In the second quiz, we insert the correct answer into one of these non-preferred positions (randomly chosen between B and C) for each question and repeat this quiz three times. The model achieves
72\%
,
73\%
, and
74\%
accuracy across the three runs (mean
73\%
), all substantially above the random baseline of
25\%
. This result strongly suggests that
Gemini-3.1-Pro
has likely been exposed to these questions during training, pointing to high-level data contamination as a plausible explanation for its cross-lingual performance advantage.
Difficulty
#questions
#explanations
Avg. #question tokens
(En / Ru)
Avg. # answer tokens
(En / Ru)
Very Simple (1)
417
375
37.79\pm 12.84
/
29.65\pm 9.90
2.12\pm 1.51
/
1.82\pm 1.09
Simple (2)
453
426
38.64\pm 12.84
/
30.79\pm 10.13
2.20\pm 1.73
/
1.87\pm 1.30
Medium (3)
436
413
38.11\pm 12.13
/
30.34\pm 10.08
2.06\pm 1.38
/
1.87\pm 1.10
Hard (4)
371
355
38.61\pm 13.43
/
30.60\pm 10.35
2.13\pm 1.88
/
1.85\pm 1.39
Very Hard (5)
384
359
37.92\pm 13.05
/
30.27\pm 10.45
2.12\pm 1.46
/
1.89\pm 1.22
Total
\mathbf{2061}
\mathbf{1928}
\mathbf{38.22\pm 12.84}
/
\mathbf{30.33\pm 10.18}
\mathbf{2.13\pm 1.60}
/
\mathbf{1.86\pm 1.22}
Table 2:
Final benchmark statistics by difficulty level.
(a)
Knowledge domains.
(b)
Creativity concepts.
(c)
Cultures/demographics.
Figure 6:
Distribution of number of knowledge domains, creative language constructs and cultures/languages per question.
Features
Benchmarks
Conv.
Div.
Lat.
Analogy
Com.Sense
Real-World KG
Domains
BRAINTEASER
(
36
)
✔
✔
✔
✗
✔
✗
Various
RiddleSense
(
42
)
✔
✗
✗
✗
✔
✔
Various
Briangle
24
✗
✔
✔
✗
✗
✗
Math/Logic
SPLAT
(
8
)
✗
✔
✔
✗
✔
✗
Daily-Life
LatEval
(
30
)
✗
✔
✔
✗
✔
✗
Daily-Life
Columbus
(
39
)
✔
✔
✔
✗
✗
✗
Logic
Connections
(
62
)
✗
✔
✔
✗
✔
✗
Various
DAT
(
5
)
✗
✔
✗
✗
✗
✗
Various
McGyver
(
61
)
✔
✔
✗
✗
✔
✔
Daily-Life
OnlyConnect
(
3
)
✔
✗
✗
✗
✔
✔
Various
CREATE
(
63
)
✔
✔
✗
✗
✗
✔
Various
(
40
)
✔
✗
✗
✔
✗
✗
Logic
NEOCODER
(
45
)
✔
✔
✗
✗
✗
✗
Coding
CreativeMath
(
68
)
✔
✔
✗
✗
✗
✗
Math
OMEGA
(
58
)
✔
✗
✗
✗
✗
✗
Math
CresOWLve
(Ours)
✔
✔
✔
✔
✔
✔
Various
Table 3:
Comparison of different creative problem-solving benchmarks. Our benchmark notably requires employing several creative thinking strategies and involves various domains. (Conv.=Convergent, Div.=Divergent, Lat.=Lateral, Com.Sense=Commonsense reasoning)
Appendix B
Error Examples
Missing creative connection
Question
In the first half of the 20th century, many American farmers had to move—some due to economic difficulties, others due to dust storms. For example,
SHE
moved to another country in 1911. Name
HER
name.
Gold answer
Dorothy.
Model output
In the first half of the 20th century, a significant migration of American farmers occurred, often driven by economic opportunities and agricultural developments. One of the most notable periods of this migration was between 1896 and 1914, when hundreds of thousands of American farmers moved to the Canadian Prairies (Alberta, Saskatchewan, and Manitoba) to take advantage of cheap, fertile land. This period is historically recognized as the “Great Migration” of American farmers to Canada. In 1911 alone, over 100,000 Americans emigrated to Canada.
The placeholder
SHE
in the puzzle refers to a personification of the American farmer or a symbolic figure of the United States. Following the logic that SHE represents the collective identity of these migrants or a specific personification often used for ships or entities, the puzzle likely refers to the ship
“The American Farmer.”
However, a more direct interpretation of the riddle, considering the symbolic “SHE” often used for ships that transported these families, or the name of a famous individual, may point towards a specific ship or a personified noun. Yet, in the context of lateral thinking puzzles with these clues, the answer often points to a vessel or a specific entity named “The American Farmer” that “moved” (in maritime terms) to another country (destination) during that era. Looking at ship records, the SS
American Farmer
was a famous ship, though it was renamed later.
Alternatively, some lateral thinking puzzles use “SHE” to refer to the
“Mona Lisa”
, which was stolen in
1911
and “moved” to Italy (another country). However, the connection to American farmers in the first sentence is typically a contextual setup for migration patterns of that era. Re-evaluating the direct clue, “SHE” is an example of an American farmer. In 1911, the vast number of farmers moving to Canada included many individuals, but the “SHE” most famously associated with the concept of the American farmer (in a figurative sense through art or iconography) or simply the ship named
American Farmer
fits the phrasing. In maritime culture, a ship named
American Farmer
would be referred to as “SHE.”
The American Farmer
Predicted answer:
The American Farmer
Overthinking
Question
Pyotr Vail writes about one of the properties of cinema, that essentially D’Artagnan is no better than Rochefort, but
IT
convinces the viewer of D’Artagnan’s rightness. “Barbarossa” was also
IT
. Name
IT
with one word.
Gold answer
Close-up.
Model output
To solve this lateral thinking puzzle, we look to the writings of
Pyotr Vail
, a renowned Russian-American interviewer and essayist. In his cultural commentary, Vail noted that in cinema, the protagonist (like
D’Artagnan
in
The Three Musketeers
) is often no more morally upright than the antagonist (
Rochefort
); however, cinema uses specific techniques to make the viewer empathize with the hero.
The specific property Vail cites is the
“close-up”
(or in Russian,
“
крупный план
”
). By focusing the camera closely on the hero, cinema creates an intimate connection that “convinces the viewer of the hero’s rightness.”
The second clue refers to
“Barbarossa.”
In a historical and military context, “Barbarossa” was the name of a famous
plan
(the German invasion of the Soviet Union, known as
Operation Barbarossa
).
In Russian, the word
“
план
”
(plan) can refer both to a military plan and to a cinematic shot scale (as in
крупный план
). Based on this ambiguity, the model concludes that the correct answer is
plan
.
Plan
Predicted answer:
Plan
Hallucination or unsupported fabrication
Question
Upon learning about George Bush’s upcoming visit to Israel, a representative of “Al-Qaeda” called him one of
THEM
and urged to meet Bush with weapons in hand. Name the writer who made
THEM
his main characters.
Gold answer
Henryk Sienkiewicz.
Model output
In the context of the puzzle, “Al-Qaeda” is interpreted as a pun referring to a group of alcoholics (phonetically similar to “Al-Kashi” in Russian jokes). The model assumes that George Bush, as a recovering alcoholic, is being described as one of “THEM.” It further interprets “weapons” as bottles and concludes that the question refers to literary works centered on alcoholics.
Based on this reasoning, the model identifies
Venedikt Yerofeyev
, known for
Moscow-Petushki
, as the correct answer.
Venedikt Yerofeyev
Predicted answer:
Venedikt Yerofeyev
Hallucination or unsupported fabrication
Question
In Paris, Thomas Jefferson was presented with a gift made by French craftsmen from African materials. The gift has survived to this day, although not completely: two elements are missing. Perhaps they were not lost but deliberately removed to avoid putting the official representative of the young republican state in an awkward position. Name these two elements as precisely as possible.
Gold answer
The white and black kings.
Model output
The model interprets the gift as a decorative object made with African materials such as ebony and ivory. It hypothesizes that the missing elements were symbols of slavery (chains and shackles), which may have been removed to avoid political awkwardness given Jefferson’s role and the ideals of liberty.
chains and shackles
Predicted answer:
chains and shackles
Incorrect concept anchoring
Question
Michael Chabon compared a river glistening with metal, whose waves crash between the embankments, to an X. The internet project “X,” conceived by Boris Berezovsky, was intended to report on violations of civil liberties in Russia. Name X.
Gold answer
Bell.
Model output
The model interprets the metaphor as referring to a zipper (“molniya” in Russian), and connects this to a supposed internet project named “Molniya” associated with Boris Berezovsky.
Molniya
Predicted answer:
Molniya
Incorrect concept anchoring
Question
In his descriptions, the author of the novel mentions the radiance of a rainbow, often adorning the head, as well as the crown of a weeping willow. Name this author.
Gold answer
Herman Melville.
Model output
The model interprets the clues as referring to the butterfly
Apatura iris
and associates this imagery with Vladimir Nabokov, known for his interest in butterflies and descriptive prose.
Vladimir Nabokov
Predicted answer:
Vladimir Nabokov
Incorrect concept anchoring
Question
In a humorous translation of the first part of the film “The Lord of the Rings,” Gandalf says to Frodo: “I love your Hobbiton — the nature here is beautiful.” In Gandalf’s line, we replaced two words with one. Write the replaced words.
Gold answer
New Zealand.
Model output
The model refers to a humorous Russian translation and interprets “Hobbiton” as a placeholder for the phrase “Tikhiy Omut” (“still waters”), concluding that the replaced words are “Still waters.”
Still waters
Predicted answer:
Still waters
Wrong hypothesis
Question
There are currently four such countries on the world map, although there used to be many more. Among them are Egypt, France, and Spain. Name the fourth country.
Gold answer
Panama.
Model output
The model identifies a shared property among Egypt, France, and Spain as countries bordering the Mediterranean Sea and having an additional coastline. It constructs a geographical explanation involving countries with dual coastlines and concludes that Morocco fits this pattern.
Morocco
Predicted answer:
Morocco
Wrong reference
Question
An alien from Zelazny’s novel gnaws off pieces of a diamond brick due to the peculiarities of its physiology. What did a character from an 1892 story involuntarily use for similar purposes?
Gold answer
Blue carbuncle.
Model output
The model interprets the question through a biological analogy involving continuously growing teeth, drawing a parallel between the alien and animals such as hares. It references a 19th-century story featuring a frightened hare and concludes that the character used his own teeth involuntarily.
His own teeth
Predicted answer:
His own teeth
Clue misinterpretation
Question
Sculptor Henry Moore equated women with nature, so
DEFINITIONS
in his works could symbolize ravines or depressions eroded by the forces of wind and water. What two words did we replace with the word “DEFINITIONS”?
Gold answer
Curvy lines.
Model output
The model focuses on Henry Moore’s use of negative space and interprets the description as referring to hollow spaces or holes within sculptures, drawing an analogy to natural formations such as caves or valleys.
hollow spaces
Predicted answer:
hollow spaces
Pipeline Step
ID
Question (Russian)
Question (English)
Answer
Step 1: Filtering unanswerable questions
(removed 295 samples)
Requires physical materials
7e149044a7
Статья в ‘‘Нью-Йорк Таймс’’, посвященная идее возврата к золотому стандарту в целях укрепления доллара, сопровождалась изображением, фрагмент которого мы вам раздали. Воспроизведите то, что мы закрыли черным прямоугольником.
An article in “The New York Times” about the idea of returning to the gold standard to strengthen the dollar was accompanied by an image, a fragment of which we distributed to you. Reproduce what we covered with a black rectangle.
In God We Trust
Step 2: Filtering Russian-specific questions
(removed 799 samples)
Russian language/ culture specific
fbce8ea84b
В произведении Евгения Лукина ОН в ответ на вопрос героя призывает отрицать эпидемию. Не спрашиваем, какие четыре слова мы заменили словами ‘‘отрицать эпидемию’’. Назовите ЕГО.
In the work of Yevgeny Lukin, IT, in response to the hero’s question, urges to deny the epidemic. We do not ask which four words we replaced with “deny the epidemic.” Name IT.
Ворон
/ Raven
(Pun on Poe’s Raven: “
Не верь в мор!
” — “Don’t believe in plague!”)
Step 3: Translation
(all remaining 2,695 samples translated with
GPT-4o
)
Translated
c97e6fd16d
По предположению американских ученых, складки, которые образуются на мокрых пальцах, выполняют ту же функцию, что и ИКС. После смерти отца ИКСОМ был провозглашен Ричард. Назовите его фамилию.
According to American scientists, the wrinkles that form on wet fingers serve the same function as X. After the death of his father, Richard was proclaimed X. Name his surname.
Кромвель
/ Cromwell
(X = Lord Protector; wrinkles protect grip.)
Step 4: Human validation
(removed 282 samples)
Still Russian-specific
506e7ac07e
Старейший американский клуб, основанный в 1903 году, называется ‘‘Белые медведи Кони-Айленда’’. Сезон у них длится с ноября по апрель. Каким словом мы называем участников подобных клубов?
The oldest American club, founded in 1903, is called the “Coney Island Polar Bears.” Their season lasts from November to April. What word do we use to refer to members of such clubs?
Моржи
/ Walruses
(Russian-specific term for winter swimmers.)
Requires external materials
17cad876d2
Перед вами фрагмент альтернативного постера к известному фильму. Назовите этот фильм.
In front of you is a fragment of an alternative poster for a famous film. Name this film.
‘‘Кофе и сигареты’’
/ “Coffee and Cigarettes”
Multiple questions
e089848e4f
Блиц. 1. На рекламной этикетке художника Баранова есть надпись ‘‘С кружкой пивца и работается!’’ и изображение человека. Назовите этого человека. […]
Blitz. 1. On the advertising label by artist Baranov, there is an inscription “With a mug of beer, work is dear!” and an image of a person. Name this person. […]
1.
Ленин
/ Lenin
2.
Ленин
/ Lenin
3.
Ленин
/ Lenin
Outdated info
c95beab3bb
Михаил Прохоров, отвечая на неудобный вопрос, сказал, что ЕВА у него появилась в 17 лет. Назовите имя и фамилию нынешней американской ЕВЫ.
Mikhail Prokhorov, answering an awkward question, said that EVA appeared in his life at the age of 17. Name the first and last name of the current American EVA.
Мишель Обама
/ Michelle Obama
(Answer is outdated.)
Step 5: Creative vs. factual annotation
(352 factual, 2,061 creative)
Factual
8947f691ac
Вольф Шнайдер пишет, что один кристалл ЕГО в 1970-х годах лишил работы 45 тысяч швейцарцев. Назовите ЕГО.
Wolf Schneider writes that one crystal of IT in the 1970s deprived 45,000 Swiss people of their jobs. Name IT.
Кварц
/ Quartz
(Quartz watches replaced Swiss mechanical watches.)
Creative
83a712ed75
В романе о военном времени герой смотрит на небо, где инверсионные следы самолетов и разрывы зенитных снарядов образуют неприятную, по его мнению, картину. Какое слово мы заменили на ‘‘картину’’?
In a wartime novel, the hero looks at the sky, where contrails of airplanes and bursts of anti-aircraft shells form an unpleasant, in his opinion, picture. What word did we replace with “picture”?
Мелодию
/ Melody
(Contrails are staff lines; bursts are notes.)
Table 4:
Examples of questions affected at each stage of the data processing pipeline. For each step, we show a representative question in Russian (original) and English (translated), along with its answer and the reason for filtering or annotation.
External Material Filtering Prompt
You are a strict annotator. Your task is to decide whether the given puzzle EXPLICITLY instructs the participant to check external materials to answer the question.
Answer "yes" ONLY if the puzzle requires explicit external resource lookup, such as:
-- A handout
(‘‘раздаточный материал’’)
or statements like
‘‘на розданной вам фотографии’’, ‘‘смотрите раздатку’’, ‘‘не озвучивать текст раздатки’’, ‘‘в раздатке’’,
or similar
-- References to a hidden or closed element such as:
‘‘мы закрыли символ’’, ‘‘мы закрыли букву’’, ‘‘мы закрыли часть текста’’, ‘‘прочерк’’,
, when the missing content is NOT provided within the puzzle text.
-- Instructions requiring external lookup such as
‘‘зайдите по ссылке и посмотрите курс’’, ‘‘вставьте число из источника’’
etc.
Answer "no" if:
-- The puzzle does not EXPLICITLY instruct the participant to check external materials.
Output ONLY ’yes’ or ’no’ in lowercase, with no additional text.
Puzzle:
{question}
Additional notes:
{comment}
Answer:
Table 5:
Zero-shot prompt used to filter questions requiring physical external materials (See §
3.1
) for details.
Translation Prompt
You are a professional Russian to English translator. Your task is to translate Russian puzzles into English with absolute fidelity.
Translate EXACTLY, preserving:
-- all logical clues
-- all named entities
-- sentence order and structure
-- rhetorical devices
-- ambiguity
-- references and style
-- the puzzle’s original difficulty
Do NOT:
-- paraphrase
-- simplify or summarize
-- interpret hidden meanings
-- add explanations
-- rephrase stylistically
If a segment is unclear, translate it literally.
You are given the puzzle question, answer, comment and notes in Russian enclosed in , , , and tags.
Translate each section into English, preserving the structure and meaning, and output the translated puzzle in the same format with , , , and tags.
Output ONLY the English translation, nothing else.
Russian Puzzle:
{question}
{answer}
{comment}
{notes}
English Translation:
Table 6:
Zero-shot prompt used for translating puzzles from Russian to English (§
3.1
).
Russian-Specific Filtering Prompt (Instruction)
You are a strict annotator. You are given a puzzle in Russian, along with its answer, comment, and notes and translation of the puzzle into English. Your task is to decide whether the puzzle can not be solved in English because solving the puzzle REQUIRES knowledge specific to the Russian LANGUAGE.
Answer ’yes’ ONLY if the solution depends on Russian-specific linguistic features such as:
-- idioms, sayings, or set expressions that break in translation
-- wordplay, puns, or jokes that work ONLY in Russian
-- Russian-specific homonyms or near-homonyms
-- phonetic or rhyming clues that exist ONLY in Russian
-- letter-based tricks involving the Russian alphabet or spelling
-- translating the puzzle into another language would change or destroy the meaning, making the puzzle unsolvable or fundamentally different.
Answer ’no’ if:
-- the puzzle depends on general knowledge, culture, history, geography, literature, or context, even if these are Russian.
-- the puzzle uses Russian names/entities but no Russian linguistic tricks
-- the translated puzzle still is solvable without losing the key idea.
Given the puzzle question, answer, comment, notes in Russian and the English translation of the puzzle, output your reasoning in English within ... tags and your answer within ... tags.
Russian-Specific Filtering Prompt (Few-Shot Example Format)
Russian puzzle:
Question: {question}
Answer: {answer}
Comment: {comment}
Notes: {notes}
English translation:
Question: {question_en}
Answer: {answer_en}
Comment: {comment_en}
Notes: {notes_en}
{reasoning}
{shot_answer}
Table 7:
Few-shot prompt used to filter Russian-specific questions (See §
3.1
for details).
Creative vs. Factual Reasoning Annotation Prompt (Instruction)
You are an expert in annotating type of reasoning involved in solving creative thinking puzzles. Given a creative thinking puzzle question, its answer, comments about the puzzle and other acceptable answers (if any), do TWO tasks:
1) Decide whether solving the puzzle mainly requires SIMPLE FACTUAL reasoning or CREATIVE ASSOCIATION of distant pieces of knowledge/facts.
-- Answer ’factual’ when the puzzle is answered by simply retrieving specific facts, dates, names, or straightforward commonsense reasoning, with no need to make a non-obvious, creative leaps between facts.
-- Answer ’creative’ when the solver in addition to retrieving the facts, must also make a non-obvious, creative connection, see a hidden twist, reinterpret words, or combine clues in an indirect way.
2) If your answer to the previous task is ’CREATIVE’, then choose ALL relevant creativity concepts involved in solving the puzzle from the following list (use EXACT spelling, lowercase):
poem, metaphor, idiom, proverb, joke, pun, simile, sarcasm, hyperbole, neologism, analogy, abstraction, lateral thinking, divergent thinking, commonsense reasoning, compositionality.
-- Choose ALL that apply (multi-label).
-- If none of the specific labels fit, suggest a list of new labels that are relevant for the puzzle.
Carefully read the given puzzle and output:
-- One line with the reasoning type: factual OR creative.
-- If you chose ’creative’, on separate lines output ONE OR MORE tags, each containing either exactly one label from the allowed list or a new label if none fits.
-- If you chose ’factual’, do NOT output any tags.
Creative vs. Factual Reasoning Annotation Prompt (Few-Shot Example Format)
Example {index}:
Puzzle:
{question}
Answer: {answer}
Comment: {comment}
Acceptable Answers: {notes}
Annotation:
{shot_answer}
Table 8:
Few-shot prompt used for creative vs. factual reasoning annotation ( See §
3.1
for details).
Domain Annotation Prompt (Instruction)
You are an expert in annotating creative thinking puzzles. Given a creative thinking puzzle, its answer, comments about the puzzle and other acceptable answers (if any), identify a list of domains that are involved in solving the puzzle. A domain is a specific area of knowledge, expertise, or human activity such as physics, literature, sports, music, etc. Wrap each domain on a separate line in ... tags.
Domain Annotation Prompt (Few-Shot Example Format)
Example {index}:
Puzzle:
{question}
Answer: {answer}
Comment: {comment}
Acceptable Answers: {notes}
Domains:
{domains}
Table 9:
Few-shot prompt used for knowledge domain annotation (§
3.2
).
Knowledge Annotation Prompt (Instruction)
You are an expert in annotating creative thinking puzzles. Given a creative thinking puzzle, identify a list of knowledge/facts that are explicitly required to answer the puzzle, and write them in the form of independent questions. Don’t solve the problem. Don’t include the answer itself. Wrap each question on a separate line in ... tags.
Knowledge Annotation Prompt (Few-Shot Example Format)
Example {index}:
Puzzle:
{question}
Answer: {answer}
Comment: {comment}
Acceptable Answers: {notes}
Knowledge:
{knowledge}
Table 10:
Few-shot prompt used for knowledge decomposition annotation (See §
6
).
Culture/Demographics Annotation Prompt (Instruction)
You are an expert in annotating creative thinking puzzles. Given a creative thinking puzzle, its answer, comments about the puzzle and other acceptable answers (if any), identify the languages/cultures that are involved in solving the puzzle. Output only your final answer and separate them with commas.
Culture/Demographics Annotation Prompt (Few-Shot Example Format)
Example {index}:
Puzzle:
{question}
Answer: {answer}
Comment: {comment}
Acceptable Answers: {notes}
Cultures/Languages: {cultures}
Table 11:
Few-shot prompt used for culture and demographics annotation (§
3.2
).
Chain-of-Thought Prompt (English)
You are an expert at solving creative thinking puzzles. You are given a puzzle question that requires creative reasoning over real-world knowledge. Note that the question might contain several placeholder words (often in all capital case such as X, Y, THIS, FIRST, SECOND, ALPHA, BETA, HE, SHE, IT, HIM, HIS, HER, THEY, THEM etc.) that substitute for specific entities, objects, or concepts. These placeholders are crucial for solving the puzzle, and their meaning can only be inferred through careful reasoning about the question. Additionally, some placeholders might be gendered (e.g., HE/HIM/HIS vs SHE/HER), but do not assume that the gendered pronouns necessarily refer to human characters; they could refer to any entities, and their gender might be different. Given a puzzle question, think step by step and write your reasoning within ... tags. Then provide only the final answer within ... tags.
Puzzle:
{question}
Chain-of-Thought Prompt (Russian)
You are an expert at solving creative thinking puzzles. You are given a puzzle question in Russian that requires creative reasoning over real-world knowledge. Note that the question might contain several placeholder words (often in all capital case such as
ИКС, ИГРЕК, ЭТО, ПЕРВЫЙ, ВТОРОЙ, АЛЬФА, БЕТА, ОН, ОНА, ЕГО, ЕЕ, ОНИ
etc.) that substitute for specific entities, objects, or concepts. These placeholders are crucial for solving the puzzle, and their meaning can only be inferred through careful reasoning about the question. Given a puzzle question, think step by step and write your reasoning in Russian within ... tags. Then provide only the final answer in Russian (unless specified otherwise) within ... tags.
Puzzle:
{question}
Thinking Model Prompt (English)
You are an expert at solving creative thinking puzzles. You are given a puzzle question that requires creative reasoning over real-world knowledge. Note that the question might contain several placeholder words (often in all capital case such as X, Y, THIS, FIRST, SECOND, ALPHA, BETA, HE, SHE, IT, HIM, HIS, HER, THEY, THEM etc.) that substitute for specific entities, objects, or concepts. These placeholders are crucial for solving the puzzle, and their meaning can only be inferred through careful reasoning about the question. Additionally, some placeholders might be gendered (e.g., HE/HIM/HIS vs SHE/HER), but do not assume that the gendered pronouns necessarily refer to human characters; they could refer to any entities, and their gender might be different. Please provide your final answer within ... tags.
Puzzle:
{question}
Thinking Model Prompt (Russian)
You are an expert at solving creative thinking puzzles. You are given a puzzle question in Russian that requires creative reasoning over real-world knowledge. Note that the question might contain several placeholder words (often in all capital case such as
ИКС, ИГРЕК, ЭТО, ПЕРВЫЙ, ВТОРОЙ, АЛЬФА, БЕТА, ОН, ОНА, ЕГО, ЕЕ, ОНИ
etc.) that substitute for specific entities, objects, or concepts. These placeholders are crucial for solving the puzzle, and their meaning can only be inferred through careful reasoning about the question. Please provide your final answer in Russian (unless specified otherwise) within ... tags.
Puzzle:
{question}
Table 12:
Evaluation prompts for non-thinking (CoT) and thinking (reasoning) models in English and Russian (§
4
).
LLM-as-a-Judge Prompt
You are an expert in judging model responses. Given a question, a reference answer, a model answer, and additional context such as comments and other acceptable answers, decide whether the model’s answer is correct based on the reference answer and context. Ignore minor typos, articles, capitalization, and formatting. If the factual core is the same, answer yes. Also ignore the number of words constraint in the question if the model answer is semantically correct but does not meet the word count requirement. Answer only: Yes or No.
Question:
{question}
Reference Answer:
{answer}
Additional Context:
Comments: {comment}
Acceptable Answers: {notes}
Model Answer:
{prediction}
Table 13:
Zero-shot prompt used for LLM-as-a-judge evaluation (§
4
).
Figure 7:
Performance by domains on
CresOWLve
-En.
Figure 8:
Performance by creativity concepts on
CresOWLve
-En.
Figure 9:
Performance by cultures on
CresOWLve
-En.
Figure 10:
Performance by domains on
CresOWLve
-Ru.
Figure 11:
Performance by creativity concepts on
CresOWLve
-Ru.
Figure 12:
Performance by cultures on
CresOWLve
-Ru.
Figure 13:
Exact Match Performance by difficulty.
Figure 14:
Exact Match Performance by reasoning category.
Figure 15:
Distribution of difficulty levels for creative and factual questions.
Figure 16:
Correlations between question difficulty and complexity features.
Figure 17:
Correlations between model performance and complexity features.