Spark : A System for Scientifically Creative Idea Generation

Aishik Sanyal, Samuel Schapiro, Sumuk Shashidhar, Royce Moon, Lav R. Varshney, Dilek Hakkani-Tur · 2025

Read Spark : A System for Scientifically Creative Idea Generation by Aishik Sanyal, Samuel Schapiro, Sumuk Shashidhar, Royce Moon, Lav R. Varshney, Dilek Hakkani-Tur (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20090.txt.

Introduction

Research ideas are the currency of science, and language models now mint them freely; judging them is the bottleneck. Aishik Sanyal and co-authors present Spark, a system that pairs retrieval-fed idea generation with a reviewer model called Judge, trained on six hundred thousand peer reviews from OpenReview. The generator proposes; the judge scores like a program committee might.

The demonstration is also a manifesto, inviting computational creativity researchers to ground machine ideation in their field's principles. The annotated review dataset is released for others to build on. Whether models can be genuinely creative stays open; Spark gives the question a testbed.

Catalog record

CREATORAishik Sanyal, Samuel Schapiro, Sumuk Shashidhar, Royce Moon, Lav R. Varshney, Dilek Hakkani-Tur
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-20090

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Spark : A System for Scientifically Creative Idea Generation Aishik Sanyal, Samuel Schapiro, Sumuk Shashidhar, Royce Moon, Lav R. Varshney, Dilek Hakkani-Tur Abstract Recently, large language models (LLMs) have shown promising abilities to generate novel research ideas in science, a direction which coincides with many foundational principles in computational creativity (CC). In light of these developments, we present an idea generation system named Spark that couples retrieval-augmented idea generation using LLMs with a reviewer model named Judge trained on 600K scientific reviews from OpenReview. Our work is both a system demonstration and intended to inspire other CC researchers to explore grounding the generation and evaluation of scientific ideas within foundational CC principles. To this end, we invite other researchers to explore the use of LLMs for idea generation and creative evaluations, and release the annotated dataset used to train Judge . 1 1 1 https://huggingface.co/datasets/spiralworks/openreview_wildcard_2025_v2 Introduction Automating parts of the scientific process has long been a research goal in the field of computer science and artificial intelligence (AI) [ \citeauthoryearLangley1977 , \citeauthoryearThagard1988 , \citeauthoryearJumper et al.2021 ] , with computational creativity (CC) principles playing a crucial role in many parts of this endeavor. Crucially, the initial task of idea or hypothesis generation is an area that requires creative intuition but has historically been restricted to the ingenuity of human scientists. Recent research, however, has demonstrated that large language models (LLMs) are capable of generating novel research ideas [ \citeauthoryearWang et al.2024 , \citeauthoryearSi, Yang, and Hashimoto2024 , \citeauthoryearYang et al.2023 ] . In fact, a handful of fully autonomous AI systems capable of writing full-length papers at near-human quality, or workshop-length papers that successfully pass through human peer review, have been developed [ \citeauthoryearLu et al.2024 , \citeauthoryearIntology2025 ] . In each of these systems, idea generation remains the crucial first step. In this paper, we present a scientific idea generation system named Spark that generates novel scientific ideas and an evaluator model named Judge , trained on a 600K size dataset of expert human reviews from OpenReview, that can evaluate the creativity of Spark ’s proposed ideas. Our goal is to ground recent developments in scientific idea generation with the foundational computational creativity principles that underpin much of their work, focusing on the first stage of idea generation. To facilitate reproducibility and community progress, we are releasing the full OpenReview dataset used to train our Judge model. Finally, to provide background context, we first start by summarizing recent developments in the use of LLMs for scientific discovery. Related Work Large language models (LLMs) have been increasingly explored as tools to support creative ideation in scientific research. Early efforts such as Meta’s Galactica utilized specialized scientific corpora to facilitate technical tasks like knowledge retrieval and hypothesis generation [ \citeauthoryearTaylor et al.2022 ] . Despite strong benchmark performance, Galactica encountered significant issues with factual grounding and hallucinated references, highlighting the need for improved reliability. Addressing grounding explicitly, SciPIP integrated semantic retrieval and citation awareness into idea generation, producing hypotheses informed by current literature but primarily evaluated within NLP domains [ \citeauthoryearWang et al.2025 ] . Extending this further, SciMON employed iterative refinement to systematically maximize the novelty of ideas, but their outputs often suffered from low practical feasibility and experimental utility [ \citeauthoryearWang et al.2024 ] . Drawing inspiration from collaborative research processes, VirSci orchestrated multiple LLM agents within a virtual ecosystem to simulate team-based ideation, achieving measurable improvements in idea diversity and research-trend alignment [ \citeauthoryearWang and Clark2024 ] . Nevertheless, like SciMON, VirSci primarily focused on novelty without directly optimizing for experimental usefulness. Other efforts aimed at automating entire research cycles include Sakana AI’s AI Scientist, which autonomously generates and experimentally validates hypotheses but still requires human oversight for accuracy and consistency [ \citeauthoryearLu et al.2024 ] . Similarly, IntologyAI’s Zochi employed multi-agent collaboration to produce workshop-accepted research papers across domains, yet it involves significant human intervention and computational resources [ \citeauthoryearIntology2025 ] . In this paper, we present Spark , a demonstration aimed at grounding scientific idea generation in foundational computational creativity principles. We illustrate this approach by integrating literature retrieval, idea generation, and simulated peer-review evaluation into a unified workflow. Additionally, we invite further exploration from the CC community by releasing the OpenReview dataset used to train the Judge model. Methods The Spark system integrates AI-driven components that support computational creativity in the generation of scientific ideas, utilizing specialized agent-based modules for the retrieval of literature and the synthesis, refinement and evaluation of ideas. Figure 1 illustrates the interaction between the Spark and Xplor systems. Literature Retrieval via Xplor Inspired by PaperQA [ \citeauthoryearLála et al.2023 ] , Xplor handles literature retrieval and analysis tasks tailored specifically for the Spark system. It utilizes OpenAI embeddings, FAISS indexing, maximal marginal relevance (MMR), and a recursive, agent-based retrieval loop to systematically gather, analyze, and refine relevant research documents to later be used for idea generation. Text Embedding and Retrieval: Xplor employs OpenAI’s text-embedding-3-large model [ \citeauthoryearOpenAI2024 ] to convert retrieved documents into 1536-dimensional semantic vectors. These embeddings allow accurate semantic comparison of research papers. FAISS [ \citeauthoryearDouze et al.2024 ] efficiently indexes these vectors, enabling rapid retrieval through cosine similarity-based nearest-neighbor searches when provided with user-generated input queries. Relevance Scoring and Maximal Marginal Relevance: After an initial vector-based retrieval, each candidate text chunk is also evaluated using an LLM-based summary model. This model assigns relevance scores (1 to 10) to each chunk according to its relevance to the research query, complementing the earlier cosine similarity measures. Subsequently, Xplor applies maximal marginal relevance (MMR) to re-rank these scored chunks, a technique which prioritizes both relevance and diversity. This ensures that selected documents collectively provide comprehensive coverage with minimal redundancy. Recursive Retrieval and Evidence Gathering Loop: The retrieval process in Xplor is recursive and agent-driven. Initially, the system queries databases (e.g., arXiv, Google Scholar) using a programmable search engine, retrieving documents relevant to the initial question. First, documents are segmented into overlapping chunks and embedded. Then, an agentic loop iteratively refines the search by generating new, increasingly precise queries informed by intermediate summaries from an LLM response. This process continues until specific depth and coverage criteria, such as accumulating at least five evidence snippets from multiple sources, are met. Figure 1: Spark ’s end‑to‑end pipeline: Xplor performs embedding‑based, recursive literature retrieval with MMR re‑ranking; the Spark Idea Generator extracts key concepts and uses chain‑of‑thought LLM prompting to synthesize seed research proposals; and the Spark Filter applies our supervised fine-tuned judge model and a decision agent for automated peer‑style critique and ranking. Xplor has two distinct operating modes: interactive and autonomous. In interactive mode, Xplor is directly prompted by the user query, whereas in autonomous mode, Xplor takes queries from the Spark Idea Generator agent to aid in the generation and refinement of ideas, as explained in the following section. Idea Generation with Spark Idea Generator The Spark Idea Generator constitutes the initial creative stage of the Spark system, formulating seed research proposals based on the earlier Xplor search results. The idea generation system identifies open research problems related to the query and then synthesizes concepts to generate structured, contextually relevant, and well-motivated ideas. A) Input: • Problem: “LLMs struggle with consistency”

Read the complete text