How Do Hackathons Foster Creativity? Towards AI Collaborative Evaluation of Creativity at Scale
Jeanette Falk, Yiyi Chen, Janet Rafner, Mike Zhang, Johannes Bjerva, Alexander Nolte · 2025
Read How Do Hackathons Foster Creativity? Towards AI Collaborative Evaluation of Creativity at Scale by Jeanette Falk, Yiyi Chen, Janet Rafner, Mike Zhang, Johannes Bjerva, Alexander Nolte (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04290.txt.
Introduction
Jeanette Falk, Yiyi Chen, Janet Rafner, Mike Zhang, Johannes Bjerva, and Alexander Nolte study hackathons in a 2025 paper. Hackathons are short team events where people build new ideas and early versions of products fast. Case reports show creative results, but large studies of how the format shapes those results have been missing.
The team ran a computer based study of 193,353 hackathon projects and narrowed the set to 10,363 for close review. They scored creative work by how useful and how new it was. Their study links team traits, work patterns, and event setup to creative results. They also test large language models as helpers for judging creative work at scale, and weigh the gains and limits of that use.
Catalog record
| CREATOR | Jeanette Falk, Yiyi Chen, Janet Rafner, Mike Zhang, Johannes Bjerva, Alexander Nolte |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 116 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-04290
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Do Hackathons Foster Creativity? Towards AI Collaborative Evaluation of Creativity at Scale Jeanette Falk, Yiyi Chen, Janet Rafner, Mike Zhang, Johannes Bjerva, Alexander Nolte Abstract . Hackathons have become popular collaborative events for accelerating the development of creative ideas and prototypes. There are several case studies showcasing creative outcomes across domains such as industry, education, and research. However, there are no large-scale studies on creativity in hackathons which can advance theory on how hackathon formats lead to creative outcomes. We conducted a computational analysis of 193,353 hackathon projects. By operationalizing creativity through usefulness and novelty, we refined our dataset to 10,363 projects, allowing us to analyze how participant characteristics, collaboration patterns, and hackathon setups influence the development of creative projects. The contribution of our paper is twofold: We identified means for organizers to foster creativity in hackathons. We also explore the use of large language models (LLMs) to augment the evaluation of creative outcomes and discuss challenges and opportunities of doing this, which has implications for creativity research at large. 1. Introduction Reacting fast to rapid changes in society and technology development has generally been considered a prerequisite for innovation ( Ringel et al., 2015 ) . Originating in niche software development environments in Silicon Valley during the late 1990’s, hackathons have become a popular format for accelerating people’s creativity and developing creative ideas, prototypes, products, or services in a variety of contexts such as education ( Schulten and Chounta, 2024 ) , entrepreneurship ( Irani, 2015 ) , corporations ( Pe-Than et al., 2022 ) , scientific communities ( Huppenkothen et al., 2018 ) , civic engagement ( Schrock, 2016 ) and others ( Falk Olesen and Halskov, 2020 ) . Hackathons are time-bounded, participant-driven design events, often spanning only a few days, where participants form teams and collaborate on projects to address a common theme or challenge, resulting in a perceptible outcome, such as an interactive prototype ( Falk et al., 2024 ) . They are often celebrated for their potential for creative outcomes, see e.g. ( Attalah et al., 2023 ; Briscoe, 2014 ; Taylor and Clarke, 2018 ; Frey and Luks, 2016 ; Justine Lobbe and Sagot, 2021 ) . For this reason, hackathons have become an attractive approach for entrepreneurs, companies, educators and researchers to develop creative solutions to problems they face or to complement or enhance their existing innovation processes ( Falk Olesen and Halskov, 2020 ) . Since 2016, the CHI community has shown a growing interest in hackathons. A search for “hackathon” in the CHI proceedings on the ACM Digital Library returns 87 results for full research papers. Most research focusing on creativity in hackathons, however, consists of case studies of one or only a few events ( Gama et al., 2023 ; Rosell et al., 2014 ; Briscoe, 2014 ; Frey and Luks, 2016 ) . While they can provide valuable insights, case studies are not suitable to advance and validate theory, either on hackathons as a phenomenon in itself or on how hackathon formats lead to creative outcomes. Answering fundamental research questions in this frame requires larger-scale quantitative studies in order to test hypotheses ( Edmondson and Mcmanus, 2007 ; Falk et al., 2024 ) . Current quantitative studies of hackathons focus mainly on understanding project continuation ( Nolte et al., 2020a ; McIntosh and Hardin, 2021 ) and re-use of code that was created during hackathons ( Mahmoud et al., 2022 ; Imam et al., 2021 ; Imam and Dey, 2021 ) . To the best of our knowledge, as of yet, no research has conducted large-scale analysis of hackathon projects from a creativity perspective. Our work addresses this gap by asking the following first research question: RQ1 : How can we define and implement an analysis of creativity in a way that enables large-scale analysis of creativity within hackathon projects? Large-scale creativity assessment is essential for validating the effectiveness of the different contexts in which hackathons are organized for supporting creative outcomes – such as in educational programs, workplace training, and other interventions – ensuring that the formats are successfully cultivating the creative skills demanded ( Rafner et al., 2022 ) . From this perspective, Rafner and colleagues have suggested that exploring methods which combine scalability “and product-oriented assessments (data with high ecological validity) could greatly enhance the construct validity of creativity assessment instruments” ( Rafner et al., 2022 ) . In our paper, we contribute to exploring such scalable methods with potentially high ecological validity for assessing creative products in the context of hackathons, since the data we analyze resemble real-world data rather than data created in controlled creativity test settings ( Rafner et al., 2023 ) . We thus also address the following second research question: RQ2 : What insights into participants, collaboration patterns, and hackathon setups can we gain from analyzing a large number of creative hackathon projects and how do these insights relate to fostering creativity? Similar to the contributions by ( Nolte et al., 2020a ; Imam et al., 2021 ; Imam and Dey, 2021 ; Mahmoud et al., 2022 ) , we use Devpost’s database of hackathons and hackathon projects for our large-scale analysis of creativity. 1 1 1 https://devpost.com/ Similar to Fang, Herbsleb and Vasilescu ( Fang et al., 2024 ) , we operationalize a definition of creativity which we explore as a lens to analyze our dataset. Specifically, we operationalize the concepts of novelty and usefulness to create a subset of creative projects. We conduct statistical analysis on this subset of creative projects to identify patterns in the interaction of aptitude, process and environment, a contribution which we frame as particularly valuable for researchers and practitioners who organize hackathons and wish to support hackathon participants’ creativity and increase the potential for creative outcomes. Furthermore, to address calls for exploring a Generative AI-augmented expert assessment ( Beghetto, 2019 ; Rafner et al., 2023 ; Luchini et al., 2023 ) , we explore the compelling case of Large Language Models (LLMs)-as-a-judge to augment our large-scale creativity evaluation method and address the following research question: RQ3 :How might LLMs be used to augment large-scale evaluations of creative hackathon projects? The contribution of this two-pronged large-scale exploration of creativity in hackathon projects is the following: In addressing RQ1, we arrive at an operationalization of creativity theory which guides the first prong of our two-pronged approach; a statistical analysis of a dataset consisting of 10,363 hackathon projects. We discuss the findings from this statistical analysis as take-aways for organizers, thereby addressing RQ2. As the second prong of the two-pronged approach, we address RQ3 by using LLMs to analyze the creativity of a subset consisting of 21,318 randomly sampled hackathon projects which contained both creative and non-creative projects, as defined by our operationalization (RQ1). We also compared the ratings of LLM judges and human judges on a randomly selected subset of 30 hackathon projects. Drawing from the LLM analysis, we explore how LLMs can function as supplementary judges, supporting human judgment within a human-AI hybrid intelligent system ( Sherson et al., 2024 ) , and how they can facilitate large-scale analyses of hackathon creativity. The large-scale exploration of creativity in hackathons contributes towards a greater