No Evidence for LLMs Being Useful in Problem Reframing
Joongi Shin, Anna Polyanskaya, Andrés Lucero, Antti Oulasvirta · 2025
Read No Evidence for LLMs Being Useful in Problem Reframing by Joongi Shin, Anna Polyanskaya, Andrés Lucero, Antti Oulasvirta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01631.txt.
Introduction
Joongi Shin, Anna Polyanskaya, Andrés Lucero, and Antti Oulasvirta tested in twenty twenty-five whether language models help designers reframe problems. Reframing recasts what a design problem is about, a creative step that demands fresh yet fitting perspectives. The team compared three ways of working with these models across two hundred eighty participants: free-form use, direct generation, and a structured method based on reframing theory.
None of the conditions improved the quality of the frames produced. Worse, model use widened the gap between experienced and novice designers, and novices felt less agency when working with a model. The authors conclude there is no evidence of benefit in this setting and discuss why the tools may fall flat at this particular creative task.
Catalog record
| CREATOR | Joongi Shin, Anna Polyanskaya, Andrés Lucero, Antti Oulasvirta |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 89 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01631
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
No Evidence for LLMs Being Useful in Problem Reframing Joongi Shin, Anna Polyanskaya, Andrés Lucero, Antti Oulasvirta Abstract . Problem reframing is a designerly activity wherein alternative perspectives are created to recast what a stated design problem is about. Generating alternative problem frames is challenging because it requires devising novel and useful perspectives that fit the given problem context. Large language models (LLMs) could assist this activity via their generative capability . However, it is not clear whether they can help designers produce high-quality frames. Therefore, we asked if there are benefits to working with LLMs. To this end, we compared three ways of using LLMs ( N = 280 N=280 ): 1) free-form, 2) direct generation, and 3) a structured approach informed by a theory of reframing. We found that using LLMs does not help improve the quality of problem frames. In fact, it increases the competence gap between experienced and inexperienced designers. Also, inexperienced ones perceived lower agency when working with LLMs. We conclude that there is no benefit to using LLMs in problem reframing and discuss possible factors for this lack of effect. DOI: 10.1145/3706598.3713273 ISBN: 979-8-4007-1394-1/25/04 Conference: CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26-May 1, 2025, Yokohama, Japan CCS: Human-centered computing Human computer interaction (HCI) CCS: Human-centered computing Natural language interfaces CCS: Human-centered computing User studies 1. Introduction Problem reframing is a crucial designerly activity. By reframing a problem, a designer can make a challenging design problem more solvable ( Dorst, 2015 ; Schön, 2017 ; Haase and Laursen, 2019 ; Paton and Dorst, 2011 ) . Therefore, a good problem frame for design is both novel and useful ( Runco and Jaeger, 2012 ) . Developing such frames requires exploring alternative ways to grapple with what a problem is about ( Dorst and Cross, 2001 ; Grant and Berry, 2011 ) . These points are illustrated by the “slow elevator problem,” outlined in Figure 1 . Here, employees complain that elevators are too slow, while simply increasing the elevators’ speed would cause safety issues. By shifting perspective from seeing the problem as elevator speed (i.e., an initial frame) to the frustrating waiting experience (i.e., an alternative frame), designers could address employee complaints by improving the waiting area. Without such reframings, fundamental problems might remain hidden, rendering the attempted solutions ineffective ( Dorst and Cross, 2001 ) . However, problem reframing is inherently challenging and labor-intensive, even for expert designers ( Dorst, 2015 ) . It demands putting great effort into understanding problems and altering perspectives ( Dorst, 2015 ) . Designers need to iteratively “destructure” the initial problem, explore diverse stakeholder perspectives, synthesize overarching problem themes, and critically evaluate potential solutions ( Dorst, 2018 ) . Regrettably, such efforts can get caught in design fixation, which limits one’s ability to explore problem frames ( Dorst and Cross, 2001 ) . Failing in problem reframing may lead a designer to believe that the given problem is wicked or not solvable. An illustration showing the concept of problem reframing. It shows an example problem and how it can lead to a limited solution without reframing or an actionable solution with reframing. Figure 1. Problem reframing helps designers attack thorny design problems by rethinking what they are about . In this paper, we study whether LLMs can help designers arrive at better problem frames. An illustration showing the concept of problem reframing. It shows an example problem and how it can lead to a limited solution without reframing or an actionable solution with reframing. With this paper, we look at whether working with AI – specifically, large language models (LLMs) – benefits designers in devising high-quality problem frames 1 1 1 We present an interactive demo of our project at https://joongishin.github.io/problemReframing_llm/ . The question is timely: designers could be lured into using LLMs without understanding the effects on their creative potential. The use of LLMs in design activities has emerged as an important topic in the field of HCI ( Shin et al., 2023 ; Schmidt et al., 2024 ; Li et al., 2024c ) . Yet there are conflicting reports on how well LLMs can assist with creative activities. On the one hand, prior studies have demonstrated LLMs’ competence in generating various ideas that can inspire people to develop a wider range of ideas ( Noy and Zhang, 2023 ; Shaer et al., 2024 ; Suh et al., 2024 ; Feng et al., 2024 ) . The optimistic view has been that human–AI collaboration in creative activities lets practitioners focus on the core creative activities while delegating effortful tasks to AI ( Li et al., 2024a ; Schmidt et al., 2024 ) . On the other hand, some recent studies highlight LLMs’ limitations in generating original ideas, thus implying that reliance on what LLMs generate can lead to less diversity of ideas ( Doshi and Hauser, 2024 ; Anderson et al., 2024 ; Koivisto and Grassini, 2023 ) . Despite the conflicting evidence, scholars have not given rigorous empirical attention specifically to LLM-assisted problem reframing. Research thus far has focused on uncovering LLMs’ competence in generating alternative frames ( Chen et al., 2024 ; Jiang and Luo, 2024 ; Einarsson et al., 2024 ; Sandholm et al., 2024 ) , without considering their influence on designers. We argue that it is critical to consider designers and LLMs together when adressing this question. This paper contributes the first rigorously planned study to assess the benefits and drawbacks of using LLMs in problem reframing. When planning the study, we wanted to capture both experiential and quality-related effects. Therefore, we focused on how using LLMs influences designers’ creation of novel and original problem frames, alongside their perceptions of agency, ownership, and the helpfulness of LLMs. Furthermore, we investigated how those influences might differ with designers’ level of expertise in problem reframing. In all, we tested four hypotheses about using LLMs in problem reframing (Section 3 ). Reflecting the fact that LLMs can be used in a variety of ways beyond prompting, our research covered more than one approach to using them. We included three distinctive ways of using LLMs found in the literature about creative activities ( Anderson et al., 2024 ; Suh et al., 2024 ; Xu et al., 2024 ; Jiang and Luo, 2024 ; He et al., 2024 ) : 1) a direct approach wherein designers interact with LLM-generated problem frames only; 2) a structured one in which they reframe problems with LLMs in a step-by-step process, here following a nine-step process articulated by Kees Dorst ( Dorst, 2015 ) ); and 3) a free-form approach wherein designers converse with LLMs to reframe problems in their own strategy. We tested each approach with OpenAI’s GPT-4o 2 2 2 https://platform.openai.com/docs/models/gpt-4o , the best-performing model publicly available at the time of our experiment (in August 2024) 3 3 3 https://huggingface.co/spaces/ArtificialAnalysis/LLM-Performance-Leaderboard (note that GPT-o1 models were released in September 2024.) . The study involves a large number of participants with design experience ( N=280 ). The sample size was designed to achieve sufficient power for detecting medium effect sizes with high