Brickify : Enabling Expressive Design Intent Specification through Direct Manipulation on Design Tokens Conference: CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26-May 1, 2025, Yokoha

Xinyu Shi, Yinghou Wang, Ryan Rossi, Jian Zhao · 2025

Read Brickify : Enabling Expressive Design Intent Specification through Direct Manipulation on Design Tokens Conference: CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26-May 1, 2025, Yokoha by Xinyu Shi, Yinghou Wang, Ryan Rossi, Jian Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-21219.txt.

Introduction

Xinyu Shi and three co-authors present a design tool that lets users state visual intent by handling tokens, in a 2025 paper. Designers struggle to put visual details into words for AI tools. Their system pulls elements like subject, style, and color from reference images. Each element becomes a token that can be resized, grouped, and linked.

The token set forms a visual vocabulary for the design. It shows both which elements are wanted and how they combine. An end-to-end pipeline lets AI models read and execute that vocabulary. Experienced designers tested the tool against text prompting. They found it faster and more intuitive for detailed work. Exploring options and refining complex designs felt easier. The authors argue direct handling beats verbal description for visual intent.

Catalog record

CREATORXinyu Shi, Yinghou Wang, Ryan Rossi, Jian Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE102 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-21219

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Brickify : Enabling Expressive Design Intent Specification through Direct Manipulation on Design Tokens Conference: CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26-May 1, 2025, Yokohama, Japan DOI: 10.1145/3706598.3714087 ISBN: 979-8-4007-1394-1/25/04 CCS: Human-centered computing Graphical user interfaces CCS: Human-centered computing Interactive systems and tools CCS: Applied computing Arts and humanities Xinyu Shi, Yinghou Wang, Ryan Rossi, Jian Zhao Abstract . Expressing design intent using natural language prompts requires designers to verbalize the ambiguous visual details concisely, which can be challenging or even impossible. To address this, we introduce Brickify , a visual-centric interaction paradigm — expressing design intent through direct manipulation on design tokens . Brickify extracts visual elements (e.g., subject, style, and color) from reference images and converts them into interactive and reusable design tokens that can be directly manipulated (e.g., resize, group, link, etc.) to form the visual lexicon . The lexicon reflects users’ intent for both what visual elements are desired and how to construct them into a whole. We developed Brickify to demonstrate how AI models can interpret and execute the visual lexicon through an end-to-end pipeline. In a user study, experienced designers found Brickify more efficient and intuitive than text-based prompts, allowing them to describe visual details, explore alternatives, and refine complex designs with greater ease and control. Figure 1 . Brickify introduces a visual-centric interaction paradigm to specify design intent for controllable image generation: (a) users start with reference images and identify design elements (subjects, styles, colors, and concepts) which can be reified into interactive, reusable design tokens; (b) then directly manipulate on these tokens to build a visual lexicon to express how to construct these elements as a whole; (c) explore alternative compositions by reusing tokens and refining the visual lexicon. The figure illustrates the \textsc{\sysname{}} interaction paradigm in three stages: (a) Reifying elements into design tokens: users begin by selecting specific subjects, styles, colors, and textual concepts from reference images. These elements are transformed into interactive design tokens. For example, subject tokens (such as an owl or a car) and style tokens (like a colorful abstract pattern) are extracted from the images. (b) Direct manipulation on tokens: users can manipulate these design tokens to construct a visual lexicon, specifying relationships between elements. The figure shows the interface for adjusting positions, sizes, and linking textual descriptors such as ``colorful'' or ``standing, facing left.'' (c) Alternative token constructions: Users can explore variations by reusing design tokens and fine-tuning the visual lexicon. The figure shows examples of different compositions generated by rearranging and modifying the visual lexicon, resulting in alternative creative outcomes. 1. Introduction Design is about the choices of visual elements (e.g., subject, style, color) and the construction of them towards an intended effect ( Dondis 1974 ) ; the decisions involved in this process are the art of design, embraced with a designer’s creativity ( Goldschmidt and Tatsa 2005 ) . However, current text-to-image generation tools (e.g., DALL-E, Midjourney) shift design decision-making from designers to models, which are designed to create for users rather than work with designers ( Satyanarayan and Jones 2024 ; Weisz et al. 2024 ; Subramonyam et al. 2024 ) . Despite their ability to produce aesthetically appealing images for casual use, without designer’s thoughtful planning, they lack the capability to create meaningful and professional design solutions that effectively convey intended visual messages ( Shi et al. 2023 ; Chen et al. 2023 ; Choi et al. 2024 ) . One key barrier to keeping designers in-the-loop is communication ( Weisz et al. 2024 ; Subramonyam et al. 2024 ) — generative AI tools are designed to receive instructions textually , while designers often prefer to think and communicate visually ( Tschimmel 2012 ; Ware 2010 ) . Consider the design case of crafting a Halloween poster: the designer collects some reference images (Figure 2 a- 2 f), thinks about what visual elements to use, and after a while, forms a rough idea (Figure 2 g) about how to compose them into a whole, and wants to work with generative AI tools for quick prototyping. However, the following three challenges arise. First, clearly describing what visual elements to use in natural language is challenging due to inconsistent naming standards ( Heider 1972 ; Murphy 2004 ) . For example, colors in Figure 2 a might be described as “light red, greenish teal, navy blue, with greyish black” by some designers, and simply as “orange, green, blue, and black” by others. These subtle differences can lead to significant variations in hues and shades. Uploading reference images for model to “see” ( Choi et al. 2024 ; Peng et al. 2024 ) can help when the desired element dominates the image as in Figure 2 d. However, in complex images as Figure 2 c, specifying an exact element is harder. Designers might describe it as “the abstract shape with curves in the middle right” , but models often struggle with such ambiguous descriptions regarding the shape and position. Further, precisely verbalizing how to construct those visual elements into a whole is hard. Deciding on relative scale and proximity is to relate the isolated elements with each other as interacting parts ( Dondis 1974 ; Ware 2010 ) . However, both scale and proximity are continuous values, while language is often too discrete for fine-grained instructions. For example, describing Figure 2 h as “five pumpkins in front of a large building, with a moon above” lacks precise size and position details, making it hard for the model to interpret their nuanced spatial relationship. The designer might finally obtain a reasonable version after multiple rounds of conversations with the tool, e.g., “make the building a little bit larger” , but this process is often time-consuming and tedious without guaranteed outcomes. Lastly, the choices of elemental construction are infinite; however, prompting with texts limits the flexibility to reuse those visual elements to explore alternative constructions. Designers need to copy-and-paste an entire paragraph from previous prompts, then modify certain parts to change relationships or replace visual elements. Each iteration of exploration requires users to manually track where to change and where to keep among lengthy texts, which is tedious and error-prone. It is because existing generative AI tools treat each prompt as an independent request, without a mechanism to selectively separate persistent information (e.g., visual characteristics) from single-use prompts, making it inefficient to share information across multi-turn interactions. Figure 2 . Design example of a Halloween party poster, showing (a) the color palette, (b-f) reference images with highlighted elements, and (g) the envisioned poster in designer’s mind. (h) illustrates the spatial relationships between pumpkins, building, and moon. We have obtained the designer’s consent to include this design in the paper. The figure illustrates how different reference images contribute to the composition of a Halloween poster. On the left, various reference images are shown: a color palette with four swatches (red, blue, green, and black), a stylized sun over water, an architectural design highlighting a red curved shape, a classic Halloween pumpkin decoration, a modern building with a diagonal roof structure, and a vector-style bat illustration with one bat

Read the complete text