FronTalk : Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback

Xueqing Wu, Zihan Xue, Da Yin, Shuyan Zhou, Kai-Wei Chang, Nanyun Peng, Yeming Wen · 2026

Read FronTalk : Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback by Xueqing Wu, Zihan Xue, Da Yin, Shuyan Zhou, Kai-Wei Chang, Nanyun Peng, Yeming Wen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04203.txt.

Introduction

Xueqing Wu and six co-authors introduce FronTalk, a benchmark for building website front ends through conversation with mixed feedback. In real front-end work, sketches, mockups, and marked screenshots carry design intent as much as words do. Multi-turn code tools, though, have barely been tested with such visual input.

FronTalk holds one hundred multi-turn dialogues drawn from real sites across news, finance, art, and more, each turn giving the same intent in text and in visual form. A web agent explores the built site to score both function and user experience. Across twenty models, two faults stand out: models forget and overwrite features built earlier, and visual feedback is hard to read, above all for open models. Their AceCoder baseline critiques each past instruction and cuts forgetting nearly to zero while lifting scores.

Catalog record

CREATORXueqing Wu, Zihan Xue, Da Yin, Shuyan Zhou, Kai-Wei Chang, Nanyun Peng, Yeming Wen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE108 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-04203

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FronTalk : Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback Xueqing Wu, Zihan Xue, Da Yin, Shuyan Zhou, Kai-Wei Chang, Nanyun Peng, Yeming Wen Abstract We present FronTalk , a benchmark for front-end code generation that pioneers the study of a unique interaction dynamic: conversational code generation with multi-modal feedback . In front-end development, visual artifacts such as sketches, mockups and annotated screenshots are essential for conveying design intent, yet their role in multi-turn code generation remains largely unexplored. To address this gap, we focus on the front-end development task and curate FronTalk , a collection of 100 multi-turn dialogues derived from real-world websites across diverse domains such as news, finance, and art. Each turn features both a textual instruction and an equivalent visual instruction, each representing the same user intent. To comprehensively evaluate model performance, we propose a novel agent-based evaluation framework leveraging a web agent to simulate users and explore the website, and thus measuring both functional correctness and user experience. Evaluation of 20 models reveals two key challenges that are under-explored systematically in the literature: (1) a significant forgetting issue where models overwrite previously implemented features, resulting in task failures, and (2) a persistent challenge in interpreting visual feedback , especially for open-source vision-language models (VLMs). We propose a strong baseline to tackle the forgetting issue with AceCoder , a method that critiques the implementation of every past instruction using an autonomous web agent. This approach significantly reduces forgetting to nearly zero and improves the performance by up to 9.3% (56.0% → \rightarrow 65.3%). Overall, we aim to provide a solid foundation for future research in front-end development and the general interaction dynamics of multi-turn, multi-modal code generation. Code and data are released at https://github.com/shirley-wu/frontalk † † affiliation: Meta Superintelligence Labs † † affiliation: University of California, Los Angeles † † affiliation: Duke University † † contribution: Work done at MSL † † date: August 10, 2026 † † correspondence: Xueqing Wu at [email protected] † † Code: https://github.com/shirley-wu/frontalk Figure 1 : Focusing on the front-end development task, FronTalk benchmark explores a novel setting with multi-turn interactions and multi-modal user feedback. 1 Introduction The success of Large Language Models (LLMs) in code generation has popularized the application of AI-assisted coding involving multi-turn interaction , as seen in trends like “vibe coding” and a surge of commercial coding assistants. Reflecting this trend, recent benchmarking efforts have moved beyond single-turn code generation ( Austin et al. 2021 ; Chen et al. 2021 ) to better measure how effectively LLMs respond to feedback across multiple turns ( Wang et al. ; Laban et al. 2025 ; Han et al. ) . However, these benchmarks predominantly focus on algorithmic or API-driven tasks, where multi-modal feedback is less central, and thus limit their evaluation to text-only interactions. This results in under-exploration of domains such as data visualization, application design, and front-end development where multi-modal feedback is a critical communication channel. For example, in front-end development, sketching a UI layout or annotating a screenshot is a natural and efficient method for communicating design changes. Therefore, this unique dynamic of multi-turn, multi-modal coding remains a significantly underexplored challenge. To advance this frontier, we introduce FronTalk , a benchmark for front-end development that integrates multi-turn code generation with multi-modal feedback. Our benchmark begins with a diverse and realistic collection of user intents , derived from real-world websites in the C4 dataset ( Raffel et al. 2020 ) and spanning diverse domains such as e-commerce platforms, financial service sites, and digital art portfolios. To capture realistic conversational dynamics, we employ an LLM-based user simulator that generates context-aware instructions conditioned on prior dialogue. Crucially, to enable our simulated users to generate visual feedback, we equip them with a suite of drawing tools to communicate intents through sketching and annotating. The resulted FronTalk dataset comprises 1,000 conversational turns across 100 dialogues, paired with 3,676 manually refined test cases, offering a robust foundation for evaluating multi-modal, multi-turn coding systems. For evaluation, we propose an agent-based evaluation framework to evaluate both instruction following and user experience . For instruction following, we measure pass rate against human-annotated test cases using an automated web agent to interact with the website and verify task completion. This agent is uniquely equipped with image manipulation tools (e.g., cropping, image comparison), enhancing model’s visual perception capabilities and improving alignment by 1.7% with human annotators on design tasks. For the nuanced dimension of user experience, we employ a pairwise comparison protocol. In this setup, first-time users simulated by LLMs interact with each website to learn the interface and attempt self-proposed tasks. A secondary LLM then compares the resulting trajectories and judges which interface is more usable , with criteria grounded in learning efficiency and satisfaction in task completion. Our evaluation protocol achieves a Cohen’s Kappa of 0.63 for pass rate and 0.67 for usability, indicating significant alignment. We evaluated a range of open- and closed-source models on our FronTalk benchmark, including seven text-only LLMs evaluated on textual feedback and seven VLMs evaluated on both textual and visual feedback. Our experimental results reveal several key findings: 1. Open-source vs. proprietary gap : As shown in Table 2 , proprietary models maintain a substantial lead over open-source models. This disparity is particularly pronounced with visual feedback, where the performance gap is 24.1%, compared to the 12.5% gap with textual feedback. Since FronTalk is the first to evaluate multi-turn multi-modal code generation, we attribute this gap to the superior generalization capabilities of proprietary models, an area where contemporary open-source VLMs still fall short. 2. Forgetting issue: The multi-turn setting introduces a significant forgetting issue, causing previously implemented functions or design elements to be contradicted or overwritten by later code. This issue consistently occurs across all leading models and results in a performance degradation of up to 46%, as measured by the forgetting rate metric reported in Table 2 . 3. Challenges of visual feedback : Visual feedback proves consistently more difficult than textual feedback, resulting in significantly lower pass rate. Figure 4 shows a detailed breakdown of failure modes, with most common ones being (1) the literal replication of layout sketches without implementing the underlying functionality, and (2) overlooking textual annotations in densely annotated images. 4. Key factors affecting usability : The determinants of usability evolve with model capability, as discussed in § 3.2 . For weaker models, poor usability is primarily linked to broken or non-functional features ; However, for stronger models that achieve high pass rates, usability is usually impacted by design flaws , such as inefficient navigation or a lack of interactive feedback. This highlights that achieving high usability requires implementing implicit, common-sense design principles that are often omitted from user instructions. To address the critical forgetting issue, we propose AceCoder as a simple yet strong baseline using a gent c ritique to e nhance user instructions. For each turn in the conversation, AceCoder begins with the LLM generating an initial code solution based on user instruction. Then, a web agent, powered by the same LLM, autonomously explore the rendered website to verify the implementation of both the current and past instructions. Coding failures or oversights identified by the agent are appended to the user instruction,

Read the complete text