Sycamore: Characterizing Synthetic Personas for Evaluating Genomics Visualization Retrieval
Huyen N. Nguyen \href https://orcid.org/0000-0001-6554-2327 Harvard Medical School, Astrid van den Brandt \href https://orcid.org/0000-0002-3676-1341 Harvard Medical School, Nils Gehlenborg \href https://orcid.org/0000-0003-0327-8297 Harvard Medical School · 2026
Read Sycamore: Characterizing Synthetic Personas for Evaluating Genomics Visualization Retrieval by Huyen N. Nguyen \href https://orcid.org/0000-0001-6554-2327 Harvard Medical School, Astrid van den Brandt \href https://orcid.org/0000-0002-3676-1341 Harvard Medical School, Nils Gehlenborg \href https://orcid.org/0000-0003-0327-8297 Harvard Medical School (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08630.txt.
Introduction
Huyen Nguyen, Astrid van den Brandt, and Nils Gehlenborg ask what fake users actually tell us when they test a real system. Judging niche tools like genomics visualization search is hard because true experts are scarce, so language model personas are tempting stand ins. Their probe, Sycamore, tests one real genomics search engine three ways: ungrounded personas, personas grounded in past interview material, and a published expert study.
Grounded personas echo documented user language and concerns, while ungrounded ones drift toward operational details real users never raised. Both fake groups miss a key expert preference for image based material. Evaluation research gains a clear account of where synthetic testers help and where they mislead.
Catalog record
| CREATOR | Huyen N. Nguyen \href https://orcid.org/0000-0001-6554-2327 Harvard Medical School, Astrid van den Brandt \href https://orcid.org/0000-0002-3676-1341 Harvard Medical School, Nils Gehlenborg \href https://orcid.org/0000-0003-0327-8297 Harvard Medical School |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08630
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sycamore: Characterizing Synthetic Personas for Evaluating Genomics Visualization Retrieval Huyen N. Nguyen \href https://orcid.org/0000-0001-6554-2327 Harvard Medical School, Astrid van den Brandt \href https://orcid.org/0000-0002-3676-1341 Harvard Medical School, Nils Gehlenborg \href https://orcid.org/0000-0003-0327-8297 Harvard Medical School Abstract Evaluating visualization systems in niche domains such as genomics is challenging due to scarcity of domain experts and difficulty recruiting a representative user base. While LLM-based synthetic personas are increasingly used to ease evaluation bottlenecks, they face well-founded skepticism. Rather than weighing synthetic personas as substitutes for real users, we ask a fundamental open question: when synthetic personas evaluate a real visualization system, what do they actually produce, and how does that output change when grounded in documented human contexts? We present Sycamore , an exploratory three-condition probe design using Geranium, a search engine for multimodal genomics visualization, as a case study. Sycamore evaluates Geranium using: (1) ungrounded synthetic personas from generic LLM priors; (2) grounded synthetic personas constrained by voice-of-customer artifacts from a prior interview study; and (3) a published baseline study of real domain experts. We observe that grounding shifts synthetic feedback toward the language and concerns of documented users, while ungrounded evaluators drift toward operational specifics that real participants did not raise; both synthetic conditions, however, converge on a find-and-adapt frame and miss the image-modality preference observed in the expert study. We discuss what these observations imply for where synthetic personas might fit alongside expert studies in domain-specific visualization evaluation. All supplemental materials are available at \url https://osf.io/kdfr3/. \onlineid 0 \vgtccategory Research \vgtcinsertpkg \teaser Diagram of the Sycamore system. Left: three‑condition evaluation on a common visualization retrieval system to address research questions \hyperref [rq1]RQ1 and \hyperref [rq2]RQ2. Right: details of components (1) grounded synthetic evaluators and (2) domain experts. keywords Synthetic persona, visualization retrieval, genomics visualization, visualization evaluation, multimodal system. 1 Introduction Visualization research has long employed user studies as the primary means for understanding tasks and evaluating systems. In niche application domains, however, this standard practice poses a challenging constraint: domain experts are scarce and participant recruitment is slow. More importantly, even successful recruitment captures only a fraction of the relevant user types. Genomics data visualization is an exemplary case. Tools such as Circos [ 10 ] , Gosling [ 11 ] , and IGV [ 21 ] serve users with diverse backgrounds and goals [ 23 ] , but evaluation studies of such tools typically reach only a small portion of that population. The user study of search engine Geranium [ 16 ] reached seven bioinformatics researchers; the Blended Interface user study [ 12 ] reached twelve domain experts in genomics; the van den Brandt et al. interview study [ 23 ] , after recruitment through multiple channels, reached twenty. In parallel, large language models (LLMs) capable of adopting personas have become widely available, and are increasingly used in design and evaluation settings inside and outside research. Yet their use in HCI has received well-founded skepticism [ 5 , 20 ] , with prior pushback from the UI/UX community, cautioning that synthetic users could only complement, but not replace, real users [ 7 ] . Rather than weighing synthetic personas as a substitute for real users, we approach a more basic question as still open: when synthetic personas are pointed at a real visualization system, what do they actually produce, and how does that output change with grounding from real human users? We present Sycamore ( Sy nthetic C h a racterization for Evaluating Geno m ics Visualizati o n Re trieval), an exploratory three-condition probe that examines this question through the following research questions: \phantomsection RQ1. What shifts in feedback from synthetic personas when they are grounded in characterizations of real domain users? \phantomsection RQ2. How does such feedback differ from real expert feedback on the same system? For both questions, we examine where the accounts overlap, where they diverge, and what each surfaces that the other does not. Sycamore applies three conditions to a single evaluation object, Geranium [ 16 ] , a multimodal search engine for genomics visualizations: (1) ungrounded synthetic personas from generic LLM priors; (2) grounded synthetic personas via PersonaCite [ 22 ] , constrained to retrieved evidence from voice-of-customer artifacts–here, the documented characterization of genomics experts in van den Brandt et al. [ 23 ] ; and (3) the published Geranium user study [ 16 ] . \hyperref [rq1]RQ1 contrasts the two synthetic conditions to isolate the effect of grounding; \hyperref [rq2]RQ2 reads both against the Geranium user study as a contextual reference. We find that grounding shifts synthetic feedback toward concerns of documented users, while ungrounded evaluators drift toward operational specifics real participants did not raise; both converge on a “runnable template” frame and miss the image-modality preference observed in the expert study. This paper offers the following contributions: (1) A three-condition probe design for examining synthetic-persona feedback in domain-specific visualization evaluation. (2) A grounded instantiation of the probe using a published persona characterization of genomics visualization experts. (3) An observational account of what each condition surfaces when applied to a genomics visualization retrieval system, and a discussion of where the three accounts overlap, diverge, and complement one another. 2 Related Work A persona is a fictional but realistic description of a typical or target user [ 3 ] , traditionally grounded in user research and represented as a static artifact. LLMs open new directions for persona creation and use: LLM-based personas can simulate users by drawing on the human behavior and language in their training data [ 2 ] , and can be engaged conversationally rather than read as static artifacts [ 8 , 22 ] . Building on this, generative agents extend LLM personas with memory and reflection, and can serve as believable proxies of individuals and communities [ 18 ] . Follow-up work shows that grounding agents in interview and survey data improves their accuracy [ 19 ] . Design and UX researchers are also actively exploring AI personas in design workflows [ 9 ] . On the other hand, genomics is a fast-evolving, data-intensive field, creating continuous demand for new visualization, management, and authoring tools. This demand is reflected in the wide design space of existing visualization authoring tools, ranging from GUI-based systems (e.g., genome browsers such as IGV [ 21 ] ) to visualization grammars (e.g., Gosling [ 11 ] ) and recommender and retrieval systems (e.g., GenoREC [ 17 ] and Geranium [ 16 ] ). This diversity of tools reflects a diverse user pool [ 23 ] , as also reported in related biomedical and data science studies [ 4 , 24 , 13 ] . Designing for such users requires ongoing user research, but recruitment in niche domains like genomics is slow and resource-intensive, creating a persistent bottleneck. This bottleneck motivates explorations into approaches that can scale user research, such as LLM-based experiments. 3 Methodology Terminology. Geranium refers to the visualization retrieval system under evaluation, while the Geranium user study [ 16 ] refers to the prior published user study, distinct from the cross-condition evaluation we present in Sycamore. A synthetic persona is the characterization being represented; a synthetic evaluator is an LLM instance performing the protocol under that characterization. Grounded personas are instantiated from documented