ReVISit 2: A Full Experiment Life Cycle User Study Framework

\authororcid Zach Cutler0000-0002-2656-3413, \authororcid Jack Wilburn0000-0002-7672-0798, \authororcid Hilson Shrestha0009-0006-6603-6606, \authororcid Yiren Ding0000-0001-8983-9117, \authororcid Brian Bollen0000-0002-9557-1783, \authororcid Khandaker Abrar Nadib0009-0006-7940-501X, \authororcid Ti · 2025

Read ReVISit 2: A Full Experiment Life Cycle User Study Framework by \authororcid Zach Cutler0000-0002-2656-3413, \authororcid Jack Wilburn0000-0002-7672-0798, \authororcid Hilson Shrestha0009-0006-6603-6606, \authororcid Yiren Ding0000-0001-8983-9117, \authororcid Brian Bollen0000-0002-9557-1783, \authororcid Khandaker Abrar Nadib0009-0006-7940-501X, \authororcid Ti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03876.txt.

Introduction

Zach Cutler and co-authors present a full framework for online user studies of data displays in 2025, using the byline given in the text head. Such studies are common in visualization research. Running them well is still a heavy load. Past tools cover only parts of the life cycle, or make repeat runs hard.

Their system, called reVISit 2, spans design, pilot, data gathering, analysis, and sharing. It can replay each person's actions for review. A support community is part of the design. The system has served in published quality studies. The team shows this through repeat runs of past experiments. They also reflect on design through interviews and a tech review.

Catalog record

CREATOR\authororcid Zach Cutler0000-0002-2656-3413, \authororcid Jack Wilburn0000-0002-7672-0798, \authororcid Hilson Shrestha0009-0006-6603-6606, \authororcid Yiren Ding0000-0001-8983-9117, \authororcid Brian Bollen0000-0002-9557-1783, \authororcid Khandaker Abrar Nadib0009-0006-7940-501X, \authororcid Ti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE93 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-03876

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ReVISit 2: A Full Experiment Life Cycle User Study Framework \authororcid Zach Cutler0000-0002-2656-3413, \authororcid Jack Wilburn0000-0002-7672-0798, \authororcid Hilson Shrestha0009-0006-6603-6606, \authororcid Yiren Ding0000-0001-8983-9117, \authororcid Brian Bollen0000-0002-9557-1783, \authororcid Khandaker Abrar Nadib0009-0006-7940-501X, \authororcid Tingying He0000-0002-9670-5587, \authororcid Andrew McNutt0000-0001-8255-4258, \authororcid Lane Harrison0000-0003-3029-2799, and \authororcid Alexander Lex0000-0001-6930-5468 Abstract Online user studies of visualizations, visual encodings, and interaction techniques are ubiquitous in visualization research. Yet, designing, conducting, and analyzing studies effectively is still a major burden. Although various packages support such user studies, most solutions address only facets of the experiment life cycle, make reproducibility difficult, or do not cater to nuanced study designs or interactions. We introduce reVISit 2, a software framework that supports visualization researchers at all stages of designing and conducting browser-based user studies. ReVISit supports researchers in the design, debug & pilot, data collection, analysis, and dissemination experiment phases by providing both technical affordances (such as replay of participant interactions) and sociotechnical aids (such as a mindfully maintained community of support). It is a proven system that can be (and has been) used in publication-quality studies—which we demonstrate through a series of experimental replications. We reflect on the design of the system via interviews and an analysis of its technical dimensions. Through this work, we seek to elevate the ease with which studies are conducted, improve the reproducibility of studies within our community, and support the construction of advanced interactive studies. \onlineid 0 \vgtccategory Research \vgtcpapertype system \authorfooter Zach Cutler, Jack Wilburn, Brian Bollen, Khandaker Abrar Nadib, Tingying He, Andrew McNutt, and Alexander Lex are with the University of Utah. Hilson Shrestha, Yiren Ding, and Lane Harrison are with WPI. \teaser ReVISit 2 supports each stage of the online user study life cycle. In addition to this linear path, the experiment life cycle is populated by internal loops between stages. For instance, issues revealed in the piloting of an experiment can lead back to the design phase to address those issues, and the process of disseminating an experiment can beget ideas for new experiments or replications. Introduction keywords User studies, crowdsourcing, visualization experiments. Experimental research is a mainstay method to infer causal relationships in visualization, HCI, and related areas [ 19 ] . Experiments in visualization (often called user studies) range from perceptual studies [ 8 , 65 ] , to studies of visualization techniques [ 20 , 35 ] , studies of interaction techniques [ 5 , 38 ] , and studies of full visualization systems [ 52 , 44 ] . Experimental approaches are also used outside of quantitative, controlled research, such as in eliciting expert feedback on systems [ 53 ] or designs [ 37 , 26 , 42 ] , or understanding insight formation [ 11 , 40 ] . While these studies historically have been conducted in the lab [ 8 , 72 , 15 ] , they are now predominantly run online through crowd work platforms [ 27 , 52 , 63 ] , which support rapid participant recruiting and study execution. Most desktop-focused visualization tools are designed to be accessed through the browser, suggesting that web applications are an effective means to deliver study stimuli. These trends suggest that experimental research in the future will be predominantly conducted through browsers and asynchronously (as opposed to in a lab). Despite the ubiquity of this form of inquiry, conducting these studies remains difficult for a variety of reasons. For instance, designing effective stimuli presents nuanced challenges (such as effective instrumentation of interactions) that are time-consuming to implement for experienced developers, let alone early-career researchers. Similarly, the design of experiments offers a maze of complex decisions (such as choice of factors, randomization, sampling strategies, and partitioning of conditions) that, when navigated poorly, can invalidate entire experiments. Compounding these design challenges is that making everything work as intended and ensuring that data is collected correctly can be tedious and error-prone. To help with these challenges, researchers in visualization, HCI, psychology, and other fields draw on software frameworks to support their experiments. Whereas commercial tools, such as Qualtrics [ 29 ] , excel at survey design, experimental design, and deployment, their closed-source and commercial nature hamper reproducibility, and they do not meet the more specialized needs of visualization experiments—such as sophisticated tracking of complex interactions. A variety of academic-led systems have been developed to help run studies; however, our discussions with stakeholders from the visualization community revealed that most are used only by individual research groups, use outdated technology, are difficult to maintain, and result in brittle deployments. Other academic systems are specific to certain visualization techniques (e.g., graphs [ 54 ] ), or cover only a small slice of the experimental life cycle (such as factorial experiment design [ 50 ] ). To address these issues, we introduce reVISit 2, a software framework designed to support visualization researchers (and others) across the full life cycle of an experiment, including its design, debugging, and deployment. ReVISit is a proven system capable of supporting complex study designs at a publication-quality level. It has been used in various studies [ 42 , 11 , 10 , 46 ] . For the former we have made a concerted effort to reach out to groups who may benefit from this tool via tutorials at universities and conferences, intentionally expanding and extending our documentation, and simplifying the startup cost for experiments by making common experiment components (such as VLAT [ 41 ] , Mini-VLAT [ 56 ] , BeauVis [ 25 ] , etc.) available as libraries. In addition to quality of life upgrades (such as improved UIs, form elements, data download, testing), on the technical side we make a variety of novel contributions to experimentation platforms. Central among these is an enriched domain-specific language (DSL) for specifying experiments, as shown in Fig. 1 . This language includes complex to implement features, such as Latin square participant distribution and randomized attention checks. Complementing this expansion, we also provide high-level Python bindings, reVISitPY , which offers Altair [ 68 ] style declarative specification and enables even more complex study designs (e.g., factorial designs) with little code. A natural venue for use of this library is Jupyter notebooks, where we offer an experiment prototyping pipeline wherein developers cannot only design their experiment, but see and debug the experiment within the notebook, and even access collected data. This pipeline enables researchers to design and test their study, as well as prototype analyses from within the notebook. In contrast to other platforms, reVISit provides sophisticated tools for debugging and piloting, such as a study browser, participant view simulator, and data previews, as well as tools to manage data collection in ongoing studies. We also make strides in simplifying the specification of visualization stimuli by making Vega visualizations first-class citizens, allowing us to provide automated provenance tracking across user interactions with Vega visualizations, which supports both low-level interaction analysis and fine-grained replay ( Fig. 4 ). We evaluate this collection of contributions via two strategies. First, we demonstrate its expressiveness through a collection of three study replications, where each study demonstrates different capabilities of reVISit (such as dynamic sequencing, capturing speech, and analyzing provenance). Then we reflect on the design choices made in the system via three interviews with reVISit

Read the complete text