PLanet: Formalizing and Analyzing Assignment Procedures in the Design of Experiments
London Bielicke, Anna Zhang, Shruti Tyagi, Emery Berger, Adam Chlipala, Eunice Jun · 2025
Read PLanet: Formalizing and Analyzing Assignment Procedures in the Design of Experiments by London Bielicke, Anna Zhang, Shruti Tyagi, Emery Berger, Adam Chlipala, Eunice Jun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-09094.txt.
Introduction
London Bielicke and five colleagues build a programming language for designing experiments, in a 2025 paper. Every design quietly assumes how variables relate, and those assumptions decide which causal questions the data can answer, yet design tools rarely surface them.
Their system, PLanet, offers composable building blocks for assignment procedures, grounded in matrix algebra. Programs compile into constraint problems, and a static analysis reports which causal queries stay testable under stated assumptions. In an expressivity contest against rival design languages, PLanet covered the most ground. In think-aloud trials, researchers explored alternatives and spotted assumptions they would otherwise have missed. The authors aim to make experimental assumptions explicit by default.
Catalog record
| CREATOR | London Bielicke, Anna Zhang, Shruti Tyagi, Emery Berger, Adam Chlipala, Eunice Jun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-09094
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PLanet: Formalizing and Analyzing Assignment Procedures in the Design of Experiments London Bielicke, Anna Zhang, Shruti Tyagi, Emery Berger, Adam Chlipala, Eunice Jun Abstract . Experimental designs reflect assumptions about variable relationships that determine what causal queries researchers can answer through the experiment. Accounting for and communicating these assumptions is essential for drawing valid, generalizable conclusions from scientific experiments. Unfortunately, existing experimental design tools elide these details, expecting researchers to reason about design decisions and assumptions on their own. To surface assumptions and enable design exploration, we introduce a grammar of composable operators for constructing experimental assignment procedures grounded in matrix algebra. The PLanet DSL implements this grammar and compiles PLanet programs into constraint satisfaction problems over matrices. Together, PLanet’s composable grammar and matrix representation enable a static analysis to determine which causal queries are testable under different assumptions. In an expressivity evaluation, PLanet was the most expressive of existing DSLs. Critical reflections with the authors of these DSLs revealed that PLanet makes design choices explicit without requiring procedural specification. Think-aloud studies showed that PLanet facilitated design exploration and surfaced assumptions researchers may otherwise overlook. DOI: XXXXXXX.XXXXXXX Conference: UIST 2026; November 2–November 5, 2026; Detroit, Michigan ISBN: 978-1-4503-XXXX-X/2018/06 1. Introduction Experiments are a primary tool in the scientist’s toolbox for understanding the world. Experiments enable scientists to isolate and test causal relationships among variables of interest. Evidence from experiments can challenge assumptions, test hypotheses, and inform the development of new theories in a range of scientific disciplines ( Fisher, 1935 ) . For instance, in human-computer interaction (HCI), experiments are commonly used to evaluate the efficacy of new interfaces, understand user behavior, and explore the impact of design choices on user experience. Consider the following familiar scenario: A researcher wants to establish whether a new graphical interface leads to faster task completion than a text-based alternative. To answer this question, the researcher must design an experiment that isolates the effect of the interface from other factors that could influence completion time, such as task type, participant experience, and the order in which conditions are experienced. The researcher decides to use a Latin square because they have seen it used in a previously published study. They conduct the study and find that participants who used the graphical interface completed the task in less time, concluding that the graphical interface causes people to complete the task faster than the text-based interface. What the researcher did not realize is that they made the implicit assumption that the interface will have the same effect on completion times no matter the type of task. If the researcher includes an additional task variable, they must make additional considerations. For example, should they counterbalance both the task and interface conditions? Should participants experience all combinations of task and interface conditions? These subtle considerations reflect assumptions about whether the effect of the interface depends on task or the order in which each interface is administered. This example illustrates a fundamental yet often overlooked tension when designing experiments. Researchers face practical constraints, most often time to run an experiment or access to participants, that may require them to consider alternatives to an ideal fully randomized experiment. These alternatives embed assumptions about variable interactions and time-based effects. When these assumptions do not align with the researchers’ understanding of their domain, the resulting design cannot test the effects that might matter to them or their discipline. Yet, current experimental-design workflows require the researcher to be aware of a design’s implicit assumptions and manually ensure that their domain assumptions and causal queries align. As the number of variables and potential interactions grows, manually reasoning about which designs preserve which testable effects becomes intractable. For these reasons, researchers may default to familiar designs, falsely assuming that they serve their purposes. What is missing is a way to make design decisions, assumptions, and their connections explicit. Some existing tools for experimental design constrain researchers to predefined designs ( Eiselmayer et al., 2019 ) , while others offer fine-grained control but require low-level specification ( Tanaka, 2021 ; Musslick et al., 2022 ) . All these tools lack support for detecting which causal effects the experiment can test for. Tools that do check for threats to validity target large-scale online experiments ( Tosch et al., 2019 ; Bakshy et al., 2014 ) , where design choices differ from those of human-subjects research. A critical gap in the landscape of experimental-design tooling is enabling systematic consideration and comparison of alternative designs suitable for researchers’ goals and assumptions. We address this gap by developing a grammar for expressing assignment procedures in experimental design. Through an iterative design process informed by experimental-design theory and practice, we identified a set of primitive operations that correspond to matrix manipulations. A key technical insight is that each primitive operation translates into a logical constraint on an underlying matrix representation. We instantiate this grammar in the PLanet domain-specific language (DSL) and implement a static analysis to detect when a causal effect is testable. We assess the expressivity of the PLanet DSL compared to edibble and Touchstone 2, two existing DSLs for experimental design. We found that PLanet was the most expressive, covering 14 of 15 experiments. In addition, we engaged the authors of these DSLs in critical-reflection ( Satyanarayan et al., 2020 ) sessions to dig into how the design and analysis capabilities of PLanet could enhance scientific practice. We found that PLanet provides a language for researchers to articulate design choices without selecting pre-defined categories or requiring procedural details, which can facilitate scientific transparency and deepen understandings of design decisions. Finally, in order to observe in-situ how PLanet influences researchers’ reasoning about and exploration of experimental designs, we conducted think-aloud studies with six researchers 1 1 1 We also conducted three case studies with researchers using an earlier version of PLanet, which we report in supplemental material and the appendix. who specified PLanet programs using PLanet’s web interface. All participants reported prior exposure to designs expressible in PLanet, but they lacked precise language to describe them. Yet, after initial exploration with the tool, they could articulate how design choices reflected different assumptions and hypotheses. This paper makes the following contributions: • a grammar and DSL defining assignment procedures grounded in matrix algebra ( Section 3 , Section 4 ); • a static analysis over PLanet programs to detect differences in testable causal queries between experimental designs ( Section 5 ); • an evaluation demonstrating PLanet DSL’s comparative expressivity for a broad range of designs found in a sample of recent HCI publications ( Section 6 ); • critical reflections with experimental-design DSL developers, illuminating how PLanet DSL enhances scientific practice, communication, and understanding ( Section 7 ); and • empirical evidence from think-aloud studies with researchers demonstrating how PLanet facilitates exploration and reasoning about a variety of experimental designs ( Section 8 ). 2. Background and Related Work We situate our work in the context of experimental-design theory, tools for designing studies, and domain-specific languages (DSLs). We define terminology related to experimental design in Box 1 . 2.1. Experimental-Design Theory The goal of an experiment is to estimate the causal effect of an intervention on an outcome. Determining valid causal effects