The Illusion of Intervention: Your LLM-Simulated Experiment is an Observational Study
Victoria Lin, Taedong Yun, Maja Matarić, John Canny, Arthur Gretton, Alexander D’Amour · 2026
Read The Illusion of Intervention: Your LLM-Simulated Experiment is an Observational Study by Victoria Lin, Taedong Yun, Maja Matarić, John Canny, Arthur Gretton, Alexander D’Amour (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20767.txt.
Description
Victoria Lin and five colleagues argue that tests with language-model-simulated users are observational studies, not true interventions. Models trained largely on past data can shift latent user traits when a treatment is applied, so the simulated group drifts across treatment groups. That drift can inflate or shrink observed treatment gaps through confounding or selection bias. The authors propose control outcomes, traits that should stay fixed under treatment, to diagnose drift, and study persona tweaks to ease it.
Catalog record
| CREATOR | Victoria Lin, Taedong Yun, Maja Matarić, John Canny, Arthur Gretton, Alexander D’Amour |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61953 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20767