Finding Non-Redundant Simpson’s Paradox from Multidimensional Data
Yi Yang, Jian Pei, Jun Yang, Jichun Xie · 2025
Read Finding Non-Redundant Simpson’s Paradox from Multidimensional Data by Yi Yang, Jian Pei, Jun Yang, Jichun Xie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00748.txt.
Description
Yi Yang, Jian Pei, Jun Yang, and Jichun Xie study Simpson's paradox, the effect where a trend seen in pooled data reverses when the data are split into subgroups. Hunting for such reversals in large multidimensional data turns up many copies of the same pattern, which waste time and hide what matters.
They define three kinds of redundancy between paradoxes and show the relation between them is an equivalence, so copies can be grouped and summarized. Their algorithms combine depth first table building with redundancy aware search. On real and synthetic data, over forty percent of paradoxes on some sets were redundant, run times fell by up to sixty percent, and the search scaled to millions of records.
Catalog record
| CREATOR | Yi Yang, Jian Pei, Jun Yang, Jichun Xie |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 296 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-00748