Finding Non-Redundant Simpson’s Paradox from Multidimensional Data

Yi Yang, Jian Pei, Jun Yang, Jichun Xie · 2025

Read Finding Non-Redundant Simpson’s Paradox from Multidimensional Data by Yi Yang, Jian Pei, Jun Yang, Jichun Xie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00748.txt.

Description

Yi Yang, Jian Pei, Jun Yang, and Jichun Xie study Simpson's paradox, the effect where a trend seen in pooled data reverses when the data are split into subgroups. Hunting for such reversals in large multidimensional data turns up many copies of the same pattern, which waste time and hide what matters.

They define three kinds of redundancy between paradoxes and show the relation between them is an equivalence, so copies can be grouped and summarized. Their algorithms combine depth first table building with redundancy aware search. On real and synthetic data, over forty percent of paradoxes on some sets were redundant, run times fell by up to sixty percent, and the search scaled to millions of records.

Catalog record

CREATORYi Yang, Jian Pei, Jun Yang, Jichun Xie
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE296 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-00748

Browse the full library · About the Tower of Babel