When Numbers Mislead Us

Arthur Charpentier · 2025

Read When Numbers Mislead Us by Arthur Charpentier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03628.txt.

Introduction

Arthur Charpentier warns that numbers can mislead when hidden factors are left out, in this 2025 essay. Data never speak for themselves: every set rests on choices about what to measure, how, when, and on whom. Those choices carry quiet assumptions about what counts.

What goes unmeasured can matter as much as what is captured. A missing key variable can twist the links seen between the rest, faking correlations or hiding real ones. In some cases, adding the hidden factor flips the verdict of a surface analysis. That mechanism, he notes, is exactly what drives Simpson's paradox, where group trends reverse when groups are pooled.

Catalog record

CREATORArthur Charpentier
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-03628

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Numbers Mislead Us Arthur Charpentier Abstract Believing that there is a single, objective way to describe phenomena using numbers is to forget that data does not “speak” for itself. Collecting data involves making choices: what to measure, how, when, on whom, etc. This implies implicit (or even ideological) assumptions about what counts as a measurable fact. And in any data analysis, what is not measured can be as important as what is observed. When an influential variable is overlooked—whether ignored, neglected, or simply unknown—the apparent relationships between other variables can become misleading. This is known as “omitted variable bias”: a hidden effect distorts comparisons and can make a correlation appear where there is none, or mask a real one. Sometimes, introducing this “forgotten” variable can even completely reverse the conclusions that would have been drawn from a naive reading of the data. 1 Rankings Hospitals, Doctors, Schools, etc. In an era of widespread rankings—from the best hospitals to the best doctors, including schools, as frequently found in the press—performance indicators are playing an increasingly important role in public decisions and social perceptions. But as Goodhart’s law states, “when a measure becomes a target, it ceases to be a good measure.” When a hospital seeks to climb the rankings, it may do so not by actually improving the quality of care, but by acting on the levers that influence the indicators: admitting less serious patients, avoiding complex cases, or optimizing its discharge statistics. The risk is then to produce a biased image of efficiency, disconnected from medical reality. Consider the following (entirely fictional) example, from Table 1 , with two hospitals of the same size, very loosely based on a famous example of kidney stone treatment (with real data) that appeared in the British Medical Journal in the 1980s. Based, on a quick comparison of mortality rates in two hospitals, Hospital A has better outcomes within both subgroups (healthy and non-healthy patients), its overall death rate is higher than Hospital B’s. hospital A hospital B total death ratio total death ratio non-healthy 60 36 60% < 20 14 70% healthy 20 4 20% < 60 18 30% total 80 40 50% > 80 32 40% Table 1: Mortality rates by hospital and patient condition (source: author). 2 Not Really a Paradox… In 1951, British statistician Edward Simpson published an article on contingency table analysis, in Simpson 1951 . In it, he described a phenomenon whereby the relationship between two variables changes when a third hidden variable is taken into account. He did not consider this a paradox, but rather a statistical effect worth noting. It was not until the 1970s that this behavior was rediscovered, named in his honor, and elevated to the status of a “paradox” because it defies our intuition. Despite its name, Simpson’s paradox is not a paradox in the mathematical sense, as it is not based on any logical contradiction or error in formal reasoning. Rather, it is a statistical surprise linked to our intuition about averages. In reality, this phenomenon occurs when ratios calculated in subgroups (e.g., recovery rates in two categories of patients) are compared and then aggregated without taking into account the structure of the data (group size, distribution of cases). The misinterpretation stems from the fact that the overall ratio is not the average of the ratios: in general, percentages from groups of different sizes cannot be added together and used to derive a meaningful average without weighting them. In the previous example, for Hospital A, the total death rate is not the average of the two rates (otherwise we would have obtained 40%, the average of 60% and 20%) but 50%. And then we have a small counterintuitive property about fractions, namely that we can have {\color[rgb]{0.72,0,0}\frac{36}{60}}<{\color[rgb]{0.06,0.46,1}\frac{14}{20}}\text{ and }{\color[rgb]{0.72,0,0}\frac{4}{20}}<{\color[rgb]{0.06,0.46,1}\frac{18}{60}} but, at the same time, {\color[rgb]{0.72,0,0}\frac{36+4}{60+20}}={\color[rgb]{0.72,0,0}\frac{40}{80}}>{\color[rgb]{0.06,0.46,1}\frac{14+18}{20+60}}={\color[rgb]{0.06,0.46,1}\frac{32}{80}} More formally, in Table 2 , we have an abstract representation of the paradox (inspired from Table 1 ). Each group (A and B) is split into two subgroups. Even though group B shows worse ratios in both subgroups, its overall rate is better—due to differences in the distribution of totals across groups. A graphical depiction of the paradox: line slopes represent subgroup ratios. In both cases, group B’s ratios are higher, but once aggregated, group A’s overall slope becomes steeper—demonstrating the paradox in action. group A group B total deaths ratio total deaths ratio subgroup 0 a b \frac{\displaystyle b}{\displaystyle a} < A B \frac{\displaystyle B}{\displaystyle A} subgroup 1 c d \frac{\displaystyle d}{\displaystyle c} < C D \frac{\displaystyle D}{\displaystyle C} total a+c b+d \frac{\displaystyle b+d}{\displaystyle a+c} > A+C B+D \frac{\displaystyle B+D}{\displaystyle A+C} Table 2: Formal version of Table 1 , with counts, by hospital and patient condition. xxx {\color[rgb]{0.72,0,0}\frac{b}{a}}<{\color[rgb]{0.06,0.46,1}\frac{B}{A}}\text{ and }{\color[rgb]{0.72,0,0}\frac{d}{c}}<{\color[rgb]{0.06,0.46,1}\frac{D}{C}} but, at the same time, {\color[rgb]{0.72,0,0}\frac{b+d}{a+c}}>{\color[rgb]{0.06,0.46,1}\frac{B+D}{A+C}} These inequalities can be seen in Figure 1 . The segment connecting (20,4) from the origin (0,0), in red, has a slope of 4/20, which is below the segment connecting (60,18), because its slope of 18/60 is greater. The same applies to the red segment connecting (60,36), which is below the blue segment connecting (20,14) because 14/20>36/60 . However, when we aggregate, we obtain the points on the right, (80,32) in blue and (80,40) in red, and the blue slope is lower than the red slope. x (total) y (deaths) (a,b) (c,d) (a+c,b+d) (A,B) (C,D) (A+C,B+D) Figure 1: Geometric visualization of Simpson’s paradox, based on Figure 2 . This discrepancy between the overall ratio and the partial ratios fuels many well-known “paradoxes” in mathematics concerning averages: harmonic mean vs. arithmetic mean, average speed, or even the “average of averages.” Simpson’s paradox is a sophisticated version of the latter case: it reminds us that a misinterpreted average can not only be misleading, but can completely reverse a trend. It is therefore not a mathematical oddity, but an invitation to caution in the interpretation of aggregates. 3 Berkeley Admissions and Gender Bias: A Real Case A famous case highlighted this pitfall: in 1973, the University of California, Berkeley was accused of discriminating against women in graduate admissions. The overall figures were unequivocal: 44% of men were admitted, compared to only 35% of women—an alarming difference that was enough to trigger a lawsuit. This significant difference raised suspicions of gender discrimination in the selection process. However, upon closer examination of the data, something surprising emerged: in most departments, the admission rate for women was equal to or higher than that for men. The data provided by Peter Bickel, Eugene Hammel, Eugene, and William O’Connell shows the same kind of phenomenon observed in hospitals, in Bickel et al. 1975 or Bickel et al. 1977 . Men Women Total Admitted Rate Total Admitted Rate A 825 512 62.1% < 108 89 82.4% B 560 353 60.0% < 25 17 68.0% C 325 120 36.9% > 593 202 34.0% D 417 138 33.1% < 375 131 34.9% E 191 53 27.7% > 393 94 23.9% F 373 22 5.9% < 341 24 7.0% Total 2691 1198 44.5% > 1835 557 30.4% Table 3: UC Berkeley graduate admissions, from Bickel et al. 1975 (real admission data). Table 3 reveals that in four of the six departments, women had a higher admission rate than men. Nevertheless,

Read the complete text