Jacob Sussmilch

Normalization and Order Statistics

4 min read

Normalization and Order Statistics

Why normalization transforms work, grounded in the relationship between order statistics, CDFs, and power-law distributions.

The Core Insight

Sample two uniform random numbers in [0,1] and take their maximum. The result has CDF r². Separately, take one uniform random number and compute its square root. That also has CDF r². Both processes produce identical distributions.

This isn’t coincidence — it’s the probability integral transform. Applying a distribution’s own CDF to its samples always yields uniform data. For the max-of-2 distribution, the CDF is r², so squaring (the inverse) maps it back to uniform. That’s what normalization is: inverting the distortion.

Generalisation: The max of n uniform random variables has CDF rⁿ (Beta(n,1)). Raising to the nth power normalizes it. Taking the nth root of a uniform produces the same distribution.

Why Transforms Work

Every normalization follows the same logic:

  1. Some generative process creates a non-uniform CDF
  2. The correct transform is the one that inverts that CDF back toward uniformity (or normality)
  3. When you know the process, you can derive the transform theoretically
  4. When you don’t, you search (Box-Cox) or go non-parametric (ranks)

Common Transforms and Their Generative Processes

Log Transform

Distortion: Multiplicative processes. Values grow by percentages rather than fixed amounts.

Why: If X grows by repeated multiplication (X × 1.05 × 0.97 × 1.12…), the central limit theorem applies to log(X), since log turns multiplication into addition. So log(X) is approximately normal, meaning X itself is lognormal — heavily right-skewed.

Fix: log(X) converts multiplicative structure back to additive structure.

Used for: Income, asset prices, gene expression, city populations, earthquake magnitudes (the Richter scale is a log transform).

Square Root Transform

Distortion: Count data. Poisson-like distributions where variance scales with the mean.

Why: For a Poisson distribution, Var(X) = μ. High-count categories have more noise than low-count ones. By the delta method, Var(√X) ≈ 1/4 — roughly constant regardless of μ.

Fix: √X stabilizes variance, making it independent of the mean.

Used for: Species counts in ecology, word frequencies in NLP, pixel photon counts in astronomy.

Power Transform (Box-Cox)

Distortion: Unknown skew, suspected power-law CDF structure.

Why: Box-Cox searches over the family X^λ for the λ that makes the result closest to normal. λ = 0.5 gives square root, λ → 0 gives log, λ = 1 is identity. It’s an empirical search over the same family of transforms that undo order-statistic-like distortions.

Fix: Find optimal λ, apply X^λ.

Used for: When you need normality but lack theoretical grounds for a specific transform.

Rank / Quantile Normalization

Distortion: Unknown or pathological — heavy tails, multiple modes, no simple function fits.

Why: Replace each value with its percentile rank. This applies the empirical CDF directly, which by the probability integral transform always yields uniform. Then map to any target distribution via inverse CDF.

Fix: Rank → percentile → inverse target CDF.

Tradeoff: Always works, but destroys magnitude information. The gap between 100 and 101 looks the same as between 1 and 50 if they’re adjacent in rank.

Used for: Gene expression microarrays, ML feature preprocessing, anything where robustness matters more than interpretability.

Logit Transform

Distortion: Bounded proportions in (0,1). Values near boundaries get compressed — moving from 0.98 to 0.99 halves the remaining gap.

Why: Proportions arise from binomial-like processes. The natural additive scale is log-odds: log(p/(1-p)). This maps (0,1) → (-∞, +∞), spreading out the compressed tails.

Fix: logit(p) = log(p/(1-p)).

Used for: Logistic regression (this is literally why it’s called that), click-through rates, mortality rates, test pass rates.

Inverse Hyperbolic Sine (arcsinh)

Distortion: Same as log, but data contains zeros or negative values.

Why: Log breaks at zero. arcsinh(x) ≈ log(2x) for large x, so it behaves like log in the tail but is defined everywhere including zero and negatives.

Fix: arcsinh(X) instead of log(X).

Used for: Financial data with gains and losses, survey data with true zeros (income studies where some respondents earn nothing).

Discovery Process

When encountering unfamiliar data:

  1. Plot the empirical CDF — for each value r, what fraction of samples are ≤ r? The shape of this curve is the distortion
  2. If the CDF looks like a power law (rⁿ) → use a power transform
  3. If the data spans orders of magnitude → try log
  4. If it’s count data with mean-dependent variance → try square root
  5. If it’s bounded in (0,1) → try logit
  6. If nothing fits → use rank normalization (the non-parametric nuclear option)
  7. If unsure → Box-Cox will search for you

Sources

  • Matt Parker (Stand-up Maths) — maximum of random variables equals nth root distribution
  • Probability integral transform (foundational result in statistics)
  • Box-Cox (1964) — power transform family

Graph View