Normality and Data Transformation
Normality means the data follow a normal (bell-shaped) distribution closely enough for methods that assume it, such as standard capability indices, individuals control charts, and small-sample t-tests. Check it with a histogram, a normal probability plot, and a test such as Anderson-Darling.
If the data are not normal, first look for a reason, such as mixed sources, a natural boundary, or outliers. Then either fit the right distribution, transform the data (log, square root, Box-Cox), or use a nonparametric method.
Normal probability plot
A normal probability plot puts each ordered data value against the value a normal distribution would predict for that rank. If the data are normal, the points fall close to a straight line. Common patterns are easy to read:
- Curve bending upward at the right: right skew, typical of times, sizes, and counts with a floor of zero.
- S-shape: tails that are heavier or lighter than normal.
- Two straight segments or a gap: a mixture of two populations, such as two machines or shifts. Stratify rather than transform.
- Flat steps: coarse measurement resolution (see MSA).
- One or two points far off the line: possible outliers or special causes. Investigate them; do not just delete them.
How do you test data for normality?
| Test | Notes |
|---|---|
| Anderson-Darling | Default in many quality packages; gives extra weight to the tails, which is where capability and ppm estimates live |
| Shapiro-Wilk | Powerful for small and moderate samples |
| Kolmogorov-Smirnov (Lilliefors) | Less powerful; mostly of historical interest for normality |
| Ryan-Joiner | Correlation-based, similar to Shapiro-Wilk |
All of them test H₀: the data are normal. A small p-value (below 0.05) is evidence of non-normality. Two cautions apply. With very large samples, trivial departures become significant. With very small samples, almost nothing is rejected. Always look at the plot as well, and check which distribution makes physical sense for the characteristic.
Common causes of non-normal data
- Natural boundaries: times, flatness, runout, and impurity levels cannot go below zero, so they skew right.
- Mixtures: data from several machines, operators, or suppliers pooled together.
- Special causes and outliers: shifts or one-off events. Check a control chart first.
- Truncation: parts outside specification were sorted out before you measured.
- Measurement resolution: too few distinct values.
- A genuinely non-normal process that is better described by a lognormal, Weibull, or exponential model.
Which transformation should you use?
| Transformation | Formula | Typical use |
|---|---|---|
| Natural log | y = ln(x) | Right-skewed positive data, such as times and concentrations (lognormal) |
| Square root | y = √x | Counts, Poisson-like data |
| Reciprocal | y = 1/x | Strong right skew, such as rates |
| Box-Cox | y = (xλ − 1)/λ; ln(x) when λ = 0 | Finds the power λ that makes positive data most normal |
| Johnson | A family of transformations | Data that Box-Cox cannot fix, including negative values |
Box-Cox power transformation
Box-Cox searches over λ, usually between −5 and 5, for the value that makes the transformed data closest to normal, and software reports the best λ and a confidence interval. In practice you round to a convenient value: λ = 1 means no change, 0.5 a square root, 0 the natural log, and −1 the reciprocal. Box-Cox needs all values to be positive. Transform the specification limits the same way when you compute capability, and back-transform results before reporting them.
Worked example: repair times
Fifty repair times range from 5.9 to 106.9 minutes, with a mean of 33.6, a median of 30.6, and a skewness of 2.08. The service target is an upper limit of 90 minutes, and 2 of the 50 jobs went over it.
- Raw data: Anderson-Darling A² = 2.01 (p < 0.01); Shapiro-Wilk p < 0.0001. Not normal.
- Box-Cox estimates λ ≈ 0.13, close to 0, which points to a log transformation.
- ln(x): A² = 0.45 (p > 0.15); Shapiro-Wilk p = 0.32; skewness -0.31. No evidence against normality, so the data are well described as lognormal.
| Capability method | Ppu | Expected over 90 min |
|---|---|---|
| Assume normal (wrong here) | 0.98 | about 1,618 ppm |
| Lognormal, via ln(x) and ln(USL) | 0.70 | about 17,899 ppm (1.8%) |
| Percentile method (median to the 99.865th percentile) | 0.52 | same lognormal fit |
Treating the data as normal would understate the overrun rate about tenfold. The lognormal fit predicts about 1.8% over 90 minutes, which is consistent with the 2 in 50 actually observed.
Capability for non-normal data
There are three sound approaches. The first is to transform the data and the specification limits, then compute ordinary indices on the transformed scale. The second is to fit a distribution (lognormal, Weibull, and so on) and compute the expected fraction out of specification directly. The third is the percentile (Clements) method: replace the ±3σ distance with the distance from the median to the 0.135th and 99.865th percentiles of the fitted distribution, so Ppu = (USL − median)/(P99.865 − median). The methods give different index values for the same fit, so report which one you used. The predicted ppm, rather than the index, is usually the clearest result for the customer. See process capability for the normal case.
Frequently asked questions
Do I always need normal data for Six Sigma tools?
No. X̄ charts and t-tests on means are fairly robust because of the central limit theorem, and many tools such as Pareto charts, fishbone diagrams, and attribute charts do not assume normality. Normality matters most for capability and ppm estimates, individuals charts, and small-sample tests.
Should I remove outliers to make the data normal?
Only if you find an assignable cause, such as a recording error or a special event, and you document it. Removing inconvenient points to pass a normality test hides real process behavior.
What if no transformation works?
Check for mixtures and stratify first. Otherwise, fit another distribution, use the percentile method for capability, or switch to nonparametric tests such as Mann-Whitney or Kruskal-Wallis.
Can I transform data with zeros or negative values?
Not directly with logs or Box-Cox. You can add a constant first, but that changes the result and must be applied to the limits too. A Johnson transformation, or a distribution suited to the data, is usually cleaner.
Related guides
Last updated: · Published by Lean Six Sigma Training Guide (sixsigmatrainingfree.com)