ANOVA and Non-Parametric Tests
ANOVA (analysis of variance) tests whether the means of three or more groups differ. It compares the variation between groups with the variation within them, and a large F ratio says at least one group mean is different. Non-parametric tests, such as Mann-Whitney, Kruskal-Wallis, Mood’s median, Wilcoxon and Friedman, answer the same questions without assuming normal data.
They work on ranks or medians, which makes them the safe choice for skewed, ordinal or outlier-prone Six Sigma data such as cycle times.
Which test should you use? A decision tree
Pick the test from three facts: the type of Y, whether the data (strictly, the residuals) are roughly normal, and how the samples are structured (one sample, two independent groups, paired, three or more groups, or blocked).
Parametric versus non-parametric tests
| Parametric | Non-parametric | |
|---|---|---|
| Assumes | A distribution (usually normal) with parameters such as the mean and standard deviation | Only that samples are random and independent; some tests assume similar shapes across groups |
| Compares | Means | Medians or whole distributions (via ranks) |
| Strength | More power when assumptions hold | Robust to skew and outliers; works on ordinal data (ratings, ranks) |
| Weakness | Misleading with heavy skew, outliers or small non-normal samples | Somewhat less power with truly normal data (Mann-Whitney is about 95% as efficient as the t-test) |
A practical rule: with moderate or large samples (about 15 to 20 or more per group), ANOVA and t-tests tolerate mild non-normality because of the central limit theorem. With small samples, strong skew, outliers or ordinal data, go non-parametric or transform first.
How does one-way ANOVA work?
One-way ANOVA has one factor (for example machine, supplier or shift) at k levels. Hypotheses: H₀: μ₁ = μ₂ = … = μₖ against H₁: at least one mean differs. Total variation splits into two parts:
- SS between = Σ nᵢ(x̄ᵢ − x̄)²: how far the group means sit from the grand mean (df = k − 1)
- SS within = ΣΣ(x − x̄ᵢ)²: the scatter inside each group (df = N − k)
- F = MS between / MS within, where each MS = SS / df. Under H₀, F follows an F distribution with (k − 1, N − k) degrees of freedom.
Assumptions: independent observations, roughly normal residuals and similar variances across groups. ANOVA is fairly robust to unequal variances when group sizes are equal. When they are not, use Welch’s ANOVA.
Worked example: fill weight from three filling heads
A filler has three heads. A Black Belt weighs 8 consecutive packs (target 500 g) from each head. The data are illustrative and were analyzed in scipy:
| Head | Weights (g) | Mean | Std dev |
|---|---|---|---|
| Head 1 | 499.4, 500.0, 502.2, 501.0, 498.2, 500.2, 499.5, 500.4 | 500.11 | 1.18 |
| Head 2 | 499.7, 501.9, 501.9, 503.5, 502.0, 502.2, 499.8, 504.3 | 501.91 | 1.59 |
| Head 3 | 497.7, 501.3, 499.6, 498.9, 499.2, 499.2, 500.5, 499.9 | 499.54 | 1.08 |
| Source | SS | df | MS | F | p |
|---|---|---|---|---|---|
| Between heads | 24.56 | 2 | 12.28 | 7.24 | 0.0041 |
| Within (error) | 35.64 | 21 | 1.697 | ||
| Total | 60.20 | 23 |
F = 12.28 / 1.697 = 7.24, above the 5% critical value F(2, 21) = 3.47, so p = 0.0041: at least one head fills differently. The heads explain η² = SS between / SS total = 0.41, or about 41% of the variation.
Assumption checks: residual normality, Shapiro-Wilk p = 0.29 (fine). Equal variances, Levene’s test p = 0.76 (fine). Observations were taken in time order within each head, so check a run chart for drift as well.
Testing equal variances: Levene and Bartlett
Before trusting ANOVA, or when variation itself is the problem you are solving, test H₀: σ₁² = σ₂² = … = σₖ².
- Bartlett’s test is the most powerful when data are normal, but it is very sensitive to non-normality.
- Levene’s test runs an ANOVA on the absolute deviations |x − center| of each value from its group center. The original version uses the group mean. The Brown-Forsythe version uses the group median and is the robust default in most software, including Minitab’s “Levene’s” result. It is the right choice when data are not normal.
- For exactly two normal groups, the F test compares s₁²/s₂².
In the fill example, Levene’s W = 0.28 (mean-centered, p = 0.76; median-centered p = 0.76), so there is no evidence the heads differ in spread. Head 2’s larger standard deviation is within chance.
After a significant F: Tukey comparisons
ANOVA says “at least one differs” but not which. Tukey’s HSD compares every pair while holding the family-wise error rate at 5%:
| Pair | Difference (g) | 95% simultaneous CI | Adjusted p |
|---|---|---|---|
| Head 2 − Head 1 | 1.80 | 0.16 to 3.44 | 0.030 |
| Head 2 − Head 3 | 2.38 | 0.73 to 4.02 | 0.004 |
| Head 1 − Head 3 | 0.57 | -1.07 to 2.22 | 0.657 |
Head 2 overfills by about 1.8 to 2.4 g compared with the other two, whose intervals overlap. The fix is to recalibrate Head 2, not to adjust the whole filler. Running three separate t-tests instead would inflate the false-alarm rate to about 1 − 0.95³ ≈ 14%.
Non-parametric tests at a glance
| Test | Question it answers | Parametric counterpart | Notes |
|---|---|---|---|
| 1-sample sign | Is the median equal to a target? | 1-sample t | Uses only the signs of x − target; very few assumptions, low power |
| 1-sample / paired Wilcoxon | Is the median (or median difference) equal to a target? | 1-sample t, paired t | Uses signs and ranks; assumes a symmetric distribution of differences |
| Mann-Whitney | Do two independent groups differ in location? | 2-sample t | Also called the Wilcoxon rank-sum test; assumes similar shapes to read it as a median test |
| Kruskal-Wallis | Do 3+ independent groups differ? | One-way ANOVA | Rank-based; sensitive to the whole distribution, robust to outliers |
| Mood’s median | Do 3+ groups have the same median? | One-way ANOVA | Counts values above and below the grand median; very robust to outliers, less power |
| Friedman | Do 3+ treatments differ when each block (operator, batch) gets every treatment? | Two-way ANOVA without replication (randomized block) | Ranks within each block, removing block-to-block differences |
| Levene (Brown-Forsythe) | Do groups have equal variances? | Bartlett, F test | Robust to non-normality |
Kruskal-Wallis: comparing three or more groups
A service desk compares ticket resolution times at three sites. Times are right-skewed: most tickets close quickly, but a few take a long time.
An ANOVA on these data gives p = 0.010, but its residuals fail normality (Shapiro-Wilk p = 0.0024), so the ANOVA p-value is not trustworthy. Kruskal-Wallis replaces the 30 values with their ranks (1 = fastest) and asks whether the average ranks differ more than chance allows:
H = [12 / (N(N + 1))] · Σ (Rᵢ² / nᵢ) − 3(N + 1), compared with χ² on k − 1 df
| Site | Median (h) | Rank sum Rᵢ | Mean rank |
|---|---|---|---|
| A | 7.45 | 147.0 | 14.70 |
| B | 5.75 | 105.5 | 10.55 |
| C | 11.45 | 212.5 | 21.25 |
H = 7.51 (with a small correction for one tied value), above χ²(2) = 5.99, so p = 0.023: the sites differ. Site C has the highest mean rank.
Mood’s median test
Mood’s median test asks a blunter question: does each group have the same share of values above the grand median? Here the grand median is 7.45 h:
| Site A | Site B | Site C | |
|---|---|---|---|
| Above 7.45 h | 5 | 2 | 8 |
| At or below | 5 | 8 | 2 |
A chi-square test on this 2 × 3 table gives χ² = 7.20, p = 0.027. This agrees with Kruskal-Wallis. Mood’s test ignores how far each value is from the median, so it is extremely resistant to outliers but needs more data for the same power. Prefer Kruskal-Wallis unless there are outliers you cannot explain.
Mann-Whitney: two groups, and which sites differ
Mann-Whitney ranks two samples together and computes U, the number of times a value from one group beats a value from the other. Pairwise on the sites:
| Pair | U | p (unadjusted) |
|---|---|---|
| A vs B | 70.0 | 0.140 |
| A vs C | 22.0 | 0.038 |
| B vs C | 20.5 | 0.028 |
Both comparisons with Site C are below 0.05, but three comparisons call for a Bonferroni-adjusted threshold of 0.05 / 3 ≈ 0.017, and neither clears it. The honest conclusion: the sites differ overall and Site C is the likely outlier, but confirming which pairs differ needs a larger sample or a Dunn post-hoc test on the Kruskal-Wallis ranks. Small, skewed samples often give this answer, and saying so is better than overclaiming.
Paired data: Wilcoxon signed-rank and sign test
Ten operators each did a changeover before and after a quick-change fixture (minutes; illustrative data):
| Operator | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Before | 51 | 27 | 47 | 42 | 49 | 44 | 51 | 42 | 39 | 45 |
| After | 49 | 23 | 43 | 41 | 48 | 40 | 47 | 34 | 41 | 41 |
| Difference | -2 | -4 | -4 | -1 | -1 | -4 | -4 | -8 | +2 | -4 |
| Rank of |d| | 3.5 | 7 | 7 | 1.5 | 1.5 | 7 | 7 | 10 | 3.5 | 7 |
Wilcoxon signed-rank test
Rank the absolute differences (ties share the average rank), then add the ranks of the positive and negative differences separately: W⁺ = 3.5 and W⁻ = 51.5. The test statistic is the smaller, W = 3.5. For n = 10, the two-sided 5% critical value is 8, and W ≤ 8 rejects H₀, so p ≈ 0.012. The fixture cuts the median changeover by about 4 minutes. The 1-sample Wilcoxon test is the same calculation with d = x − target.
Sign test
The sign test uses only the direction of each difference: 9 of 10 operators improved. Under H₀ (median difference 0) each sign is a coin flip, so p = P(9 or more of 10, either direction) = 0.021 from the binomial distribution. It reaches the same conclusion with fewer assumptions (no symmetry needed) but less power. It throws away the sizes of the differences, so it needs more data to detect the same effect. Zero differences are dropped before testing. The 1-sample sign test compares each value against a target median in the same way.
Friedman test: blocked designs
When every block, for example an operator, a batch or a day, receives every treatment, block differences can swamp the treatment effect. Friedman’s test ranks the treatments within each block. Here 8 operators each assembled a unit using three work-instruction methods (seconds; illustrative data):
| Method | Op 1 | Op 2 | Op 3 | Op 4 | Op 5 | Op 6 | Op 7 | Op 8 | Rank sum |
|---|---|---|---|---|---|---|---|---|---|
| Method 1 | 33.5 (3) | 32.1 (3) | 30.2 (2) | 19.9 (2) | 31.3 (2) | 23.8 (2) | 32.8 (2) | 32.1 (3) | 19 |
| Method 2 | 29.6 (1) | 28.6 (1) | 27.8 (1) | 19.6 (1) | 29.8 (1) | 19.5 (1) | 31.2 (1) | 31.1 (2) | 9 |
| Method 3 | 31.2 (2) | 31.9 (2) | 32.7 (3) | 20.6 (3) | 32.7 (3) | 26.7 (3) | 33.2 (3) | 31.0 (1) | 20 |
χ²ᶠ = [12 / (n·k(k + 1))] · ΣRⱼ² − 3n(k + 1) = [12 / (8·3·4)] · (19² + 9² + 20²) − 3·8·4 = 9.25
That is above χ²(2) = 5.99, so p = 0.0098: the methods differ. Method 2 was fastest for 7 of the 8 operators, even though operators themselves range from about 20 s to 33 s. That spread would hide the effect in an ordinary Kruskal-Wallis or one-way ANOVA. The parametric equivalent is a randomized-block (two-way, no replication) ANOVA.
Beyond one-way ANOVA
Two-way ANOVA studies two factors at once and their interaction, for example head × shift. With planned factor settings this becomes a designed experiment. MANOVA handles several Y’s together (see MANOVA). When the Y is pass/fail rather than continuous, use proportion tests, chi-square or logistic regression (see the decision tree above).
Frequently asked questions
Why not just run several t-tests instead of ANOVA?
Each t-test has a 5% false-alarm risk, so three pairwise tests together carry about a 14% chance of at least one false positive. ANOVA tests all groups at once at 5%, and a post-hoc method such as Tukey then finds which pairs differ while controlling that risk.
When should I use a non-parametric test?
Use one when samples are small and clearly non-normal, when outliers are real and cannot be removed, when the data are ordinal (ratings or ranks), or when the median matters more than the mean, as with cycle and resolution times.
Is Kruskal-Wallis a test of medians?
Only if the groups have similar shapes and spreads. In general it tests whether one group tends to produce larger values than another. Mood’s median test is a direct test of medians, at the cost of some power.
What does a significant Levene’s test mean for my ANOVA?
The groups have different variances. With equal group sizes, ANOVA is usually still acceptable. With unequal sizes, use Welch’s ANOVA or a non-parametric test, and treat the variance difference itself as a finding, because reducing variation is often the Six Sigma goal.
Related guides
Last updated: · Published by Lean Six Sigma Training Guide (sixsigmatrainingfree.com)