Hypothesis Testing in Six Sigma

Q: What does it mean to fail to reject the null hypothesis?

It means the sample did not provide enough evidence of a difference at the chosen α. It does not prove the null is true. With a small sample, a real and important difference can easily go undetected, which is a Type II error.

Q: Should I use a one-tailed or two-tailed test?

Use a one-tailed test only when the question is truly one-directional and you decided that before seeing the data, for example “is the new supplier’s defect rate lower?” Otherwise use two-tailed. Switching to one-tailed after the fact to get a smaller p-value is not valid.

Q: Is statistical significance the same as practical significance?

No. With very large samples, tiny differences become significant. Always report the size of the difference, ideally with a confidence interval, and judge it against cost, customer requirements, and the specification.

Q: Where does hypothesis testing fit in DMAIC?

Mainly in Analyze, to confirm which X’s really affect the Y, and in Improve, to show that a change made a real difference. Belt exams also test it heavily; try the Analyze-phase practice test .

Q: When should I use a z test instead of a t test?

Only when the population standard deviation is known from long, stable history, or for proportions. In almost all project work σ is estimated from the sample, so use a t test. With large samples, the two give nearly identical answers.

Hypothesis testing is a method for deciding whether a difference you see in sample data is large enough that chance alone is an unlikely explanation. You state a null hypothesis (H₀: no difference or no effect) and an alternative (H₁), pick a significance level α (often 0.05), compute a test statistic, and reject H₀ when the p-value is below α.

In Six Sigma it is the main Analyze-phase tool for confirming that a suspected X really moves the Y.

An original worked example. The test statistic falls beyond the critical value, so the difference in mean cycle time between the two lines is statistically significant at α = 0.05.

What are the steps in a hypothesis test?

State the practical question and the hypotheses. Example: H₀: μ_A = μ_B; H₁: μ_A ≠ μ_B.
Choose α and the test. α is the risk of a false alarm you will accept. The test depends on the data type and the question (see the table below).
Check assumptions and collect data. Random or representative samples, independence, and, for t-tests and ANOVA, roughly normal data or large enough samples. Plan the sample size for the difference that matters in practice.
Compute the statistic and p-value. Software does this; on exams you may use a t-table or z-table to find the critical value.
Decide and translate back. Reject or fail to reject H₀, then say what it means for the process, and whether the size of the difference matters, not only whether it is significant.

Errors, power, and the p-value

	H₀ is actually true	H₀ is actually false
Reject H₀	Type I error (α, producer’s risk): a false alarm	Correct decision; probability = power (1 − β)
Fail to reject H₀	Correct decision; probability = 1 − α	Type II error (β, consumer’s risk): a missed effect

The p-value is the probability of seeing a result at least as extreme as yours if H₀ were true. It is not the probability that H₀ is true. A large p-value does not prove “no difference”; it means the data are not strong enough to show one, often because the sample was small. Power rises with larger samples, larger true effects, less noise, and a larger α.

Which hypothesis test should you use?

Question	Data	Parametric test	Nonparametric option
One mean vs a target	Continuous	One-sample t (z if σ known)	One-sample sign or Wilcoxon
Two independent means	Continuous	Two-sample t	Mann-Whitney
Before vs after on the same units	Continuous, paired	Paired t	Wilcoxon signed-rank
Three or more means	Continuous	One-way ANOVA	Kruskal-Wallis or Mood’s median
Two or more variances	Continuous	F-test or Bartlett	Levene
One or two proportions	Pass/fail counts	One- or two-proportion test	Fisher’s exact (small counts)
Association between categories	Counts in a table	Chi-square test of independence	—
Several responses at once	Multiple continuous Y’s	MANOVA	—

For the parametric branch in more detail, see choosing the right parametric test. The sampling distributions behind these tests are explained separately. Critical values for t, chi-square, and F are on the Six Sigma tables page.

Worked example: cycle time on two lines

A team suspects line A is slower. A random sample of 30 orders from each line gives line A a mean of 12.4 minutes (s = 2.1) and line B a mean of 11.2 minutes (s = 1.9). With α = 0.05 and a two-sided Welch two-sample t-test:

Standard error = √(2.1²/30 + 1.9²/30) = 0.517
t = (12.4 − 11.2) ÷ 0.517 = 2.32, with about 57 degrees of freedom
Critical value ≈ 2.00, and the two-sided p ≈ 0.024

Since 0.024 < 0.05, reject H₀: the 1.2-minute difference is unlikely to be chance. The next questions are practical ones. Is 1.2 minutes worth acting on, and what on line A causes it? A fishbone diagram and a scatterplot with regression on candidate X’s are natural next steps, and a designed experiment can confirm the best settings.

One- and two-sample tests: formulas and examples

Every test below follows the same pattern: test statistic = (estimate − hypothesized value) / standard error, compared with a reference distribution. The table is the quick reference, and each card that follows has a computed example. The data are illustrative, and the p-values are two-sided and computed in scipy.

Test	Use when	Statistic	Reference distribution
1-sample z	One mean vs a target, σ known from long history	z = (x̄ − μ₀) / (σ/√n)	Standard normal
1-sample t	One mean vs a target, σ estimated by s	t = (x̄ − μ₀) / (s/√n)	t, n − 1 df
2-sample z	Two independent means, both σ known	z = (x̄₁ − x̄₂) / √(σ₁²/n₁ + σ₂²/n₂)	Standard normal
2-sample t	Two independent means, σ’s estimated	t = (x̄₁ − x̄₂) / SE (pooled or Welch)	t, df as below
Paired t	Two measurements on the same units	t = d̄ / (s_d/√n)	t, n − 1 df
1-proportion z	A defect rate vs a target	z = (p̂ − p₀) / √(p₀(1 − p₀)/n)	Standard normal (or exact binomial)
2-proportion z	Two defect rates	z = (p̂₁ − p̂₂) / √(p̄(1 − p̄)(1/n₁ + 1/n₂))	Standard normal
χ² for one variance	One standard deviation vs a target	χ² = (n − 1)s² / σ₀²	χ², n − 1 df
F for two variances	Two standard deviations	F = s₁² / s₂² (larger on top)	F, (n₁ − 1, n₂ − 1) df

1-sample z test

Use it only when the population σ is genuinely known, for example from a long, stable control chart, and you test a new mean against a target. A filler has a long-run σ = 1.5 g. After maintenance, 16 packs average 500.8 g against a 500 g target. SE = 1.5/√16 = 0.375, z = 0.8/0.375 = 2.13, and p = 0.033. The mean has shifted (95% CI 500.06 to 501.54 g), so re-center the filler. If σ is not truly known, use the t test.

1-sample t test

This is the usual choice, because σ is estimated from the sample. 12 invoices average 4.18 days to approve (s = 0.31) against a 4.0-day standard. SE = 0.31/√12 = 0.0895, and t = 0.18/0.0895 = 2.011 on 11 df. The critical value is ±2.201, so p = 0.069: not significant at α = 0.05, and the 95% CI (3.983 to 4.377) includes 4.0. With n = 12, a 0.18-day shift is within noise. The sample-size page shows how many invoices would be needed to detect it. The t test assumes roughly normal data, but it is robust for moderate n. For small, skewed samples, use the 1-sample Wilcoxon or sign test.

2-sample z test for two means

Use it when both population σ’s are known, or with very large samples, where s ≈ σ. Two shifts, 50 units each, have known σ’s of 2.0 and 2.4 and means of 15.2 and 14.3. SE = √(2.0²/50 + 2.4²/50) = 0.4418, z = 0.9/0.4418 = 2.037, and p = 0.042.

2-sample t test, and whether to pool variances

This compares two independent means when the σ’s are estimated. Supplier 1 has n = 10, x̄ = 23.1 and s = 2.4. Supplier 2 has n = 12, x̄ = 20.9 and s = 2.1. There are three versions:

Version	Standard error	df	t	p
Pooled (equal variances assumed)	s_p√(1/n₁ + 1/n₂) = 0.9591	n₁ + n₂ − 2 = 20	2.294	0.0328
Welch (unequal variances)	√(s₁²/n₁ + s₂²/n₂) = 0.9713	Welch-Satterthwaite = 18.1	2.265	0.0360
Conservative hand method	same as Welch	smaller of n₁ − 1 and n₂ − 1 = 9	2.265	0.0498

The pooled standard deviation is s_p = √[((n₁ − 1)s₁² + (n₂ − 1)s₂²)/(n₁ + n₂ − 2)] = 2.240. All three versions agree here (p ≈ 0.03 to 0.05) because the variances are similar (ratio 1.31).

Why pool, and when not to: if the two populations really share one σ, pooling combines both samples into a better estimate of it, which gives more degrees of freedom and slightly more power. If the σ’s differ, especially with unequal sample sizes, the pooled test gives the wrong false-alarm rate. Welch’s test is nearly as powerful when variances are equal and stays valid when they are not, so most practitioners and software now default to Welch. Pool only with good reason to believe the variances are equal, not because an F test failed to reject.

Paired t test

Use it when each unit is measured twice, such as before and after, or by two methods. Ten clerks timed a task (minutes) on the old and new software:

Clerk	1	2	3	4	5	6	7	8	9	10
Old	14.2	11.8	16.5	13.1	12.4	15.8	10.9	13.7	14.9	12.2
New	13.1	11.5	15.2	12.6	12.6	14.3	10.1	12.9	13.6	11.8
d = new − old	-1.1	-0.3	-1.3	-0.5	+0.2	-1.5	-0.8	-0.8	-1.3	-0.4

d̄ = -0.78, s_d = 0.535, and t = -0.78/(0.535/√10) = -4.61 on 9 df, so p = 0.0013. The new software saves about 0.8 minutes per task. Analyzing the same numbers wrongly as two independent samples gives p = 0.30, because the large clerk-to-clerk differences swamp the effect. Pairing removes them, which is why a paired design is so powerful.

1-proportion test

A supplier claims 2% defective. In 400 incoming parts, 14 are defective (p̂ = 0.035). SE under H₀ = √(0.02 × 0.98 / 400) = 0.0070, z = (0.035 − 0.020)/0.0070 = 2.143, and p = 0.032. The normal approximation needs np₀ and n(1 − p₀) of at least about 5 (here np₀ = 8). The exact binomial test gives p = 0.046, still significant but less so. Prefer the exact test when counts are small.

2-proportion test

Before a fix, 30 of 500 orders had errors (6.0%). After the fix, 14 of 500 did (2.8%). The pooled proportion is p̄ = 44/1000 = 0.044, SE = √(0.044 × 0.956 × (1/500 + 1/500)) = 0.0130, z = 0.032/0.0130 = 2.467, and p = 0.0136. The error rate fell by 3.2 points (95% CI 0.7 to 5.7 points). This is identical to a chi-square test on the 2 × 2 table (p = 0.0136). For three or more groups, use a chi-square test of independence on the full table (critical values are in the chi-square table).

Chi-square test for one variance

The spec calls for σ ≤ 0.50 mm. A sample of 20 gives s = 0.62. χ² = (n − 1)s²/σ₀² = 19 × 0.62² / 0.50² = 29.21 on 19 df. The 2.5% and 97.5% points are 8.91 and 32.85, so the two-sided p = 0.125 and the one-sided “σ is larger” p = 0.063. Neither is significant at 0.05. The 95% CI for σ is 0.472 to 0.906 mm, which is wide: variances need far more data than means to pin down. This test is very sensitive to non-normality, so check normality first or use a bootstrap interval.

F test for two variances

Machine 1 has s = 0.62 (n = 20) and machine 2 has s = 0.41 (n = 16). F = 0.62² / 0.41² = 2.287 on (19, 15) df. The upper 2.5% point is 2.773, so the two-sided p = 0.109: not significant, despite a 50% difference in s. Like the χ² test, the F test assumes normal data. For non-normal data, or more than two groups, use Levene’s or Bartlett’s test.

Frequently asked questions

What does it mean to fail to reject the null hypothesis?