Attribute MSA: Attribute Agreement Analysis and Kappa

Attribute MSA, usually run as an attribute agreement analysis, checks whether a pass/fail, category or rating judgment can be trusted. Several appraisers judge the same parts, some of which are known to be bad, more than once and without knowing which parts they are.

You then measure how often each appraiser agrees with themselves (repeatability), with each other (reproducibility) and with the known standard (accuracy). Results are summarized as percent agreement, Cohen’s or Fleiss’ kappa, and the AIAG effectiveness, miss rate and false-alarm rate. It is the attribute-data counterpart of the variable Gage R&R study.

How is attribute MSA different from variable MSA?

	Variable (Gage R&R)	Attribute (agreement analysis)
Data	Measurements on a continuous scale	Go/no-go, pass/fail, defect type, 1–5 ratings
Typical uses	Calipers, CMMs, scales, timers	Visual inspection, go/no-go gages, call-quality scoring, claim or invoice coding
Main outputs	%Study variation, %tolerance, ndc	% agreement, kappa, effectiveness, P(miss), P(false alarm)
Parts needed	About 10	At least 20, and more is better, because each judgment carries little information

Attribute data are harder to qualify than variable data. A yes/no judgment hides how close a part was to the limit, so you need more parts and repeats, and the parts must be chosen deliberately. If the judgment can be turned into a measurement or a structured score, such as a 0–10 completeness checklist instead of “OK/not OK”, do it: the data then behave more like continuous data and a regular Gage R&R or an intraclass correlation can be used.

How do you design an attribute agreement study?

Parts: at least 20 (30 to 50 is better), with roughly a quarter to half bad or defective, and include borderline parts just inside and just outside each limit. A study of only obvious parts will always pass.
Reference standard: decide each part’s true status with a better method (a variable gage, an expert panel or the customer standard) and record it, but hide it from appraisers.
Appraisers and trials: two or three appraisers who normally do the job, each judging every part two or three times.
Blind and random: code the parts, shuffle the order for every trial and leave time between trials so appraisers cannot remember earlier answers.
Same conditions as production: the same lighting, time per part, tools and work instructions.

Seeding the sort

The simplest check on a live inspection is to slip known bad (and known good) items into the normal flow anonymously. The team running the study can identify them, but the inspectors cannot. If seeded bad items get through, or good ones get rejected, the inspection needs work. Seeding is quick and realistic, but it gives only a yes/no answer. The structured study below tells you who and where.

Worked example: visual inspection of molded caps

Twenty caps (14 good, 6 with short-shot or flash defects) were judged twice each by three inspectors in random order. G = accept, B = reject. Six parts, three good and three bad, are borderline (marked *). Data are illustrative (seeded simulation), and every figure below is computed from this table.

Part	Standard	A trial 1	A trial 2	B trial 1	B trial 2	C trial 1	C trial 2
1	Good	G	B	G	G	G	G
2	Good	G	G	G	G	G	G
3*	Good	G	G	G	G	G	G
4	Good	G	G	G	G	G	G
5	Good	G	G	G	G	G	B
6*	Good	G	G	G	B	G	G
7	Good	G	G	G	G	G	G
8	Good	G	G	G	G	G	G
9	Good	G	G	G	G	G	G
10*	Good	B	G	G	G	B	B
11	Good	G	G	G	G	G	G
12	Good	G	G	G	G	G	G
13	Good	G	G	G	G	G	G
14	Good	G	G	G	G	G	G
15	Bad	B	B	B	B	B	B
16*	Bad	B	B	G	B	B	G
17	Bad	B	B	B	B	B	B
18*	Bad	B	B	B	G	G	B
19	Bad	B	B	B	B	B	B
20*	Bad	B	B	G	B	G	B

Agreement and kappa

Appraiser	Within (same answer both trials)	Within kappa	vs standard (both trials correct)	Kappa vs standard
Appraiser A	90%	0.78	90%	0.89
Appraiser B	80%	0.47	80%	0.75
Appraiser C	80%	0.52	75%	0.64

All three appraisers agreed with each other on every trial for 65% of parts, and all six judgments matched the standard for 65% of parts. The between-appraiser Fleiss’ kappa across all six judgments is 0.62.

The kappa calculation

Percent agreement flatters a system when one category dominates: if 70% of parts are good, an inspector who accepts everything agrees with the standard 70% of the time. Kappa removes the agreement expected by chance:

κ = (P_observed − P_chance) / (1 − P_chance)

For Appraiser A against the standard, over 40 judgments: P_observed = 38/40 = 0.95. A accepted 26 of 40 (0.65) and the standard says good for 28 of 40 (0.70), so P_chance = 0.65 × 0.70 + 0.35 × 0.30 = 0.56. κ = (0.95 − 0.56) / (1 − 0.56) = 0.89.

Cohen’s kappa compares two sets of ratings (for example one appraiser vs the standard). Fleiss’ kappa extends this to many raters or trials. Kappa is 1 for perfect agreement, 0 for chance-level agreement, and negative when agreement is worse than chance. Common guidance (AIAG MSA): above 0.75 is good to excellent, below 0.40 is poor. Many Six Sigma teams require 0.9 for critical characteristics and accept about 0.7 for investigation work. For ordered ratings, such as 1–5 scores, use weighted kappa or Kendall’s coefficient, which give partial credit for near misses.

Effectiveness, miss rate and false alarms

The AIAG effectiveness method turns the study into the risks that matter to the customer and the plant:

Effectiveness E = parts judged correctly on every trial / total parts
P(miss) = bad-part judgments accepted / all bad-part judgments (customer risk; β)
P(false alarm) = good-part judgments rejected / all good-part judgments (producer risk; α)

Decision	Effectiveness	P(miss)	P(false alarm)
Acceptable	≥ 0.90	≤ 0.02	≤ 0.05
Marginal, may need improvement	0.80 to 0.90	0.02 to 0.05	0.05 to 0.10
Unacceptable	< 0.80	> 0.05	> 0.10

Appraiser	Effectiveness	P(miss)	P(false alarm)
Appraiser A	0.90 (Acceptable)	0/12 = 0.000 (Acceptable)	2/28 = 0.071 (Marginal)
Appraiser B	0.80 (Marginal)	3/12 = 0.250 (Unacceptable)	1/28 = 0.036 (Acceptable)
Appraiser C	0.75 (Unacceptable)	3/12 = 0.250 (Unacceptable)	3/28 = 0.107 (Unacceptable)

Reading the result: the inspection is not acceptable as a system. Appraiser A is close, with no misses and two false alarms (parts 1 and 10, one judgment each). Appraisers B and C each passed 3 of 12 bad-part judgments, which is a customer escape risk. Note the uncertainty: with only 12 bad-part judgments, a 3/12 miss rate has a 95% confidence interval of about 0.05 to 0.57. That is why serious studies use more bad parts. Because the misses fall on borderline parts, the next steps are boundary samples (limit samples) at the station, a clearer defect definition, and training for B and C. Then repeat the study.

Signal detection: sizing the gray zone of a go/no-go gage

When parts have measurable reference values, as with a go/no-go plug gage, the signal detection approach estimates how wide the measurement system’s zone of confusion is. Measure each part with an accurate variable gage, run it through the attribute gage several times by two or more appraisers, sort by reference value, and mark three regions near each limit:

Original illustration with illustrative data: 19 parts, 6 judgments each (3 appraisers × 2 trials).

Region I: outside spec and rejected by everyone. Region III: inside spec and accepted by everyone.
Region II: the gray zone around each limit where judgments are mixed. Its width d is the distance between the closest Region I and Region III parts.
d(USL) = 10.062 − 10.044 = 0.018 mm and d(LSL) = 9.958 − 9.944 = 0.014 mm, so the average d̄ = 0.016 mm.

d̄ estimates the measurement system’s spread, comparable to the 6σ (or, in older practice, 5.15σ) width of a variable Gage R&R. Against the tolerance: %GRR ≈ d̄ / (USL − LSL) = 0.016 / 0.100 = 16%, in the usual 10–30% “marginal” band. Whether that is acceptable depends on capability. A process running at C_p 2 rarely produces parts in the gray zone, while one at C_p 1 produces many, so judge it with engineering knowledge, not the 10/30 rule alone. If the process is so capable that out-of-spec parts had to be made for the study, compare d̄ with the historical process spread (6σ), not with the spread of the study parts.

Ordinal ratings: intraclass correlation (ICC)

When appraisers give scores, such as 1–10 completeness ratings on purchase orders, treat the scores as numbers and use the intraclass correlation. Three senior buyers each rated the same 10 purchase orders (illustrative data). A two-way ANOVA (orders × buyers) gives MS orders = 12.09, MS buyers = 0.53 and MS error = 0.348. Buyer averages were 5.7, 6.1, 5.7.

ICC(2,1) = (MS_orders − MS_error) / [MS_orders + (k − 1)MS_error + k(MS_buyers − MS_error)/n] = 0.91

This form (two-way random effects, absolute agreement, single rater) answers the usual MSA question: if any one buyer rates an order, how reliable is that score? The average of all three buyers is more reliable, ICC(2,3) = 0.97. Use the single-rater form when one person will rate in production, and the average form only if production scores are averaged across raters. Guidance: above 0.9 is excellent, 0.75 to 0.9 good, 0.5 to 0.75 moderate, below 0.5 poor (Koo and Li, 2016). Many Six Sigma texts use 0.9 or higher as excellent and below 0.7 as inadequate. Here the buyers are consistent enough to use the score as a project Y.

Improving an attribute measurement system

Write operational definitions with photos or limit samples of acceptable and rejectable conditions.
Fix the environment: lighting, magnification, viewing angle and time allowed per part.
Replace judgment with a variable gage or a checklist score where possible.
Use error-proofing (poka-yoke) or automated vision for critical defects, rather than relying on inspection.
Retest after changes, and re-qualify new inspectors with the same study kit.

Frequently asked questions

How many parts and appraisers does an attribute agreement analysis need?

Use at least 20 parts, preferably 30 to 50, with a good share of bad and borderline parts, and 2 to 3 appraisers judging each part 2 to 3 times. Small studies give very wide uncertainty on miss and false-alarm rates.

What is a good kappa value?

Under AIAG guidance, above 0.75 indicates good to excellent agreement and below 0.40 is poor. Many teams require 0.9 or higher for critical or safety characteristics.

Is percent agreement enough?

No. When most parts are good, high agreement can occur by chance. Kappa corrects for this, and the miss rate shows the customer risk directly.

Can I use Gage R&R for attribute data?

Not the standard variable study. Use attribute agreement analysis, the signal detection approach if reference values exist, or convert the judgment to a score and use an intraclass correlation or a variable Gage R&R.

Last updated: November 11, 2026 · Published by Lean Six Sigma Training Guide (sixsigmatrainingfree.com)