In a test of inter-rater agreement, I planned to report a kappa and lower+upper confidence bounds. However, I found 100% perfect agreement between the two raters. Is it possible to produce a confidence interval from this? In addition, is it appropriate to report a Kappa of 1?

1 Answer 1

If your point and interval estimates are are $1.00$ then you can and should report that. However, it's important to interpret these results in the context of the design of the reliability experiment (i.e., are these estimates good?). For example, how large was the sample of objects being rated and how large was the sample of raters? Were these samples large and representative enough of the populations of interest to be trustworthy? In my experience, it is pretty rare to see perfect agreement on a large sample, so I'm wondering if this result may be revealing underlying issues in the experimental design.