Hello,
I want to determine the inter-rater reliability of ratings made from a
random selection of 11 raters and 71 subjects using the irr package. Each
rater was randomly assigned to rater only a handful of subjects. So there
will be some cells with NAs for raters who did not rate a particular person.
I have formatted my dataset in accordance with the example for calculating
the ICC in the irr package. When I run the icc() function, the output
indicates that I have zero subjects:
Single Score Intraclass Correlation
Model: twoway
Type : consistency
Subjects = 0
Raters = 11
ICC(C,1) = NA
F-Test, H0: r0 = 0 ; H1: r0 > 0
F(-1,-10) = NA , p = NA
95%-Confidence Interval for ICC Population Values:
NA < ICC < NA
Warning messages:
1: In qf(1 - alpha/2, ns - 1, (ns - 1) * (nr - 1)) : NaNs produced
2: In qf(1 - alpha/2, (ns - 1) * (nr - 1), ns - 1) : NaNs produced
I am unsure what I am doing wrong. Below is reproducible code that can be
copied directly into R and should result in the same problem and warning
above. Any help would be appreciated.
Kind regards,
Chad
library(irr)
irr.smile.data <- structure(list(rater.1 = c(NA, NA, NA, NA, NA, NA, NA,
NA, NA,
NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, 4L, 6L, 5L, 4L), rater.2 = c(NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA,
3L, NA, NA, NA, NA, NA, NA,
NA, NA, 6L, 3L, 6L, NA, NA, NA, 2L,
3L, NA, 6L, NA, 6L, NA, 3L,
NA, NA, NA, NA, NA, NA, 2L, NA, 2L,
NA, NA, NA, NA, 6L, 3L, 1L,
2L, NA, 1L, NA, NA, NA, NA, NA, 2L,
2L, NA, 2L, NA, NA, NA, 2L,
NA, NA, NA, NA, NA, NA), rater.3 c(NA, NA,
NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, 5, NA, NA, NA, NA, 5.5, 5, NA, NA,
NA, NA, 6, 5, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
7), rater.4 = c(7, 6, 6, NA, 7, 4, 4, 2, 5.5, 6.5, 4, 5, NA,
NA, 6, 5, 6.5, 5, NA, NA, 6, 7, NA, 7, 6, 4, 2, 6, 3, 5, NA,
2, 3, 6, 5, 6, 6, NA, 6, 5, NA, 2, NA, NA, NA, 6, 5, 6, 2,
NA,
6, NA, 3, 6, NA, NA, NA, NA, NA, NA, 3, 2, NA, NA, NA, NA,
NA,
NA, NA, 7, NA), rater.5 = c(NA, NA, NA, NA, NA, NA, 5, NA,
NA,
NA, NA, 5, NA, NA, NA, 4, NA,
NA, NA, NA, NA, NA, 4, NA, NA,
NA, NA, NA, NA, NA, NA, 2, NA,
NA, NA, NA, NA, NA, NA, NA, 4,
2, 4.5, 4, 5, NA, NA, NA, NA,
NA, NA, 3, 4, NA, 5, 6, NA, NA,
NA, NA, NA, NA, 3, NA, 3, 5, 6,
5, NA, NA, NA), rater.6 = c(NA,
NA, NA, 4, 6, 5.5, 4, NA, NA, NA, NA, NA, 5, 6,
NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA,
NA, NA, NA, NA, NA), rater.7 = c(NA, NA, NA, NA,
NA, NA, NA,
NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, 5L, NA, NA, NA, NA, NA,
NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
), rater.8 = c(NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, 6L, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA), rater.9 = c(5, 6,
6, 2, 3, 2, 3, 1, 5, 5, 5, 5, 6, 7, 6, 5, 3, 2, 7,
5, 4, 5, 2,
6, 5, 5, 5, 7, 4, 6, 6, 2, 4, 6, 6, 5, 6, 6, 5, 5,
5, 3.5, 6,
2, 5, 7, 7, 5, 2, 2, 3, 3, 1, 7, 6, 5, 7, 5, 2, 5,
1.5, 3, 3,
5, 5, 5, 3, 6, 7, 5, 5), rater.10 = c(NA, NA, NA,
NA, NA, NA,
NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA,
6L, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 7L,
6L, NA, NA,
NA, NA, NA, NA, NA, NA, NA, 5L, 6L, NA, NA, NA, NA,
NA, NA, NA,
NA, 6L, 6L, NA, NA, NA, 6L, 5L, NA, NA, NA, NA, NA,
NA), rater.11
= c(NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, 6L, 5L, NA, NA, NA, NA, NA, NA, NA,
NA, NA, 2L, 5L, 2L, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA)), .Names =
c("rater.1",
"rater.2",
"rater.3", "rater.4", "rater.5",
"rater.6", "rater.7",
"rater.8",
"rater.9", "rater.10", "rater.11"), class =
"data.frame", row.names = c(NA,
-71L))
icc(irr.smile.data, model="twoway")
--
Chad M. Danyluck, MA
PhD Candidate, Psychology
University of Toronto
?There is nothing either good or bad but thinking makes it so.? - William
Shakespeare
[[alternative HTML version deleted]]