Hi all,
I have an experiment where 5 raters assessed the quality of 24 web sites. (each
rater rated each site once). I want to come up with a measure of reliability of
the ratings for the web sites ie to what extent does each rater give the same
(or similar) rating to each web site. My idea was to fit a random effects model
using lme and from that, calculate the intraclass correlation as a measure of
rater reliability. So I analysed it using a two-factor, unreplicated random
effects model, with sitenumber and rater as crossed random effects. The call to
lme was:
fit <- lme(score ~ 1,
random=list(level=pdBlocked(list(pdIdent(~sitenumber-1),
pdIdent(~rater-1)))),data=scores)
which makes sense, and I then calculated the intraclass correlation using the
within sitenumber (ie among raters) variance, the within rater variance and the
residual variance (using VarCorr(fit)):
rho = var_among/(var_among + var_within + var_resid)
Now, the raters can be divided into two groups. I would like to include group as
a fixed factor, to test whether there is a difference in scores between groups
(I know the number of raters is too small. This is a pilot study and I am really
just trying to get the analysis right in case we want to expand the study).
raters therefore are nested within group. My initial guess was:
fit2 <- lme(score ~ group,
random=list(level=pdBlocked(list(pdIdent(~sitenumber-1),
pdIdent(~rater-1)))),data=scores)
but this fit implies (I think) that sitenumber AND rater are nested within group
(sitenumber is not nested within group), and results in the wrong denominator
degrees of freedom for the test of group (118). I then tried a slightly
different approach:
fit3 <- lme(score ~ group, random=list(level=pdIdent(~sitenumber-1),
rater=pdIdent(~group-1)),data=scores)
which gives the correct denominator degrees of freedom for the group effect (3).
But it implies that rater is nested within level. Given that level is a dummy
factor with only one level, I don't think this matters and I can
"pretend" that it is really a crossed design. Is that correct? Or am I
way off beam here? Can anyone suggest a more appropriate lme call? The more I
think about it the more confused I get.
Thanks in advance,
Simon.
Simon Blomberg, PhD
Depression & Anxiety Consumer Research Unit
Centre for Mental Health Research
Australian National University
http://www.anu.edu.au/cmhr/
Simon.Blomberg at anu.edu.au +61 (2) 6125 3379
