Thomas Meigen
2010-May-31 07:21 UTC
[R] Post-hoc tests for repeated measures in balanced experimental design
Hi,
I am performing experiments in the field of visual perception where
we often apply balanced designs. Within a group of normal subjects,
we vary different stimulus conditions (like contrast, luminance,
temporal frequency of stimulation) and derive some psychophysical or
electrophysiological results from our subjects. Often, the main
question is to test the effect of these parameters on the recorded
results.
When switching from Statview (old software on Mac computers, no
longer supported) to R I learned that the problem is translated to
a repeated measures ANOVA via
ano <- aov(d$Result~ d$Condition1*d$Condition2 + Error(d$Subject/(d
$Condition1*d$Condition2), data=d))
However, there are problems in performing post-hoc tests due to the
intraindividual correlation of the repeated measures data.
So I started intense online searches for a good solution in R, found
snippets of R-code here and there. One problem was that many
contributions offered help for the first step to replace the "aov"
procedure with a call to "lme" in the nlme package, but did not
perform post-hoc tests. Other contributions showed examples with only
one within-subjects factors. Finally, Achim Zeileis (thanks again!)
helped me with the following approach.
1. Use the nlme-Package and calculate a model with both within-
subjects effects "Condition1" and "Condition2"
library(nlme)
d.lme <- lme(Result~ Condition1*Condition2,data=d,random= ~1 | Subject)
2. Finally, the multcomp-Package can be used to perform post-hoc
tests for "Condition1" and "Condition2"
library(multcomp)
print(summary(glht(d.lme, linfct=mcp(Condition1="Tukey"))))
print(summary(glht(d.lme, linfct=mcp(Condition2="Tukey"))))
My problems and questions are
1) When applying this solution to my collection of data I found
several cases where the standard repeated-measures ANOVA showed a
highly significant effect for both factors, e. g.
ano <- aov(d$wPatternPulseSNR~ d$Bedingung*d$Felder + Error(d$VPerson/
(d$Bedingung*d$Felder), data=d))
Error: d$VPerson
Df Sum Sq Mean Sq F value Pr(>F)
Residuals 11 458.22 41.66
Error: d$VPerson:d$Bedingung
Df Sum Sq Mean Sq F value Pr(>F)
d$Bedingung 4 364.58 91.14 7.4429 0.0001140 ***
Residuals 44 538.81 12.25
Error: d$VPerson:d$Felder
Df Sum Sq Mean Sq F value Pr(>F)
d$Felder 5 464.17 92.83 8.3957 5.953e-06 ***
Residuals 55 608.16 11.06
but the multcomp-results indicated no significant post-hoc
differences between any pair of values for both factors (here the
values for the factor "Felder" as example)
Fit: lme.formula(fixed = wPatternPulseSNR ~ Bedingung * Felder, data
= d,
random = ~1 | VPerson)
Linear Hypotheses:
Estimate Std. Error z value Pr(>|z|)
Feld2 - Feld1 == 0 1.89748 0.84245 2.252 0.214
Feld3 - Feld1 == 0 1.41383 0.84245 1.678 0.546
Feld4 - Feld1 == 0 1.48945 0.84245 1.768 0.487
Feld5 - Feld1 == 0 -0.11133 0.84245 -0.132 1.000
Feld6 - Feld1 == 0 0.02472 0.84245 0.029 1.000
Feld3 - Feld2 == 0 -0.48366 0.84245 -0.574 0.993
Feld4 - Feld2 == 0 -0.40803 0.84245 -0.484 0.997
Feld5 - Feld2 == 0 -2.00882 0.84245 -2.385 0.162
Feld6 - Feld2 == 0 -1.87277 0.84245 -2.223 0.227
Feld4 - Feld3 == 0 0.07562 0.84245 0.090 1.000
Feld5 - Feld3 == 0 -1.52516 0.84245 -1.810 0.459
Feld6 - Feld3 == 0 -1.38911 0.84245 -1.649 0.566
Feld5 - Feld4 == 0 -1.60078 0.84245 -1.900 0.402
Feld6 - Feld4 == 0 -1.46473 0.84245 -1.739 0.506
Feld6 - Feld5 == 0 0.13605 0.84245 0.161 1.000
(Adjusted p values reported -- single-step method)
2) So my main question is whether I really applied the correct data
analysis to the data? Perhaps the discrepancy between the aov-results
and the lme-results indicate the need to perform the post-hoc tests
correctly. On the other hand, the difference between a p-value of
5.953e-06 (aov) and p>0.1 (for all pairwise comparisons) simply
indicates, that I did something wrong...
Best wishes
Thomas
[[alternative HTML version deleted]]
Thomas Meigen
2010-May-31 10:46 UTC
[R] Post-hoc tests for repeated measures in balanced experimental design
Dear Dennis, thank you for your fast response. Perhaps I should have described the experimental situation in more detail.> This tells you that Subject is being treated as a random block > factor, and that > Conditions 1 and 2 are combinations of treatments applied to each > subject. In > other words, this describes a randomized block design, so ordering > in time is > not represented in this model. Moreover, within-subject correlation > due to the > supposed repeated measures is not represented, either.In each subject we perform one recording for every possible combination of values of both factors Condition1 and Condition2. Ordering in time is not an issue here, because we do not want to study longitudinal changes in time. On the other hand, as each combination of the 2 conditions was applied in each subject within an experimental session, I thought this must be analysed with a repeated-measures approach. As an example, the following lines show the beginning of a typical result file for the first 2 subjects A and B Nr Subject Condition1 Condition2 Result 1 A C1a C2a 5.0000 2 A C1a C2b 3.0000 3 A C1b C2a 3.0000 4 A C1b C2b 3.0000 5 A C1c C2a 1.0000 6 A C1c C2b 4.0000 7 B C1a C2a 5.0000 8 B C1a C2b 4.0000 9 B C1b C2a 3.0000 10 B C1b C2b 2.0000 11 B C1c C2a 2.0000 12 B C1c C2b 3.0000 ... If this is a randomized block design, which is the best way to analyse the effect of the two factors Condition1 and Condition2 on Result? How to do post-hoc tests to see whether there significant differences, e. g., between C1a and C1b for Condition1?> ano <- aov(d$wPatternPulseSNR~ d$Bedingung*d$Felder + Error(d$VPerson/ > (d$Bedingung*d$Felder), data=d)) > > Something seems amiss here, too. Where is the time element > represented? > Where is the correlation structure on the supposed repeated > measures? I also > think that you should have Person/Bedingung:Felder as the within- > subject error > term instead....As mentioned above, there is no specific time element, but the repeated measures of the same type of result (here wPatternPulseSNR) under different experimental conditions (here d$Bedingung*d$Felder) should contain the correlation structure. As far as I understood the nomenclature, "Bedingung:Felder" means a different situation (split-plot) from the situation described above.> It's entirely possible that the 'significance' seen in the above > tests is due to > an incorrect choice of error term. If Bedingung and Felder are > both applied > within-subject, then the appropriate error term should be the > average within-person > variance. > > A repeated measures design is structured rather similarly to a > split-plot, except > that in a RM design the within-subject observations are correlated > in time and > therefore the correlation can vary as a function of time - e.g., AR > (1), where the > correlation between observations decreases exponentially with the > time gap > between them. (In contrast, observations at the split-plot level > are usually assumed > to be equi-correlated due to randomization of levels of treatments > within subjects.) > In your case, there is no 'whole-plot/between subject' treatment - > everything is at > the within-subject level....> > I don't think you have the right model if this is indeed a repeated > measures > design, but I'll let others weigh in on the subject. As alluded to > above, your > model doesn't seem to represent the time element or the correlation > structure > that one would expect in a repeated measures design. I'd worry about > getting the model right before proceeding to the multiple comparisons. >Hm, being not an expert in statistics I thank you very much for advice and I agree that the right model is essential. Perhaps my clarification of the data might help to fix this aspect. "Everything is at the within-subject level" – this is exactly true. Thanks again Thomas [[alternative HTML version deleted]]