Hello. I have come across a curious result that I cannot explain. Hopefully, someone can explain this. I am doing a 1-way ANOVA with 6 groups (example: summary(aov(y~A)) with A having 6 levels). I get an F of 0.899 with 5 and 15 df (p=0.51). I then do the same analysis but using data only corresponding to groups 5 and 6. This is, of course, equivalent to a t-test. I now get an F of 142.3 with 1 and 3 degrees of freedom and a null probability of 0.001. I know that multiple comparisons changes the model-wise error rate, but even if I did all 15 comparisons of the 6 groups, the Bonferroni correction to a 5% alpha is 0.003, yet the Bonferroni correction gives conservative rejection levels. How can such a result occur? Any clues would be helpful. Thanks. Bill Shipley Associate Editor, Ecology North American Editor, Annals of Botany Département de biologie, Université de Sherbrooke, Sherbrooke (Québec) J1K 2R1 CANADA Bill.Shipley@USherbrooke.ca <http://callisto.si.usherb.ca:8080/bshipley/> http://callisto.si.usherb.ca:8080/bshipley/ [[alternative HTML version deleted]]
The ANOVA assumes equal variances in the groups. Suppose groups 5 and 6 had much lower variances than groups 1 to 4, and group 6 had a different mean from the other 5 (which were about equal)? Given how small the groups apperat to be, this could happen. On Tue, 21 Oct 2003, Bill Shipley wrote:> Hello. I have come across a curious result that I cannot explain. > Hopefully, someone can explain this. I am doing a 1-way ANOVA with 6 > groups (example: summary(aov(y~A)) with A having 6 levels). I get an F > of 0.899 with 5 and 15 df (p=0.51). I then do the same analysis but > using data only corresponding to groups 5 and 6. This is, of course, > equivalent to a t-test. I now get an F of 142.3 with 1 and 3 degrees of > freedom and a null probability of 0.001. I know that multiple > comparisons changes the model-wise error rate, but even if I did all 15 > comparisons of the 6 groups, the Bonferroni correction to a 5% alpha is > 0.003, yet the Bonferroni correction gives conservative rejection > levels. > > How can such a result occur? Any clues would be helpful.-- Brian D. Ripley, ripley at stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UK Fax: +44 1865 272595
"Bill Shipley" <bill.shipley at usherbrooke.ca> writes:> Hello. I have come across a curious result that I cannot explain. > Hopefully, someone can explain this. I am doing a 1-way ANOVA with 6 > groups (example: summary(aov(y~A)) with A having 6 levels). I get an F > of 0.899 with 5 and 15 df (p=0.51). I then do the same analysis but > using data only corresponding to groups 5 and 6. This is, of course, > equivalent to a t-test. I now get an F of 142.3 with 1 and 3 degrees of > freedom and a null probability of 0.001. I know that multiple > comparisons changes the model-wise error rate, but even if I did all 15 > comparisons of the 6 groups, the Bonferroni correction to a 5% alpha is > 0.003, yet the Bonferroni correction gives conservative rejection > levels. > > How can such a result occur? Any clues would be helpful.It's a question of assumptions. Notice first that you have some very small groups there. Comparing two groups with 3df means that there are five observations in all, presumably two in one group and three in the other (although it could be 4-1). The joint F test assumes that all the groups have a similar (theoretical) SD, whereas the two group comparison only assumes that those two groups are similar. Suppose one of the other groups had a huge SD; then a joint comparison would clearly lose power if the actual differences were between some of the groups with a smaller SD. On the other hand, the test on 3df is extremely dependent on distributional assumptions, and if data are non-normally distributed, there may be an increased probability of getting a very small variance (quantization can do that, e.g.) and thus a falsely significant result. I.e. I'd take a closer look at the SD's for the 6 groups and perhaps make a dotplot. -- O__ ---- Peter Dalgaard Blegdamsvej 3 c/ /'_ --- Dept. of Biostatistics 2200 Cph. N (*) \(*) -- University of Copenhagen Denmark Ph: (+45) 35327918 ~~~~~~~~~~ - (p.dalgaard at biostat.ku.dk) FAX: (+45) 35327907
On 21-Oct-03 Bill Shipley wrote:> Hello. I have come across a curious result that I cannot explain. > Hopefully, someone can explain this. I am doing a 1-way ANOVA with 6 > groups (example: summary(aov(y~A)) with A having 6 levels). I get an F > of 0.899 with 5 and 15 df (p=0.51). I then do the same analysis but > using data only corresponding to groups 5 and 6. This is, of course, > equivalent to a t-test. I now get an F of 142.3 with 1 and 3 degrees > of freedom and a null probability of 0.001. I know that multiple > comparisons changes the model-wise error rate, but even if I did all 15 > comparisons of the 6 groups, the Bonferroni correction to a 5% alpha is > 0.003, yet the Bonferroni correction gives conservative rejection > levels. > > How can such a result occur? Any clues would be helpful.It's not obvious from your description. However, one possibility (which I very strongly suspect) is apparent heterogeneity of variance, coupled with paucity of data. To wit: The denominator in F is the residual sum of squares (divided by its degrees of freedom -- 15 in your first case, 3 in your second). If the data in groups 5 and 6 are very close to their group means, the group means themselves being more widely separated, then you can indeed get a large F. The very moderate F that you get from the full set of groups is quite compatible with the extreme result from the two-group analysis if the data happen to be more widely spread about their group means than they happen to be in G5+G6. This is the "heterogeneity of variance" side of it. Your denominator df = 3 for the two-group case indicates that you only have 5 data values altogether in these two groups. Your df = 15 for the six-group case indicates that you have only 21 data all told. At an average of 3.5 data per group you have a very thin data set. Your 2.5 data per group in G5+G6 is even thinner. I would be very cautious about interpreting the results in such a case. Perhaps if you told us more about your data we could give a more focussed diagnosis. Best wishes, Ted. -------------------------------------------------------------------- E-Mail: (Ted Harding) <Ted.Harding at nessie.mcc.ac.uk> Fax-to-email: +44 (0)870 167 1972 Date: 21-Oct-03 Time: 17:56:53 ------------------------------ XFMail ------------------------------