R help - Nov 2003 - Difference in ANOVA results - R vs. JMP/Minitab

Hi,

I ran a small data set from a factorial experiment through R, Minitab 
and JMP... the result from R is significantly different from what 
Minitab or JMP give... The data set is at the following link:

http://www.personal.psu.edu/nug107/Uploads/2x3_16repsANOVA.txt

The first 5 columns are the factors and the next three are responses.
In particular, for the response beta11MSE, two of the significant 
effects in the Minitab (and JMP) ANOVA table are not significant in R... 
  What may be the cause for such a difference? I used the lm function in 
R to do the model fitting and fit a model with main effects and 2-way 
interactions in the 5 factors. The design that generated this data was a 
32-run 2^5 full factorial.

Thanks,
nirmal

Dear Nirmal,

At 07:36 AM 11/19/2003 -0500, Nirmal Govind wrote:
>I ran a small data set from a factorial experiment through R, Minitab and 
>JMP... the result from R is significantly different from what Minitab or 
>JMP give... The data set is at the following link:
>
>http://www.personal.psu.edu/nug107/Uploads/2x3_16repsANOVA.txt
>
>The first 5 columns are the factors and the next three are responses.
>In particular, for the response beta11MSE, two of the significant effects 
>in the Minitab (and JMP) ANOVA table are not significant in R...  What may 
>be the cause for such a difference? I used the lm function in R to do the 
>model fitting and fit a model with main effects and 2-way interactions in 
>the 5 factors. The design that generated this data was a 32-run 2^5 full 
>factorial.
 From your description, it's not possible to tell what you did or exactly 
what happened -- either in R or in Minitab (and JMP).

Moreover, bcause the design is orthogonal and the factors are dichotomous, 
it's very difficult to get the analysis wrong, at least in R. I believe 
that you should get correct sums of squares even if you don't treat the 
factors as factors, regardless of contrast coding, and even if you use 
"sequential" SSs, as calculated by the anova function (all of which
could
get you in trouble more generally).

Here are the results that I get from R and SAS (I don't have Minitab or
JMP):

 > anova(lm(beta11MSE ~ (Cycles + Replns + VarZ + NormB + NormG)^2))
Analysis of Variance Table

Response: beta11MSE
               Df  Sum Sq Mean Sq F value    Pr(>F)
Cycles         1 0.00102 0.00102  0.1602  0.694250
Replns         1 0.18457 0.18457 28.9170 6.164e-05 ***
VarZ           1 0.26800 0.26800 41.9879 7.597e-06 ***
NormB          1 0.00491 0.00491  0.7689  0.393527
NormG          1 0.33606 0.33606 52.6497 1.921e-06 ***
Cycles:Replns  1 0.00004 0.00004  0.0059  0.939751
Cycles:VarZ    1 0.00131 0.00131  0.2060  0.656054
Cycles:NormB   1 0.00008 0.00008  0.0131  0.910348
Cycles:NormG   1 0.00088 0.00088  0.1377  0.715457
Replns:VarZ    1 0.07781 0.07781 12.1897  0.003018 **
Replns:NormB   1 0.00360 0.00360  0.5636  0.463697
Replns:NormG   1 0.08693 0.08693 13.6199  0.001982 **
VarZ:NormB     1 0.00351 0.00351  0.5498  0.469155
VarZ:NormG     1 0.08990 0.08990 14.0843  0.001737 **
NormB:NormG    1 0.00268 0.00268  0.4206  0.525839
Residuals     16 0.10213 0.00638


proc glm;
   class Cycles Replns VarZ NormB NormG;
   model beta11MSE = Cycles|Replns Cycles|VarZ Cycles|NormB Cycles|NormG
   Replns|VarZ Replns|NormB Replns|NormG VarZ|NormB VarZ|NormG NormB|NormG;
   run;

       Source                      DF       Type I SS     Mean Square    F 
Value    Pr > F

       Cycles                       1      0.00102265      0.00102265 
0.16    0.6942
       Replns                       1      0.18457369      0.18457369 
28.92    <.0001
       Cycles*Replns                1      0.00003763      0.00003763 
0.01    0.9398
       VarZ                         1      0.26800351      0.26800351 
41.99    <.0001
       Cycles*VarZ                  1      0.00131456      0.00131456 
0.21    0.6561
       NormB                        1      0.00490793      0.00490793 
0.77    0.3935
       Cycles*NormB                 1      0.00008353      0.00008353 
0.01    0.9103
       NormG                        1      0.33605652      0.33605652 
52.65    <.0001
       Cycles*NormG                 1      0.00087885      0.00087885 
0.14    0.7155
       Replns*VarZ                  1      0.07780526      0.07780526 
12.19    0.0030
       Replns*NormB                 1      0.00359764      0.00359764 
0.56    0.4637
       Replns*NormG                 1      0.08693408      0.08693408 
13.62    0.0020
       VarZ*NormB                   1      0.00350913      0.00350913 
0.55    0.4692
       VarZ*NormG                   1      0.08989860      0.08989860 
14.08    0.0017
       NormB*NormG                  1      0.00268461      0.00268461 
0.42    0.5258


       Source                      DF     Type III SS     Mean Square    F 
Value    Pr > F

       Cycles                       1      0.00102265      0.00102265 
0.16    0.6942
       Replns                       1      0.18457369      0.18457369 
28.92    <.0001
       Cycles*Replns                1      0.00003763      0.00003763 
0.01    0.9398
       VarZ                         1      0.26800351      0.26800351 
41.99    <.0001

       . . .


So, what's the problem?

I hope that this helps,
  John
-----------------------------------------------------
John Fox
Department of Sociology
McMaster University
Hamilton, Ontario, Canada L8S 4M4
email: jfox at mcmaster.ca
phone: 905-525-9140x23604
web: www.socsci.mcmaster.ca/jfox

> From: Nirmal Govind [mailto:nirmalg at psu.edu] 
> 
> Thanks for your reply, Prof. Ripley.
> 
> > And all of that is explained in the reference on the aov help page. 
> > Please do stop speculating and start reading.
> 
> Well, the reason for speculation was cos I couldn't find any 
> details on 
> that help page... thanks for letting me know that the 
> reference explains 
> this...
> 
> > AoV has nothing to do with coefficients.  There is 
> dummy.coef to get 
> > uncoded coefficients, and there is options(contrasts=) to set the 
> > coding.
> 
> Ok, I will try these...
> 
> > Yes, you should read a lot more.  Chapter 6 of Venables & Ripley 
> > (2002)
> > and chapter 5 of Chambers & Hastie (1991) would be a good start.
> 
> Oh, I most definitely should! :-) .. I guess I was hoping that such 
> details would be included in the help page for the function 
> or mentioned 
> in some R introduction or tutorial or so on the Documentation page.. 
> looks like that's not the case so I'll get a hold of the 
> books above...
> 
> Thanks,
> nirmal
It's unreasonable to expect stat software documentation to also serve as
textbooks or reference books.  What you have not learned in a course can not
be made up just by reading help pages.  That's not the purpose of help
pages.

Andy