R help - Jul 2005 - Partek has Dunn-Sidak Multiple Test Correction. Is this the same/similar to any of R's p.adjust.methods?

The Partek package (www.partek.com) allows only two selections for Multiple
Test Correction:  Bonferroni and Dunn-Sidak.  Can anyone suggest why Partek
implemented Dunn-Sidak and not the other methods that R has?  Is there any
particular advantage to the Dunn-Sidak method?
R knows about these methods (in R 2.1.1):
> p.adjust.methods
[1] "holm" "hochberg" "hommel"
"bonferroni" "BH" "BY" "fdr"
[8] "none"

BH is Benjamini & Hochberg (1995) and is also called "fdr" (I wish
R's
documentation said this clearly).  BY is Benjamini & Yekutieli (2001).

I found a few hits from Google on Dunn-Sidak, but I'm curious if anyone can
tell me on a "conservative-liberal" scale, where the Dunn-Sidak method
falls? My guess is it's less conservative than Bonferroni (but aren't
all
the other methods?), but how does it compare to the other methods?

A limited numerical experiment suggested this order to me:  bonferroni (most
conservative), hochberg and holm about the same, BY, BH (also called fdr),
and then none.

Thanks for any of  thoughts on this.

efg

"Earl F. Glynn" <efg at stowers-institute.org> writes:
> The Partek package (www.partek.com) allows only two selections for Multiple
> Test Correction:  Bonferroni and Dunn-Sidak.  Can anyone suggest why Partek
> implemented Dunn-Sidak and not the other methods that R has?  Is there any
> particular advantage to the Dunn-Sidak method?
> R knows about these methods (in R 2.1.1):
> 
> > p.adjust.methods
> [1] "holm" "hochberg" "hommel"
"bonferroni" "BH" "BY" "fdr"
> [8] "none"
> 
> BH is Benjamini & Hochberg (1995) and is also called "fdr" (I
wish R's
> documentation said this clearly).  BY is Benjamini & Yekutieli (2001).
> 
> I found a few hits from Google on Dunn-Sidak, but I'm curious if anyone
can
> tell me on a "conservative-liberal" scale, where the Dunn-Sidak
method
> falls? My guess is it's less conservative than Bonferroni (but
aren't all
> the other methods?), but how does it compare to the other methods?
As far as I gather, D-S is exact for independent tests, conservative
for comparisons of group means, and liberal for mutually exclusive
tests (in which case Bonferroni is exact). It is always less
conservative than Bonferroni, but the difference is small for typical
significance levels: when the Bonferroni level is p, the D-S level is

   1 - (1-p/N)^N

and if you put p=0.05 and vary N you'll find that it varies from 0.05
at N=1 down to 0.04877 at N=100000. (Exercise for the students: what
is the limit as N goes to infinity?)

The three H-methods play a somewhat different game, basically by only
requiring multiple-testing adjustment for non-significant tests.  The
FDR methods play yet differently by allowing the per test level to
increase with the number of significant tests.
 
> A limited numerical experiment suggested this order to me:  bonferroni
(most
> conservative), hochberg and holm about the same, BY, BH (also called fdr),
> and then none.
> 
> Thanks for any of  thoughts on this.
I'd expect the differences to be fairly small in scenarios where the
global null hypothesis is true (excluding "none"). The main difference
comes in when some of the nulls are actually false. Also, it depends
on your definitions: With the exception of "BY" and "none"
the
p.adjust methods agree on the smallest adjusted p value, so have the
same familywise error rate under the global null. If you count the
total number of rejected tests, then you get a difference due to
"cascading" in the non-bonferroni cases.

-- 
   O__  ---- Peter Dalgaard             ??ster Farimagsgade 5, Entr.B
  c/ /'_ --- Dept. of Biostatistics     PO Box 2099, 1014 Cph. K
 (*) \(*) -- University of Copenhagen   Denmark          Ph:  (+45) 35327918
~~~~~~~~~~ - (p.dalgaard at biostat.ku.dk)                  FAX: (+45) 35327907

>>>>> "Earl" == Earl F Glynn <efg at
stowers-institute.org>
>>>>>     on Thu, 14 Jul 2005 12:22:49 -0500 writes:
    Earl> The Partek package (www.partek.com) allows only two selections for
Multiple
    Earl> Test Correction:  Bonferroni and Dunn-Sidak.  Can anyone suggest
why Partek
    Earl> implemented Dunn-Sidak and not the other methods that R has?  Is
there any
    Earl> particular advantage to the Dunn-Sidak method?
    Earl> R knows about these methods (in R 2.1.1):

    >> p.adjust.methods
    Earl> [1] "holm" "hochberg" "hommel"
"bonferroni" "BH" "BY" "fdr"
    Earl> [8] "none"

    Earl> BH is Benjamini & Hochberg (1995) and is also called
"fdr" (I wish R's
    Earl> documentation said this clearly).  BY is Benjamini & Yekutieli
(2001).

The current R docu has

  >>   The '"BH"' and '"BY"' method of
Benjamini, Hochberg, and Yekutieli
  >>   control the false discovery rate, the expected proportion of false
  >>   discoveries amongst the rejected hypotheses.  The false discovery
  >>   rate is a less stringent condition than the family wise error
  >>   rate, so these methods are more powerful than the others.

so both "BH" and "BY"   ``are FDR versions''. 
"fdr" was used - unfortunately - in some older versions of R,
so we kept it working as an *alias* for the time being.  
You should rather not know about it :-)
and use "BH" or "BY" (and maybe other methods in the future)
instead.

Regards,

Martin