Dear all, I found a discrepancy while performing a power calculation for a two sample t-test in R and S-PLUS, respectively. For given values of sample number (5 each), sd (0.2) , significance level (0.01), and a desired power (80%) I looked for the difference in means. These values differ: 0.5488882 in R and 0.4322771 in S-PLUS (see dump below). Did I overlook any detail or confuse some parameters? Thanks for your help, Joern Quedenau Here are the commands & outputs from both tools: R 1.4.0> power.t.test(n=5, sd=0.2, sig.level=0.01, power=0.8, type="two.sample",alternative="two.sided") Two-sample t test power calculation n = 5 delta = 0.5488882 sd = 0.2 sig.level = 0.01 power = 0.8 alternative = two.sided NOTE: n is number in *each* group S-PLUS 2000 Professional Release 2:> normal.sample.size(n1=5, n2=5, mean=0, sd1=0.2, sd2=0.2, power=0.8,alpha=0.01, alternative="two.sided") mean1 sd1 mean2 sd2 delta alpha power n1 n2 prop.n2 1 0 0.2 0.4322771 0.2 0.4322771 0.01 0.8 5 5 1 ------------------------------------------ Dr. J?rn Quedenau Coordinator Data Management Bioinformatics Metanomics GmbH & Co. KGaA Tegeler Weg 33, D-10589 Berlin, Germany Tel +49 30 34807 125, Fax +49 30 34807 300 -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
On Fri, 1 Mar 2002 joern.quedenau at metanomics.de wrote:> > > Dear all, > > I found a discrepancy while performing a power calculation for a two sample > t-test in R and S-PLUS, respectively. > For given values of sample number (5 each), sd (0.2) , significance level > (0.01), and a desired power (80%) I looked for the difference in means. > These values differ: 0.5488882 in R and 0.4322771 in S-PLUS (see dump > below). > > Did I overlook any detail or confuse some parameters?Yes. normal.sample.size is not for a Student's t test, as its name might suggest (at least, it did to me). It uses a normal distribution and assumes the variances are known. On the other hand, power.t.test appears to be for a conventional equi-variance t-test, and as it needs to estimate the variances has lower power and hence selects a larger minimum delta. BTW, the t-test that power.t.test computes the power of is not the default t-test in R, as I understand the code. (?power.t.test is silent on which two-sample t-test but the calculations look right for the one that uses the pooled variance. Not that it necessarily matters.) This is only evident because n1 and n2 are so small: at those sample sizes you are relying critically on normality of the samples, and for power.t.test on equal variances. For n1 = n2 = 25, the difference is much smaller (0.193 vs 0.200).> > Thanks for your help, > Joern Quedenau > > Here are the commands & outputs from both tools: > > R 1.4.0 > > power.t.test(n=5, sd=0.2, sig.level=0.01, power=0.8, type="two.sample", > alternative="two.sided") > > Two-sample t test power calculation > > n = 5 > delta = 0.5488882 > sd = 0.2 > sig.level = 0.01 > power = 0.8 > alternative = two.sided > > NOTE: n is number in *each* group > > S-PLUS 2000 Professional Release 2: > > normal.sample.size(n1=5, n2=5, mean=0, sd1=0.2, sd2=0.2, power=0.8, > alpha=0.01, alternative="two.sided") > mean1 sd1 mean2 sd2 delta alpha power n1 n2 prop.n2 > 1 0 0.2 0.4322771 0.2 0.4322771 0.01 0.8 5 5 1 > > ------------------------------------------ > Dr. Jörn Quedenau > Coordinator Data Management Bioinformatics > Metanomics GmbH & Co. KGaA > Tegeler Weg 33, D-10589 Berlin, Germany > Tel +49 30 34807 125, Fax +49 30 34807 300 > > > -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- > r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html > Send "info", "help", or "[un]subscribe" > (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch > _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._ >-- Brian D. Ripley, ripley at stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272860 (secr) Oxford OX1 3TG, UK Fax: +44 1865 272595 -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
On Fri, 1 Mar 2002 joern.quedenau at metanomics.de wrote:> > > Dear all, > > I found a discrepancy while performing a power calculation for a two sample > t-test in R and S-PLUS, respectively. > For given values of sample number (5 each), sd (0.2) , significance level > (0.01), and a desired power (80%) I looked for the difference in means. > These values differ: 0.5488882 in R and 0.4322771 in S-PLUS (see dump > below). > > Did I overlook any detail or confuse some parameters?The S-PLUS routine references Fisher & van Belle. In that book the authors use a unified approximate power calculation method that works for a wide range of studies but is not very accurate for tiny sample sizes. In most cases this doesn't matter because the assumptions going into a study design aren't any more accurate, and in tiny sample sizes the power is sensitive to the assumption that the data are Normally distributed. The power.t.test formula uses the non-central t distribution and so will give more accurate, lower power values for small samples. You can see which one is correct by simulation (which is how I typically do power calculations)> table(sapply(1:10000,function(i)t.test(rnorm(5,0,s=0.2),rnorm(5,.5488882,s=0.2),var.equal=TRUE)$p.value)<=0.01) FALSE TRUE 2023 7977> table(sapply(1:10000,function(i)t.test(rnorm(5,0,s=0.2),rnorm(5,.4322771,s=0.2),var.equal=TRUE)$p.value)<=0.01) FALSE TRUE 4526 5474 So at 0.548882 there is about 80% power, at 0.4322771 there is about 55% power (with sampling uncertainties of about +/- 2% in each number). It is interesting to note that a simulation shows the unequal-variance t-test to have only 75% power at 0.5488882, indicating the sensitivity of the power calculations at this sample size. -thomas Thomas Lumley Asst. Professor, Biostatistics tlumley at u.washington.edu University of Washington, Seattle> > Thanks for your help, > Joern Quedenau > > Here are the commands & outputs from both tools: > > R 1.4.0 > > power.t.test(n=5, sd=0.2, sig.level=0.01, power=0.8, type="two.sample", > alternative="two.sided") > > Two-sample t test power calculation > > n = 5 > delta = 0.5488882 > sd = 0.2 > sig.level = 0.01 > power = 0.8 > alternative = two.sided > > NOTE: n is number in *each* group > > S-PLUS 2000 Professional Release 2: > > normal.sample.size(n1=5, n2=5, mean=0, sd1=0.2, sd2=0.2, power=0.8, > alpha=0.01, alternative="two.sided") > mean1 sd1 mean2 sd2 delta alpha power n1 n2 prop.n2 > 1 0 0.2 0.4322771 0.2 0.4322771 0.01 0.8 5 5 1 > > ------------------------------------------ > Dr. Jörn Quedenau > Coordinator Data Management Bioinformatics > Metanomics GmbH & Co. KGaA > Tegeler Weg 33, D-10589 Berlin, Germany > Tel +49 30 34807 125, Fax +49 30 34807 300 > > > -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- > r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html > Send "info", "help", or "[un]subscribe" > (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch > _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._ >Thomas Lumley Asst. Professor, Biostatistics tlumley at u.washington.edu University of Washington, Seattle ^^^^^^^^^^^^^^^^^^^^^^^^ NOTE NEW EMAIL ADDRESS -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
Could someone shed some light on the statement from Thomas that "(with sampling uncertainties of about +/- 2% in each number)". What exactly is being said about the accuracy of the simulation and how is that number +/-2% being determined. Thanks, Alan> So at 0.548882 there is about 80% power, at 0.4322771 there is about 55% > power (with sampling uncertainties of about +/- 2% in each number). > > > It is interesting to note that a simulation shows the unequal-variance > t-test to have only 75% power at 0.5488882, indicating the sensitivity of > the power calculations at this sample size. > > -thomas > > Thomas Lumley Asst. Professor, Biostatistics > tlumley at u.washington.edu University of Washington, Seattle---------------------- Alan T. Arnholt Associate Professor Department of Mathematical Sciences Appalachian State University Tel: (828) 262-2863 Fax: (828) 265-8617 http://www1.appstate.edu/~arnholta/ -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._