Hi,
I have a little problem that puzzles me about contradictory results returned
by a bootstraped t-test (as in MASS 3rd ed p. 146) versus a permutation
t-test (as in MASS 3rd ed, p 147).
Data are measurements done on 100 cells in 9 slides of fish blood. With one
method, cells are randomly sampled, and with the other method, the operator
selects cells arbitrarily (in a way it is done usually with this test). We
want to determine wheither the methods yield same results or not. Since we
are interested by the mean measurement for 100 cells, we take the average
for each slide and each method. We compare then the nine paired samples
(that is, for the nine slides) with a paired t-test. However, since we
cannot make the hypothesis that both distributions are normal, we prefer to
use a bootstraped test.
We do:
(1) 1000 simple bootstraps with:
boot(B-A, function(x,i), mean(x[i]), R=1000)
and then:
boot.ci(...)
and check wheter the CI includes 0 (no significant difference between
methods) or not.
(2) a permutation test with the perm.t.test() function of MASS p. 147
and calculate a bootstraped p-value corresponding to the fraction of values
larger or equal to the observed one. If this p-value is > 5%, we consider
there is no significant difference between both methods.
Is this correct?
The problem is that, in our particular case, both test give opposite
results: the bootstrap test indicates significant differences at 5%, while
the permutation test gives p-value = 0.35-0.45, thus no differences between
methods. I think I probably miss something here! Does somebody could help
me?
Best,
Philippe Grosjean
...........]<(({?<...............<?}))><...............................
( ( ( ( (
) ) ) ) ) Philippe Grosjean
( ( ( ( (
) ) ) ) ) IFREMER Nantes - DEL/AO
( ( ( ( ( rue de l'Ile d'Yeu, BP 21105, 44311 Nantes Cedex 3
) ) ) ) ) tel: (33) 02.40.37.42.29, fax: (33) 02.40.37.42.41
( ( ( ( ( e-mail: philippe.grosjean at ifremer.fr
) ) ) ) )
( ( ( ( ( "I'm 100% confident that p is between 0 and 1"
) ) ) ) ) L. Gonick & W. Smith (1993)
.......................................................................
Philippe Grosjean wrote:> > Hi, > > I have a little problem that puzzles me about contradictory results returned > by a bootstraped t-test (as in MASS 3rd ed p. 146) versus a permutation > t-test (as in MASS 3rd ed, p 147). > > Data are measurements done on 100 cells in 9 slides of fish blood. With one > method, cells are randomly sampled, and with the other method, the operator > selects cells arbitrarily (in a way it is done usually with this test). We > want to determine wheither the methods yield same results or not. Since we > are interested by the mean measurement for 100 cells, we take the average > for each slide and each method. We compare then the nine paired samples > (that is, for the nine slides) with a paired t-test. However, since we > cannot make the hypothesis that both distributions are normal, we prefer to > use a bootstraped test.But with a paired t-test you doesn't need to assume that both distributions are normal, it holds that the difference is? can you assume that? The differen can be large, as if both distributions are equal, the difference certainly is symmetric. Kjetil Halvorsen> > We do: > (1) 1000 simple bootstraps with: > boot(B-A, function(x,i), mean(x[i]), R=1000) > and then: > boot.ci(...) > and check wheter the CI includes 0 (no significant difference between > methods) or not. > > (2) a permutation test with the perm.t.test() function of MASS p. 147 > and calculate a bootstraped p-value corresponding to the fraction of values > larger or equal to the observed one. If this p-value is > 5%, we consider > there is no significant difference between both methods. > > Is this correct? > > The problem is that, in our particular case, both test give opposite > results: the bootstrap test indicates significant differences at 5%, while > the permutation test gives p-value = 0.35-0.45, thus no differences between > methods. I think I probably miss something here! Does somebody could help > me? > > Best, > > Philippe Grosjean > > ...........]<(({?<...............<?}))><............................... > ( ( ( ( ( > ) ) ) ) ) Philippe Grosjean > ( ( ( ( ( > ) ) ) ) ) IFREMER Nantes - DEL/AO > ( ( ( ( ( rue de l'Ile d'Yeu, BP 21105, 44311 Nantes Cedex 3 > ) ) ) ) ) tel: (33) 02.40.37.42.29, fax: (33) 02.40.37.42.41 > ( ( ( ( ( e-mail: philippe.grosjean at ifremer.fr > ) ) ) ) ) > ( ( ( ( ( "I'm 100% confident that p is between 0 and 1" > ) ) ) ) ) L. Gonick & W. Smith (1993) > ....................................................................... > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > http://www.stat.math.ethz.ch/mailman/listinfo/r-help
> Hi, > > I have a little problem that puzzles me about contradictory results returned > by a bootstraped t-test (as in MASS 3rd ed p. 146) versus a permutation > t-test (as in MASS 3rd ed, p 147). > > Data are measurements done on 100 cells in 9 slides of fish blood. With one > method, cells are randomly sampled, and with the other method, the operator > selects cells arbitrarily (in a way it is done usually with this test). We > want to determine wheither the methods yield same results or not. Since we > are interested by the mean measurement for 100 cells, we take the average > for each slide and each method. We compare then the nine paired samples > (that is, for the nine slides) with a paired t-test. However, since we > cannot make the hypothesis that both distributions are normal, we prefer to > use a bootstraped test. >that is: you have 9 differences and you hypothesis is "the underlying distribution is symmetric about zero", i.e. the two methods do not differ in this sense, right?> We do: > (1) 1000 simple bootstraps with: > boot(B-A, function(x,i), mean(x[i]), R=1000) > and then: > boot.ci(...) > and check wheter the CI includes 0 (no significant difference between > methods) or not.so you are computing a bootstrap estimate of the standard error of the mean for later use in a confidence interval for the mean? But you can calculate this directly (Section 5.2 in Efron/Tibshirani: Intro to the Boostrap) and the confidence set per t.test, if I'm not completely misguided.> > (2) a permutation test with the perm.t.test() function of MASS p. 147 > and calculate a bootstraped p-value corresponding to the fraction of values^^^^^^^^^^^ I just can't find my MASS3 at the moment, but I suspect perm.t.test computes the statistic for all possible permutations, so no bootstrap here.> larger or equal to the observed one. If this p-value is > 5%, we consider > there is no significant difference between both methods. > > Is this correct? > > The problem is that, in our particular case, both test give opposite > results: the bootstrap test indicates significant differences at 5%, while > the permutation test gives p-value = 0.35-0.45, thus no differences between > methods. I think I probably miss something here! Does somebody could help > me?maybe you should post the 9 measurements ;-) For the shoes data, one could do the following, which looks consitent to me: R> R> library(MASS) R> library(exactRankTests) R> data(shoes) R> attach(shoes) R> t.test(A,B, paired=TRUE, conf.int=TRUE) Paired t-test data: A and B t = -3.3489, df = 9, p-value = 0.008539 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -0.6869539 -0.1330461 sample estimates: mean of the differences -0.41 R> wilcox.exact(A,B, paired=TRUE, conf.int=TRUE) Exact Wilcoxon signed rank test data: A and B V = 3, p-value = 0.007812 alternative hypothesis: true mu is not equal to 0 95 percent confidence interval: -0.7 -0.1 sample estimates: (pseudo)median -0.4 R> wilcox.test(A,B, paired=TRUE, exact=FALSE) Wilcoxon signed rank test with continuity correction data: A and B V = 3, p-value = 0.01431 alternative hypothesis: true mu is not equal to 0 R> perm.test(A*10,B*10, paired=TRUE) # map into integers 1-sample Permutation Test data: A * 10 and B * 10 T = 3, p-value = 0.01367 alternative hypothesis: true mu is not equal to 0 best, Torsten> > Best, > > Philippe Grosjean > > ...........]<(({?<...............<?}))><............................... > ( ( ( ( ( > ) ) ) ) ) Philippe Grosjean > ( ( ( ( ( > ) ) ) ) ) IFREMER Nantes - DEL/AO > ( ( ( ( ( rue de l'Ile d'Yeu, BP 21105, 44311 Nantes Cedex 3 > ) ) ) ) ) tel: (33) 02.40.37.42.29, fax: (33) 02.40.37.42.41 > ( ( ( ( ( e-mail: philippe.grosjean at ifremer.fr > ) ) ) ) ) > ( ( ( ( ( "I'm 100% confident that p is between 0 and 1" > ) ) ) ) ) L. Gonick & W. Smith (1993) > ....................................................................... > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > http://www.stat.math.ethz.ch/mailman/listinfo/r-help >