R help - Jan 2012 - null distribution of binom.test p values

Dear R-help,

I must be missing something very obvious, but I am confused as to why 
the null distribution for p values generated by binom.test() appears to 
be non-uniform.  The histogram generated below has a trend towards 
values closer to 1 than 0.  I expected it to be flat.

hist(sapply(1:1000, function(i,n=100) 
binom.test(sum(rnorm(n)>0),n,p=0.5,alternative="two")$p.value))

This trend is more pronounced for small n, and the distribution appears 
uniform for larger n, say n=1000.  I had expected the distribution to be 
discrete for small n, but not skewed.  Can anyone explain why?

Many thanks,

Chris.

I believe that what you are seeing is due to the discrete nature of the binomial
test.  When I run your code below I see the bar between 0.9 and 1.0 is about
twice as tall as the bar between 0.0 and 0.1, but the bar between 0.8 and 0.9 is
not there (height 0), if you average the top 2 bars (0.8-0.9 and 0.9-1.0) then
the average height is similar to that of the lowest bar.  The bar between 0.5
and 0.6 is also 0, if you average that one with the next 2 (0.6-0.7 and 0.7-0.8)
then they are also similar to the bars near 0.



-- 
Gregory (Greg) L. Snow Ph.D.
Statistical Data Center
Intermountain Healthcare
greg.snow at imail.org
801.408.8111

> -----Original Message-----
> From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-
> project.org] On Behalf Of Chris Wallace
> Sent: Thursday, January 26, 2012 5:44 AM
> To: r-help at r-project.org
> Subject: [R] null distribution of binom.test p values
> 
> Dear R-help,
> 
> I must be missing something very obvious, but I am confused as to why
> the null distribution for p values generated by binom.test() appears to
> be non-uniform.  The histogram generated below has a trend towards
> values closer to 1 than 0.  I expected it to be flat.
> 
> hist(sapply(1:1000, function(i,n=100)
>
binom.test(sum(rnorm(n)>0),n,p=0.5,alternative="two")$p.value))
> 
> This trend is more pronounced for small n, and the distribution appears
> uniform for larger n, say n=1000.  I had expected the distribution to
> be
> discrete for small n, but not skewed.  Can anyone explain why?
> 
> Many thanks,
> 
> Chris.
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-
> guide.html
> and provide commented, minimal, self-contained, reproducible code.

Greg, thanks for the reply.

Unfortunately, I remain unconvinced!

I ran a longer simulation, 100,000 reps.  The size of the test is 
consistently too small (see below) and the histogram shows increasing 
bars even within the parts of the histogram with even bar spacing.  See 
https://www-gene.cimr.cam.ac.uk/staff/wallace/hist.png

y<-sapply(1:100000, function(i,n=100)
  binom.test(sum(rnorm(n)>0),n,p=0.5,alternative="two")$p.value)
mean(y<0.01)
# [1] 0.00584
mean(y<0.05)
# [1] 0.03431
mean(y<0.1)
# [1] 0.08646

Can that really be due to the discreteness of the distribution?

C.

On 26/01/12 16:08, Greg Snow wrote:
> I believe that what you are seeing is due to the discrete nature of the
binomial test.  When I run your code below I see the bar between 0.9 and 1.0 is
about twice as tall as the bar between 0.0 and 0.1, but the bar between 0.8 and
0.9 is not there (height 0), if you average the top 2 bars (0.8-0.9 and 0.9-1.0)
then the average height is similar to that of the lowest bar.  The bar between
0.5 and 0.6 is also 0, if you average that one with the next 2 (0.6-0.7 and
0.7-0.8) then they are also similar to the bars near 0.
>
>
>