Hi Rolf,
Thank you for clarifying. After reading the help page over more carefully, I
realize that I misunderstood what chisq.test. was doing. I thought it was using
simulation whenever the expected frequencies were given using the "p"
argument, whereas it used the asymptotic chi-squared distribution of the test
statistic when "p"was not given. I now see that, based on the help
page, the asymptotic distribution is always used, unless simulate.p.value is
TRUE.
Mike
On Dec 25, 2011, at 3:06 AM, Rolf Turner wrote:
>
> When the p-value is calculated via simulation the test is not, strictly
> speaking a ``chi-squared test''. That is, it is *not* assumed
that the
> distribution of the test statistic, under the null hypothesis, is a
chi-squared
> distribution. Instead an empirical distribution derived from simulating
> samples under the null hypothesis is used. The *same statistic* is
calculated,
> but the chi-squared distribution is never invoked. The simulated p-value
> facility is provided so that tests can be conducted even when the
requirements
> for the validity of the chi-squared (null) distribution are not met.
>
> The test, whether you want to call it a chi-squared test or not, is a
perfectly
> valid goodness of fit test. (Modulo the caveat that I previously
mentioned,
> i.e. that it is required that there be no ties amongst the values of the
test
> statistics. Which is a mild worry, but has nothing to do with continuity
corrections
> which are meaningless in this context.)
>
> cheers,
>
> Rolf Turner
>
> On 25/12/11 08:26, Michael Fuller wrote:
>> On Dec 22, 2011, at 8:56 PM, Rolf Turner wrote:
>>
>>> On 20/12/11 10:24, Michael Fuller wrote:
>>>> TOPIC
>>>> My question regards the philosophy behind how R implements
corrections to chi-square statistical tests. At least in recent versions
(I'm using 2.13.1 (2011-07-08) on OSX 10.6.8.), the chisq.test function
applies the Yates continuity correction for 2 by 2 contingency tables. But when
used as a goodness of fit test (GoF, aka likelihood ratio test), chisq.test does
not appear to implement any corrections for widely recognized problems, such as
small sample size, non-uniform expected frequencies, and one D.F.
>>>>
>>>>> From the help page:
>>>> "In the goodness-of-fit case simulation is done by random
sampling from the discrete distribution specified by p, each sample being of
size n = sum(x)."
>>>>
>>>> Is the thinking that random sampling completely obviates the
need for corrections?
>>> Yes.
>>>> Wouldn't the same statistical issues still apply
>>> No.
>>>> (e.g. poor continuity approximation with one D.F.,
>>> There are no degrees of freedom involved. There is no
continuity involved.
>>> The observed test statistics (say "Stat") is compared
with a number of
>>> test statistics, Stat_1, ..., Stat_N, calculated from data sets
simulated under
>>> the null hypothesis. If the null is true, then Stat and Stat_1,
...., Stat_N are
>>> all of ``equal status''. If there are m values of the
Stat_i which are greater
>>> than Stat, then the ``probability of observing, under the null
hypothesis,
>>> data as extreme as, or more extreme than, what you actually
observed''
>>> is the probability of randomly selecting one of a specified set
of m+1 ``slots''
>>> out of a total of N+1 slots (where each slot has probability
1/(N+1)).
>> But if the test is truly a chi-square Goodness of Fit (GoF) test, then:
>>
>> (1) the test compares the Stat to a chi-square distribution with (k-1)
degrees of
>> freedom, for k frequency categories.
>>
>> (2) the shape of the distribution depends on the degrees of freedom
>>
>> (3) continuity is an issue, because the values for Stat are discrete,
whereas
>> the chi-square distribution is continuous. Therefore the p-value is not
exact.
>>
>> If I understand your description, the function generates a probability
>> distribution by sampling a set of values with equal probability. The
probability of
>> any value in the final distribution depends on its frequency in the
sample set. This
>> is a standard Monte Carlo method, but the PDF it generates is not
chi-square (the
>> resulting distribution depends upon the distribution of the sample
set). Clearly,
>> using a permutation method such as this to determine the statistical
significance of
>> a given test statistic is non-parametric and in particular, not
chi-square.
>>
>> If what you say is true, then the chisq.test function in R does not
implement a true GoF
>> test. I don't understand why the GoF test is not computed the
standard way. It should
>> compute the chi-square statistic, using the values of p to generate
expected
>> frequencies. And it should use a continuity correction when degrees of
freedom = 1.
>>
>> Thank you for taking the time to respond to my message.
>>
>> Cheers,
>> Mike
>>
>>> Thus the p-value is (exactly) equal to (m+1)/(N+1).
>>>
>>> The only restriction is that there be no ties amongst the values
of Stat
>>> and Stat_1, ..., Stat_N. There being ties is of fairly low
probability, but is
>>> not of zero probability --- since there is a finite number of
possible samples
>>> and hence of statistic values. So this restriction is a mild
worry.
>>>
>>> However a ``continuity correction'' would be of no help
whatsoever.
>>>> problems with non-uniform expected frequencies, etc) with
random sampling?
>>> Don't understand what you mean by this.
>>>
>>> cheers,
>>>
>>> Rolf Turner
>> ===================>> Michael M. Fuller, Ph.D.
>> Department of Biology
>> University of New Mexico
>> Albuquerque, NM
>> EMAIL: mmfuller at unm.edu
>> WEB: biology.unm.edu/mmfuller
>>
>>
>>
>>
>>
>
===================Michael M. Fuller, Ph.D.
Department of Biology
University of New Mexico
Albuquerque, NM
EMAIL: mmfuller at unm.edu
WEB: biology.unm.edu/mmfuller