R help - Mar 2009 - OT: Likelihood ratio for the randomization/permutation test?

Hi guRus,

My discipline (experimental psychology) is gradually moving away from
Null Hypothesis Testing and towards measures of evidence. One measure
of evidence that has been popular of late is the likelihood ratio.
Glover & Dixon (2005) demonstrate the calculation of the likelihood
ratio from ANOVA tables, but I'm also interested in non-parametric
statistics and wonder if anyone has any ideas on how to compute a
likelihood ratio from a randomization test (aka. permutation test)?

Say one had two groups and were interested in whether the mean scores
of the two groups differ in a manner consistent with random chance or
in a manner consistent with a non-null effect of some manipulation
applied to the two groups. The randomization test addresses this by
randomly re-assigning the participants to the groups, re-computing the
difference between means, and repeating many times, yielding a
distribution of simulated difference scores that represents the
distribution expected by chance.

Within a Null Hypothesis Testing framework you then estimate the
probability of the null by observing the proportion of simulated
difference scores that are greater in magnitude than the observed
difference score. Any guesses on how to translate this into a
quantification of evidence?

Mike

-- 
Mike Lawrence
Graduate Student
Department of Psychology
Dalhousie University

Looking to arrange a meeting? Check my public calendar:
http://tinyurl.com/mikes-public-calendar

~ Certainty is folly... I think. ~

On Wed, 11 Mar 2009, Mike Lawrence wrote:
> Hi guRus,
>
> My discipline (experimental psychology) is gradually moving away from
> Null Hypothesis Testing and towards measures of evidence. One measure
> of evidence that has been popular of late is the likelihood ratio.
> Glover & Dixon (2005) demonstrate the calculation of the likelihood
> ratio from ANOVA tables, but I'm also interested in non-parametric
> statistics and wonder if anyone has any ideas on how to compute a
> likelihood ratio from a randomization test (aka. permutation test)?
>
You cannot get the likelihood ratio from just the null, you need an 
alternative. The alternative would have to provide different probabilities 
to the individual permutations than under the null I guess, so if you have 
a framework where this makes sense you are in business.

I suspect you might be aiming in the direction of "empirical
likelihood"
for which there is a literature - Google 'empirical likelihood'.

Also to turn this back to R, check out 'emplik' on CRAN.

HTH,

Chuck

> Say one had two groups and were interested in whether the mean scores
> of the two groups differ in a manner consistent with random chance or
> in a manner consistent with a non-null effect of some manipulation
> applied to the two groups. The randomization test addresses this by
> randomly re-assigning the participants to the groups, re-computing the
> difference between means, and repeating many times, yielding a
> distribution of simulated difference scores that represents the
> distribution expected by chance.
>
> Within a Null Hypothesis Testing framework you then estimate the
> probability of the null by observing the proportion of simulated
> difference scores that are greater in magnitude than the observed
> difference score. Any guesses on how to translate this into a
> quantification of evidence?
>
> Mike
>
> -- 
> Mike Lawrence
> Graduate Student
> Department of Psychology
> Dalhousie University
>
> Looking to arrange a meeting? Check my public calendar:
> http://tinyurl.com/mikes-public-calendar
>
> ~ Certainty is folly... I think. ~
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
Charles C. Berry                            (858) 534-2098
                                             Dept of Family/Preventive Medicine
E mailto:cberry at tajo.ucsd.edu	            UC San Diego
http://famprevmed.ucsd.edu/faculty/cberry/  La Jolla, San Diego 92093-0901