Dear all,
Not a problem that is very specific to R, but I think that it is not only of
interest to me... So I hope that someone finds the time to provide me with some
clues on this probably rather basic issue.
I'm performing an analysis of experimental data with a categorical response
variable. Until now I have been using a classical maximum likelihood approach.
The most basic output measure to evaluate the models is thus minus twice the log
likelihood, that is, the deviance. I am evaluating the statistical significance
of effects by calculating the absolute difference between deviances (for nested
models), which should be approximately chi-square distributed under the null
hypothesis - that is the classical Wilks' test. I use bootstrapping
sometimes where I suspect that the approximation is not good. But for the rest,
the philosophy of the test is the same.I'm quite sure that the magnitude of
effect of the experimental manipulations is highly subject-dependent. I have, in
this specific case, 7 subjects; therefore, I perform and report 7 separate
analyses. The problem afterwards lies in the model selection. Sometimes, the
best-fitting model (either based on all-possible subset fits or a sequential
procedure, with p-values, AIC or BIC based) for subject A is X1+X2, for subject
B X1+X3, for C X1+X1*X2 etc. You can imaginethat this turns into a mess. I
thought that evaluation of the fits of competing models could be based on a
likelihood statistic that is jointly determined for all subjects, namely by
aggregating (adding) deviances across subjects. In theory, if I can say that
within-subject -2 log likelihood ratios to test for effects are asymptotically
chi-square distributed, the sum should also be, with degrees of freedom added
together. So this way I'm actually doing a kind of meta-analysisby adding
deviances.(1) How defendable is this procedure? Strangely enough (or maybe
not...), I didn't find anything written about a procedure like this. This
leads me to believe I must be doing something very wrong... The problem, I
suppose, is that I'm treating effects that are actually random, as fixed.
But what are the consequences of this? Does it mean that I am making inferences
about the specific sample that I have used in the experiment, that are possibly
not generalizeable across the population, or is the problem more serious?(2) In
the kind of situation I am facing, I often see applications of generalized
linear mixed model, empirical Bayes, and the like. Somehow I think that a
hierarchical Bayes model would be overkill for this. Hypothesis tests would turn
into confidence regions. It might be computationally unfeasible anyway because
in some of my alternative models, levels of the experimental variable are
treated in a categorical way, generating a lot of free parameters that are
subject to different random distributions. I'm still getting acquainted with
this specific technique, so I count on your patience in case this is a stupid
question: If maximum-likelihood estimation is as Bayesian inference with a flat
prior on the parameters, is the meta-analysis I am trying out here then
equivalent to empirical Bayes with flat priors for the random effects?
_________________________________________________________________
Hoe herken je de vrienden van je vrienden?
[[alternative HTML version deleted]]
