R help - Sep 2011 - Question about BIC of two different regression models? how should we compare two regression models?

Hi All, 
In order to compare two different logistic regressions, I think I need to
compare them based on their BIC values, but I am not sure if the smaller BIC
would mean a better model or the reverse is true?
Thanks a lot,Andra
	[[alternative HTML version deleted]]

Andra Isan <andra_isan <at> yahoo.com> writes:
> 
> Hi All,?
> In order to compare two different logistic regressions, 
> I think I need to compare them based on their BIC
> values, but I am not sure if the smaller BIC would mean a better 
> model or the reverse is true?
> Thanks a lot,Andra
  Smaller (i.e. lower value) BIC is always better 
(even if BIC happens to be negative, as can happen in some cases; 
i.e. BIC=-1002 is better than BIC=-1000, BIC=1000 is better than BIC=1002).

  I would suggest however that (a) there are better venues for this
question (e.g. stats.stackexchange.com), since it's a stats and not
an R question; (b) it might be a good idea to review a stats text,
or even http://en.wikipedia.org/wiki/Bayesian_information_criterion ,
since this is a pretty basic question.

On Fri, 2 Sep 2011, Patrick Breheny wrote:
> On 09/02/2011 08:48 AM, John Sorkin wrote:
>> I believe when using BIC one needs to compare nested models
>
> This is wrong.  Hypothesis tests rely on nested models; information
criteria
> do not.
Actually, this is off-topic on this list. But blanket statements are 
often themselves untrue: there are hypothesis tests of non-nested 
models (most famously due to Cox, 1961), and Akaike explicitly 
considered only nested models in his paper introducing AIC. 
Certainly criteria such as AIC and BIC (in the sense of Schwarz: there 
are several criteria of that name) can be used with non-nested models 
but are much sharper tools for nested models.
>
> -- 
> Patrick Breheny
> Assistant Professor
> Department of Biostatistics
> Department of Statistics
> University of Kentucky
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

Inline:

On Fri, Sep 2, 2011 at 8:09 AM, Patrick Breheny <patrick.breheny at
uky.edu> wrote:
> On 09/02/2011 08:48 AM, John Sorkin wrote:
>>
>> I believe when using BIC one needs to compare nested models
>
> This is wrong. ?Hypothesis tests rely on nested models; information
criteria
> do not.
>
Yes, indeed. It may additionally be worth noting what has has been
noted on this list before: the actual definition of such criteria is
given only up to a constant, so different **software** may give
different answers on the same data. Hence be sure to compare results
using the same software or make any necessary additive adjustments
based on the details of how the software does the calculation when
results from different software are being compared.

Cheers,
Bert
> --
> Patrick Breheny
> Assistant Professor
> Department of Biostatistics
> Department of Statistics
> University of Kentucky
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>


-- 
"Men by nature long to get on to the ultimate truths, and will often
be impatient with elementary studies or fight shy of them. If it were
possible to reach the ultimate truths without the elementary studies
usually prefixed to them, these would not be preparatory studies but
superfluous diversions."

-- Maimonides (1135-1204)

Bert Gunter
Genentech Nonclinical Biostatistics

On 09/02/2011 11:26 AM, Prof Brian Ripley wrote:
>> This is wrong.  Hypothesis tests rely on nested models; information
criteria
>> do not.
>
> Actually, this is off-topic on this list. But blanket statements are
> often themselves untrue: there are hypothesis tests of non-nested
> models (most famously due to Cox, 1961), and Akaike explicitly
> considered only nested models in his paper introducing AIC.
> Certainly criteria such as AIC and BIC (in the sense of Schwarz: there
> are several criteria of that name) can be used with non-nested models
> but are much sharper tools for nested models.
Good point; my remark was only meant to refer to the simple case of 
logistic regression in the original post, and certainly should not be 
construed as a blanket statement applying to all possible hypothesis 
tests of all possible models.

-- 
Patrick Breheny
Assistant Professor
Department of Biostatistics
Department of Statistics
University of Kentucky