R help - Aug 2006 - logistic discrimination: which chance performance??

Hello,
I am using logistic discriminant analysis to check whether a known 
classification Yobs can be predicted by few continuous variables X.

What I do is to predict class probabilities with multinom() in nnet(), 
obtaining a predicted classification Ypred and then compute the percentage 
P(obs) of objects classified the same in Yobs and Ypred.

My problem now is to figure out whether P(obs) is significantly higher than 
chance.

I opted for a crude permutation approach: compute P(perm) over 10000 random 
permutations of Yobs (i.e., refit the multinom() model 10000 times randomly 
permuting Yobs) and consider P(obs) as significantly higher than chance if 
higher than the 95th percentile of the P(perm) distribution.

Now, the problem is that the mode of P(perm) is always really close to 
P(obs), e.g., if P(obs)=1 (perfect discrimination) also the most likely 
P(perm) value is 1!!!

I figured out that this is due to the fact that, with my data, randomly 
permuted classifications are highly likely to strongly agree with the 
observed classification Yobs, but, probably since my machine learning 
background is almost 0, I am kind of lost about how to proceed at this 
point.

I would greatly appreciate a comment on this.

Thanks
    Bruno

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Bruno L. Giordano, Ph.D.
CIRMMT
Schulich School of Music, McGill University
555 Sherbrooke Street West
Montr?al, QC H3A 1E3
Canada
http://www.music.mcgill.ca/~bruno/

Well,
If posting a possible solution to one's own problem is not part of the
netiquette of this list please correct me.

Following Titus et al. (1984) one might use Cohen's kappa to have a
chance-corrected measure of agreement between the original and reproduced
classification:

Kappa() in library vcd
kappa2() in library irr
ckappa() in library psy
cohen.kappa() in library concord......

    Bruno

Kimberly Titus; James A. Mosher; Byron K. Williams (1984), Chance-corrected 
Classification for Use in Discriminant Analysis: Ecological Applications, 
American Midland Naturalist, 111(1),1-7.


----- Original Message ----- 
From: "Bruno L. Giordano" <bruno.giordano at music.mcgill.ca>
To: <r-help at stat.math.ethz.ch>
Sent: Thursday, August 10, 2006 6:18 PM
Subject: [R] logistic discrimination: which chance performance??

> Hello,
> I am using logistic discriminant analysis to check whether a known
> classification Yobs can be predicted by few continuous variables X.
>
> What I do is to predict class probabilities with multinom() in nnet(),
> obtaining a predicted classification Ypred and then compute the percentage
> P(obs) of objects classified the same in Yobs and Ypred.
>
> My problem now is to figure out whether P(obs) is significantly higher
> than
> chance.
>
> I opted for a crude permutation approach: compute P(perm) over 10000
> random
> permutations of Yobs (i.e., refit the multinom() model 10000 times
> randomly
> permuting Yobs) and consider P(obs) as significantly higher than chance if
> higher than the 95th percentile of the P(perm) distribution.
>
> Now, the problem is that the mode of P(perm) is always really close to
> P(obs), e.g., if P(obs)=1 (perfect discrimination) also the most likely
> P(perm) value is 1!!!
>
> I figured out that this is due to the fact that, with my data, randomly
> permuted classifications are highly likely to strongly agree with the
> observed classification Yobs, but, probably since my machine learning
> background is almost 0, I am kind of lost about how to proceed at this
> point.
>
> I would greatly appreciate a comment on this.
>
> Thanks
>    Bruno
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Bruno L. Giordano, Ph.D.
> CIRMMT
> Schulich School of Music, McGill University
> 555 Sherbrooke Street West
> Montr?al, QC H3A 1E3
> Canada
> http://www.music.mcgill.ca/~bruno/
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>