R help - Jan 2004 - comparing classification methods: 10-fold cv or leaving-one-out ?

Hi
what would you recommend to compare classification methods such as LDA,
classification trees (rpart), bagging, SVM, etc:

10-fold cv (as in Ripley p. 346f)

or

leaving-one-out (as e.g. implemented in LDA)?

my data-set is not that huge (roughly 200 entries)

many thanks for a hint

Christoph
-- 
Christoph Lehmann <christoph.lehmann at gmx.ch>

Leave-one-out is very inaccurate for some methods, notably trees, but fine 
for some others (e.g. LDA) if used with a good measure of accuracy.

Hint: there is a very large literature on this, so read any good book on 
classification to find out what is known.

On Tue, 6 Jan 2004, Christoph Lehmann wrote:
> Hi
> what would you recommend to compare classification methods such as LDA,
> classification trees (rpart), bagging, SVM, etc:
> 
> 10-fold cv (as in Ripley p. 346f)
Not a valid reference:  did you mean Venables & Ripley (2000, p.346f)?
Try reading Ripley (1996), for example.
> or
> 
> leaving-one-out (as e.g. implemented in LDA)?
> 
> my data-set is not that huge (roughly 200 entries)
That's rather small to compare error rates on.

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

I would recommend reading the following:  Dietterich, T. G., (1998). 
Approximate Statistical Tests for Comparing Supervised Classification 
Learning Algorithms. Neural Computation, 10 (7) 1895-1924. 
http://web.engr.oregonstate.edu/~tgd/publications/index.html

The issues in comparing methods are subtle and difficult.  With such a 
small data set I would be a little surprised if you could get any result 
that are truly statistically significant, especially if your goal is to 
compare among good non-linear methods (i.e., in which there are unlikely to 
huge differences because of model misspecification).  However, because the 
issues are subtle, it is easy to get results that appear significant...

hope this helps,

Tony Plate

At Tuesday 04:31 PM 1/6/2004 +0100, Christoph Lehmann
wrote:
>Hi
>what would you recommend to compare classification methods such as LDA,
>classification trees (rpart), bagging, SVM, etc:
>
>10-fold cv (as in Ripley p. 346f)
>
>or
>
>leaving-one-out (as e.g. implemented in LDA)?
>
>my data-set is not that huge (roughly 200 entries)
>
>many thanks for a hint
>
>Christoph
>--
>Christoph Lehmann <christoph.lehmann at gmx.ch>
>
>______________________________________________
>R-help at stat.math.ethz.ch mailing list
>https://www.stat.math.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html
Tony Plate   tplate at acm.org