I was wandering if anybody ever tried to compare the classification accuracy of nnet to other (rpart, tree, bagging) models. From what I know, there is no reason to expect a significant difference in classification accuracy between these models, yet in my particular case I get about 10% error rate for tree, rpart and bagging model and 80% error rate for nnet, applied to the same data. Thanks.
I think that you are using nnet incorrectly. I have compared several classifiers (including that ones that you mention in your e-mail) on the same dataset and I have never found more of a 20% of difference in the missclassification error. Of course, I estimated the misclassification error by cross validation. Regards Edgar Acuna UPR-MATH On Sat, 13 Mar 2004, Albedo wrote:> I was wandering if anybody ever tried to compare the classification > accuracy of nnet to other (rpart, tree, bagging) models. From what I > know, there is no reason to expect a significant difference in > classification accuracy between these models, yet in my particular case > I get about 10% error rate for tree, rpart and bagging model and 80% > error rate for nnet, applied to the same data. > > Thanks. > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://www.stat.math.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html >
The only thing that I could have done wrong with nnet (that I can think of) is not enough nuerons in hidden layer, but then again this is actually limited by my computer memory. However, I did estimate the error a little bit different - I have enough data for test set, which I used for classification accuracy estimation only. Edgar Acuna <edgar at cs.uprm.edu>:> I think that you are using nnet incorrectly. I havecompared several> classifiers (including that ones that you mention inyour e-mail) on the> same dataset and I have never found more of a 20% ofdifference in the> missclassification error. Of course, I estimated themisclassification> error by cross validation. > > Regards > Edgar Acuna > UPR-MATH > > On Sat, 13 Mar 2004, Albedo wrote: > > > I was wandering if anybody ever tried to comparethe classification> > accuracy of nnet to other (rpart, tree, bagging)models. From what I> > know, there is no reason to expect a significantdifference in> > classification accuracy between these models, yetin my particular case> > I get about 10% error rate for tree, rpart andbagging model and 80%> > error rate for nnet, applied to the same data. > > > > Thanks. > > > > ______________________________________________ > > R-help at stat.math.ethz.ch mailing list > > https://www.stat.math.ethz.ch/mailman/listinfo/r-help > > PLEASE do read the posting guide! > http://www.R-project.org/posting-guide.html > > > >
My experience is that nnet needs a lot of tuning, not only in terms of numbers of layers, but also in terms of the other parameters. My first results where I kept very much of the default parameter values with nnet have been very bad, as bad as you say. (But as Brian Ripley already wrote, it's not straight forward to say via the net how to do it better.) Apart from that, such a large difference of classification accuracy between different methods is strange, but possible in principle. Very different structures of data exist (which means again that nobody can assess your problem without knowing the data). Christian On Sat, 13 Mar 2004, Albedo wrote:> I was wandering if anybody ever tried to compare the classification > accuracy of nnet to other (rpart, tree, bagging) models. From what I > know, there is no reason to expect a significant difference in > classification accuracy between these models, yet in my particular case > I get about 10% error rate for tree, rpart and bagging model and 80% > error rate for nnet, applied to the same data. > > Thanks. > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://www.stat.math.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html >*********************************************************************** Christian Hennig Fachbereich Mathematik-SPST/ZMS, Universitaet Hamburg hennig at math.uni-hamburg.de, http://www.math.uni-hamburg.de/home/hennig/ ####################################################################### ich empfehle www.boag-online.de
I believe the paper below did a comparison relevant to your question but possibly not exactly what you are asking (I'm not sure that nnet is artificial neural networks - on of the techniques compared in the paper). Hope this helps Volker Moisen G. G., and T. S. Frescino. 2002. Comparing five modelling techniques for predicting forest characteristics. Ecological Modelling 157(2-3):209-225. _______________________________ Volker Bahn Dept. of Wildlife Ecology - Rm. 210 University of Maine 5755 Nutting Hall Orono, Maine 04469-5755, USA Tel. (207) 581 2799 Fax: (207) 581 2858 volker.bahn at umit.maine.edu http://www.wle.umaine.edu/used_text%20files/Volker%20Bahn/home.htm ----- Original Message ----- From: "Albedo" <albedo at pisem.net> To: <r-help at stat.math.ethz.ch>; <s-news at lists.biostat.wustl.edu> Sent: Saturday, March 13, 2004 16:06 Subject: nnet classification accuracy vs. other models | I was wandering if anybody ever tried to compare the classification | accuracy of nnet to other (rpart, tree, bagging) models. From what I | know, there is no reason to expect a significant difference in | classification accuracy between these models, yet in my particular case | I get about 10% error rate for tree, rpart and bagging model and 80% | error rate for nnet, applied to the same data. | | Thanks. |