Hi. I started with a file which was a sparse 982x923 matrix and where the
last column was a variable to be predicted. I did principle component
analysis on it and arrived at a new 982x923 matrix.
Then I ran the code below to get a neural network using nnet and then wanted
to get a confusion matrix or at least know how accurate the neural net was.
I used the first 22 principle components only for the inputs for the neural
net.
I got a perfect prediction rate which is somewhat suspect ( I was using the
same data for training and prediction but I did not expect perfect
prediction anyway). So I tried using only a sample of records to build the
neural net.
Even with this sample I got 980 out of 982 correct. Can anyone spot an error
here?
crs$dataset <-
read.csv("file:///C:/dataForR/textsTweet1/cleanForPC.csv",
na.strings=c(".", "NA", "", "?"))
crs$nnet <- nnet(Value ~ ., data=crs$dataset[,c(1:22,922)], size=10,
linout=TRUE, skip=TRUE, trace=FALSE, maxit=1000)
targets=crs$dataset[,922]
rawpredictions =predict(crs$nnet, crs$dataset[, c(1:22)], type="raw")
roundedpredictions=round(rawpredictions[,1],digits = 0)
trueAndPredicted=cbind(roundedpredictions, targets)
howManyEqual=trueAndPredicted[,1]==trueAndPredicted[,2]
sum(howManyEqual)
samp <- c(sample(1:50,25), sample(51:100,25), sample(101:150,25))
samp <- c(sample(1:250,125), sample(251:500,125), sample(500:920,300))
crs$nnet <- nnet(Value ~ ., data=crs$dataset[samp,c(1:22,922)], size=10,
linout=TRUE, skip=TRUE, trace=FALSE, maxit=1000)
--
View this message in context:
http://www.nabble.com/accuracy-of-a-neural-net-tp23692699p23692699.html
Sent from the R help mailing list archive at Nabble.com.
You might want to use cross-validation or the bootstrap to get error estimates. Also, you should include the PCA step in the resampling since it does add noise to the model. Look at the pcaNNet and train functions in the caret package. Also your code for the nnet would imply that you are predicting a continuous outcome (i.e. linear function on the hidden units), so a confusion matrix wouldn't be appropriate. Max On Sun, May 24, 2009 at 7:28 AM, onyourmark <william108 at gmail.com> wrote:> > Hi. I started with a file which was a sparse 982x923 matrix and where the > last column was a variable to be predicted. I did principle component > analysis on it and arrived at a new 982x923 matrix. > Then I ran the code below to get a neural network using nnet and then wanted > to get a confusion matrix or at least know how accurate the neural net was. > I used the first 22 principle components only for the inputs for the neural > net. > I got a perfect prediction rate which is somewhat suspect ( I was using the > same data for training and prediction but I did not expect perfect > prediction anyway). So I tried using only a sample of records to build the > neural net. > Even with this sample I got 980 out of 982 correct. Can anyone spot an error > here? > > crs$dataset <- read.csv("file:///C:/dataForR/textsTweet1/cleanForPC.csv", > na.strings=c(".", "NA", "", "?")) > crs$nnet <- nnet(Value ~ ., data=crs$dataset[,c(1:22,922)], size=10, > linout=TRUE, skip=TRUE, trace=FALSE, maxit=1000) > > targets=crs$dataset[,922] > > rawpredictions =predict(crs$nnet, crs$dataset[, c(1:22)], type="raw") > > roundedpredictions=round(rawpredictions[,1],digits = 0) > > trueAndPredicted=cbind(roundedpredictions, targets) > > howManyEqual=trueAndPredicted[,1]==trueAndPredicted[,2] > sum(howManyEqual) > > > samp <- c(sample(1:50,25), sample(51:100,25), sample(101:150,25)) > samp <- c(sample(1:250,125), sample(251:500,125), sample(500:920,300)) > > crs$nnet <- nnet(Value ~ ., data=crs$dataset[samp,c(1:22,922)], size=10, > linout=TRUE, skip=TRUE, trace=FALSE, maxit=1000) >-- Max
It looks promising. I saw the pdf at:http://cran.r-project.org/web/packages/caret/vignettes/caretMisc.pdf. I will give it a try. Thank you. onyourmark wrote:> > Hi. I started with a file which was a sparse 982x923 matrix and where the > last column was a variable to be predicted. I did principle component > analysis on it and arrived at a new 982x923 matrix. > Then I ran the code below to get a neural network using nnet and then > wanted to get a confusion matrix or at least know how accurate the neural > net was. I used the first 22 principle components only for the inputs for > the neural net. > I got a perfect prediction rate which is somewhat suspect ( I was using > the same data for training and prediction but I did not expect perfect > prediction anyway). So I tried using only a sample of records to build the > neural net. > Even with this sample I got 980 out of 982 correct. Can anyone spot an > error here? > > crs$dataset <- read.csv("file:///C:/dataForR/textsTweet1/cleanForPC.csv", > na.strings=c(".", "NA", "", "?")) > crs$nnet <- nnet(Value ~ ., data=crs$dataset[,c(1:22,922)], size=10, > linout=TRUE, skip=TRUE, trace=FALSE, maxit=1000) > > targets=crs$dataset[,922] > > rawpredictions =predict(crs$nnet, crs$dataset[, c(1:22)], type="raw") > > roundedpredictions=round(rawpredictions[,1],digits = 0) > > trueAndPredicted=cbind(roundedpredictions, targets) > > howManyEqual=trueAndPredicted[,1]==trueAndPredicted[,2] > sum(howManyEqual) > > > samp <- c(sample(1:50,25), sample(51:100,25), sample(101:150,25)) > samp <- c(sample(1:250,125), sample(251:500,125), sample(500:920,300)) > > crs$nnet <- nnet(Value ~ ., data=crs$dataset[samp,c(1:22,922)], size=10, > linout=TRUE, skip=TRUE, trace=FALSE, maxit=1000) > >-- View this message in context: http://www.nabble.com/accuracy-of-a-neural-net-tp23692699p23706530.html Sent from the R help mailing list archive at Nabble.com.