similar to: Can ROC be used as a metric for optimal model selection for randomForest?

The following code fails to train a nnet model in a random dataset using caret: nR <- 700 nCol <- 2000 myCtrl <- trainControl(method="cv", number=3, preProcOptions=NULL, classProbs = TRUE, summaryFunction = twoClassSummary) trX <- data.frame(replicate(nR, rnorm(nCol))) trY <- runif(1)*trX[,1]*trX[,2]^2+runif(1)*trX[,3]/trX[,4] trY <-

I'm using caret to assess classifier performance (and it's great!). However, I've found that my results differ between R2.* and R3.* - reported accuracies are reduced dramatically. I suspect that a code change to kernlab ksvm may be responsible (see version 5.16-24 here: http://cran.r-project.org/web/packages/caret/news.html). I get very different results between caret_5.15-61 +

I'm dealing with classification problems, and I'm trying to specify a custom scoring metric (recall at p, ROC, etc.) that depends on not just the class output but the probability estimates, so that caret::train can choose the optimal tuning parameters based on this metric. However, when I supply a trainControl summaryFunction, the data given to it contains only class predictions, so the

Guys, I used Random Forest with a couple of data sets I had to predict for binary response. In all the cases, the AUC of the training set is coming to be 1. Is this always the case with random forests? Can someone please clarify this? I have given a simple example, first using logistic regression and then using random forests to explain the problem. AUC of the random forest is coming out to be

Here is a snippet to show what i'm trying to do. library(randomForest) library(ROCR) library(caret) data(iris) iris <- iris[(iris$Species != "setosa"),] fit <- randomForest(factor(Species) ~ ., data=iris, ntree=50) train.predict <- predict(fit,iris,type="prob")[,2]

I have a binary classification problem where the fraction of positives is very low, e.g. 20 positives in 10,000 examples (0.2%) What is an appropriate cross validation scheme for training a classifier with very few positives? I currently have the following setup: ======================================== library(caret) tmp <- createDataPartition(Y, p = 9/10, times = 3, list = TRUE)

I am used to packages like e1071 where you have a tune step and then pass your tunings to train. It seems with caret, tuning and training are both handled by train. I am using train and trainControl to find my hyper parameters like so: MyTrainControl=trainControl( method = "cv", number=5, returnResamp = "all", classProbs = TRUE ) rbfSVM <- train(label~., data =

Dear developers, I have jutst started working with caret and all the nice features it offers. But I just encountered a problem: I am working with a dataset that include 4 predictor variables in Descr and a two-category outcome in Categ (codified as a factor). Everything was working fine I got the results, confussion matrix etc. BUT for obtaining the AUC and predicted probabilities I had to add

Hi all, I've been using a custom summary function to optimise regression model methods using the caret package. This has worked smoothly. I've been using the default bootstrapping resampling method. For bagging models (specifically randomForest in this case) caret can, in theory, uses the out-of-bag (oob) error estimate from the model instead of resampling, which (in theory) is largely

Dear R-helpers: I want to calculate area under a Receiver Operator Characteristic curve. Where can I find related functions? Thank you in advance Xiao

Hi Sacha, On second thought, perhaps this is more the direction that you want ... X2 = cbind(X_train,y_train) colnames(X2)[3] = "y" regr2<-randomForest(y~x1+x2, data=X2,maxnodes=10, ntree=10) regr regr2 #Make prediction predictions= predict(regr, X_test) predictions2= predict(regr2, X_test) HTH, Eric On Tue, May 9, 2023 at 6:40?AM Eric Berger <ericjberger at gmail.com>

I have never been able to get class probabilities to work and I am relatively new to using these tools, and I am looking for some insight as to what may be wrong. I am using caret with kernlab/ksvm. I will simplify my problem to a basic data set which produces the same problem. I have read the caret vignettes as well as documentation for ?train. I appreciate any direction you can give. I

Hi all, I have tried a 5 fold cross validation using caret package with random forest method on iris dataset as example. Then I need ROC curve for each fold: > set.seed(1) > train_control <- trainControl(method="cv", number=5,savePredictions = TRUE,classProbs = TRUE) > output <- train(Species~., data=iris, trControl=train_control, method="rf") >

Hello all, I would like to use the train function from the caret package to train a svm with a spectral kernel from the kernlab package. Sadly a svm with spectral kernel is not among the many methods in caret... using caret to train svmRadial: ------------------ library(caret) library(kernlab) data(iris) TrainData<- iris[,1:4] TrainClasses<- iris[,5] set.seed(2)

Vuelvo al ataque, pues ya os lo pregunté. Javier Rubén me dió una posible explicación, pero finalmente no me resolvió el problema. Así que lo vuelvo a intentar, a ver si hay más suerte. Si hago, p.e., i = 1 y corro las 2 filas de dentro del loop que pongo abajo, me abre una ventana y me hace el partial() de frg, es decir, lo hace bien, pero si corro todo el loop, me abre las 9 ventanas (de 9

Hi list, I´m working on a predictive modeling task using the caret package. I found the best model parameters using the train() and trainControl() command. Now I want to evaluate my model and make predictions on a test dataset. I tried to follow the instructions in the manual and the vignettes but unfortunately I´m getting an error message I can`t figure out. Here is my code: rfControl <-

Si, Carlos. Yo hago lo mismo, pero esos mismos numeritos salen enormes. > treesize(RFfit) [1] 4304 4302 4311 4319 4343 4298 4298 4311 4349 4327 4331 4317 4294 4321 4283 4362 [17] 4300 4330 4266 4331 4308 4352 4294 4315 4372 4349 4331 4347 4329 4348 4298 4335 [33] 4346 4396 4345 4313 4293 4276 4353 4272 4304 4325 4317 4336 4308 4351 4374 4324 [49] 4386 4359 4311 4346 4300

Gracias Carlos y Javier, ntrees es el nº de árboles y treesize sus respectivos tamaños (nº de nodos) ntree: Number of trees to grow. This should not be set to too small ...... treesize: Size of trees (number of nodes) in and ensemble. Puse 1000 árboles (ntree=1000), si, pero la función treesize te da el nº de nodos: treesize(RFfit, terminal=TRUE) me da un vector de 1000 elementos (uno

Muchas gracias Carlos, como siempre. Es raro que se me pasase. En su momento miré todos los argumentos del RF, como hago siempre, pero ese lo había olvidado. La verdad es que funcionaba estupendamente, pero me parecía extraño. Aunque dado que los RF no sobreajustan, no hay problema con que sus árboles sean todo lo grandes que quieras. Lo he testado con una base de datos externa y explica

Version 4.63 of the caret package is now on CRAN. caret can be used to tune the parameters of predictive models using resampling, estimate variable importance and visualize the results. There are also various modeling and "helper" functions that can be useful for training models. caret has wrappers to over 99 different models for classification and regression. See the package vignettes