similar to: randomforest and AUC using 10 fold CV - Plotting results

Dear all, I am using the "caret" Package for predictors selection with a randomForest model. The following is the train function: rfFit<- train(x=trainRatios, y=trainClass, method="rf", importance = TRUE, do.trace = 100, keep.inbag = TRUE, tuneGrid = grid, trControl=bootControl, scale = TRUE, metric = "ROC") I wanted to use ROC as the metric for variable

The following code fails to train a nnet model in a random dataset using caret: nR <- 700 nCol <- 2000 myCtrl <- trainControl(method="cv", number=3, preProcOptions=NULL, classProbs = TRUE, summaryFunction = twoClassSummary) trX <- data.frame(replicate(nR, rnorm(nCol))) trY <- runif(1)*trX[,1]*trX[,2]^2+runif(1)*trX[,3]/trX[,4] trY <-

Guys, I used Random Forest with a couple of data sets I had to predict for binary response. In all the cases, the AUC of the training set is coming to be 1. Is this always the case with random forests? Can someone please clarify this? I have given a simple example, first using logistic regression and then using random forests to explain the problem. AUC of the random forest is coming out to be

I have a binary classification problem where the fraction of positives is very low, e.g. 20 positives in 10,000 examples (0.2%) What is an appropriate cross validation scheme for training a classifier with very few positives? I currently have the following setup: ======================================== library(caret) tmp <- createDataPartition(Y, p = 9/10, times = 3, list = TRUE)

Dear R-experts, Here below a toy example with some error messages, especially at the end of the code (Tuning the parameters). Your help to correct my R code would be highly appreciated. ####################################### #libraries library(lattice) library(ggplot2) library(caret) library(randomForest) ?? #Data

Hi Sacha, On second thought, perhaps this is more the direction that you want ... X2 = cbind(X_train,y_train) colnames(X2)[3] = "y" regr2<-randomForest(y~x1+x2, data=X2,maxnodes=10, ntree=10) regr regr2 #Make prediction predictions= predict(regr, X_test) predictions2= predict(regr2, X_test) HTH, Eric On Tue, May 9, 2023 at 6:40?AM Eric Berger <ericjberger at gmail.com>

I am used to packages like e1071 where you have a tune step and then pass your tunings to train. It seems with caret, tuning and training are both handled by train. I am using train and trainControl to find my hyper parameters like so: MyTrainControl=trainControl( method = "cv", number=5, returnResamp = "all", classProbs = TRUE ) rbfSVM <- train(label~., data =

Hi, I am quite new to R and I have two questions regarding ROCR. 1. I have tried to understand how to extract area-under-curve value by looking at the ROCR document and googling. Still I am not sure if I am doing the right thing. Here is my code, is "auc1" the auc value? " pred1 <- prediction(resp1,label1) perf1 <- performance(pred1,"tpr","fpr") plot(

Dear all I am trying to compare the performances of several methods using the AUC0.1 and not the whole AUC. (meaning I wanted to compare to AUC's whose x axis only goes to 0.1 not 1) I came to know about the Mcneil Hanley test from Bernardo Rangel Tura and I referred to the original paper for the calculation of "r" which is an argument of the function cROC. I can only find the

I'm using caret to assess classifier performance (and it's great!). However, I've found that my results differ between R2.* and R3.* - reported accuracies are reduced dramatically. I suspect that a code change to kernlab ksvm may be responsible (see version 5.16-24 here: http://cran.r-project.org/web/packages/caret/news.html). I get very different results between caret_5.15-61 +

Version 4.63 of the caret package is now on CRAN. caret can be used to tune the parameters of predictive models using resampling, estimate variable importance and visualize the results. There are also various modeling and "helper" functions that can be useful for training models. caret has wrappers to over 99 different models for classification and regression. See the package vignettes

Version 4.63 of the caret package is now on CRAN. caret can be used to tune the parameters of predictive models using resampling, estimate variable importance and visualize the results. There are also various modeling and "helper" functions that can be useful for training models. caret has wrappers to over 99 different models for classification and regression. See the package vignettes

Dear developers, I have jutst started working with caret and all the nice features it offers. But I just encountered a problem: I am working with a dataset that include 4 predictor variables in Descr and a two-category outcome in Categ (codified as a factor). Everything was working fine I got the results, confussion matrix etc. BUT for obtaining the AUC and predicted probabilities I had to add

Hi, I am having problems passing arguments to method="gbm" using the train() function. I would like to train gbm using the laplace distribution or the quantile distribution. here is the code I used and the error: gbm.test <- train(x.enet, y.matrix[,7], method="gbm", distribution=list(name="quantile",alpha=0.5), verbose=FALSE,

Hi all, I have tried a 5 fold cross validation using caret package with random forest method on iris dataset as example. Then I need ROC curve for each fold: > set.seed(1) > train_control <- trainControl(method="cv", number=5,savePredictions = TRUE,classProbs = TRUE) > output <- train(Species~., data=iris, trControl=train_control, method="rf") >

I'm dealing with classification problems, and I'm trying to specify a custom scoring metric (recall at p, ROC, etc.) that depends on not just the class output but the probability estimates, so that caret::train can choose the optimal tuning parameters based on this metric. However, when I supply a trainControl summaryFunction, the data given to it contains only class predictions, so the

Dear all, I try to compare the performances of several parameters to diagnose lameness in dogs. I have several ROC curves from the same dataset. I plotted the ROC curves and calculated AUC with the ROCR package. I would like to compare the AUC. I used the following program I found on R-help archives : From: Bernardo Rangel Tura Date: Thu 16 Dec 2004 - 07:30:37 EST

I have never been able to get class probabilities to work and I am relatively new to using these tools, and I am looking for some insight as to what may be wrong. I am using caret with kernlab/ksvm. I will simplify my problem to a basic data set which produces the same problem. I have read the caret vignettes as well as documentation for ?train. I appreciate any direction you can give. I

I might be missing something but I thought that AUC was a measure for comparing ROC curves, so there is nothing else needed to "compare" them. The larger AUC is the higher correlation of 2 variables compared. No other measures or calculations are needed. Jarek Tuszynski -----Original Message----- From: r-help-bounces at stat.math.ethz.ch [mailto:r-help-bounces at stat.math.ethz.ch] On

Please help. I found the code in the archive. The author of this script says: "The first function (seROC) calculate the standard error of ROC curve, the second function (cROC) compare ROC curves." Can some one explain to me what are the na, nn and r parameters which are used as the input to the following two functions? Thanks much in advance. > From: Bernardo Rangel Tura >