similar to: ROCR crashes for simple recall plot

If we use the ROCR package to find the accuracy of a classifier pred <- prediction(svm.pred, testset[,2]) perf.acc <- performance(pred,"acc") Do we?find the maximum accuracy?as follows?(is there a simplier way?): > max(perf.acc at x.values[[1]]) Then to find the cutoff point that maximizes the accuracy?do we do the following?(is there a simpler way): > cutoff.list <-

Dear List,   I am new to ROC analysis and the package ROCR. I want to compute the confidence intervals of sensitivity and specificity for a given cutoff value. I have used the following to calculate sensitivity and specificity:   data(ROCR.simple) pred <- prediction(ROCR.simple$predictions, ROCR.simple$labels)   se.sp <- function (cutoff, performance) {     sens <-

All, I'm trying again with a slightly more generic version of my first question. I can extract the plotted values from hist(), boxplot(), and even plot.randomForest(). Observe: # get some data dat <- rnorm(100) # grab histogram data hdat <- hist(dat) hdat #provides details of the hist output #grab boxplot data bdat <- boxplot(dat) bdat #provides details of the boxplot

Hi, My questions concerns the ROCR package and I hope somebody here on the list can help - or point me to some better place. When evaluating a model's performane, like this: pred1 <- predict(model, ..., type="response") pred2 <- prediction(pred1, binary_classifier_vector) perf <- performance(pred, "sens", "spec") (Where "prediction" and

Hi, I have an object from the prediction function from the ROCR package and I would like to extract one of the slots from the object, for example the cutoffs slot. However the usual techniques ($, [["name"]]) of subsetting don't work. How can I assess the lists in the slots? Here is an example of what I am working with: library(ROCR) data(ROCR.simple) pred <-

Dear list I'm testing a predictor and I produced nice performance plots with ROCR package utilizing the 3 standard command pred <- prediction(predictions, labels) perf <- performance(pred, measure = "tpr", x.measure = "fpr") plot(perf, col=rainbow(10)) The pred object and the perfo object are S4 with the following slots An object of class "performance"

I'm dealing with classification problems, and I'm trying to specify a custom scoring metric (recall at p, ROC, etc.) that depends on not just the class output but the probability estimates, so that caret::train can choose the optimal tuning parameters based on this metric. However, when I supply a trainControl summaryFunction, the data given to it contains only class predictions, so the

Hi everybody, I am having a problem building a ROC curve with my data using the ROCR package. I have 10 lists of proteins such as attached (proteinlist.xls). each of the lists was calculated with a different p-value. The goal is to find the optimal p-value for the highest number of true positives as well as lowaest number of false positives. As far as I understood the explanations from the

Hello All,  I have used logistic regression glm in R and I am evaluating two models both learned with glm but with different predictors. model1 <- glm (Y ~ x4+ x5+ x6+ x7, data = dat, family = binomial(link=logit))model2 <- glm (Y~ x1 + x2 +x3 , data = dat, family = binomial(link=logit))  and I would like to compare these two models based on the prediction that I get from each model: pred1 =

I am trying to use package ROCR to analyze classification accuracy, unfortunately there are some problems right at the beginning. Question 1) When I try to run demo I am getting the following error message > library(ROCR) > demo(ROCR) > if(dev.cur() <= 1) .... [TRUNCATED] Error in get(getOption("device")) : wrong first argument When I issue the command > dev.cur() it

Hi, I am using logistic regression to classify a binary psychometric data. using glm() and then predict.glm() i got the predicted odds ratio of the testing data. Next i am going to plot ROC curve for the analysis of my study. Now what i will do: 1. first select a cut-off (say 0.4) and classify the output of predict.glm() into {0,1} segment and then use it to draw ROC curve using ROCR package

Usually when using raw glmnet I let the implementation choose the lambdas. However when training via caret::train the lambda values are predetermined. Is there any way to have caret defer the lambda choices to caret::train and thus choose the optimal lambda dynamically? -- Yang Zhang http://yz.mit.edu/

I use ROCR to plot multiple runs' performance. Using the sample code as example: # plot ROC curves for several cross-validation runs (dotted # in grey), overlaid by the vertical average curve and boxplots # showing the vertical spread around the average. data(ROCR.xval) pred <- prediction(ROCR.xval$predictions, ROCR.xval$labels) perf <- performance(pred,"tpr","fpr")

Hi, I don't really understand how ROCR works. Here's another example with a randomforest model: I have the training dataset(bank_training) and testing dataset(bank_testing) and I ran a randomForest as below: bankrf<-randomForest(y~., bank_training, mtry=4, ntree=2, keep.forest=TRUE,importance=TRUE) bankrf.pred<-predict(bankrf, bank_testing)

Hi, I'm involved in a bioinformatics project at my university, and we're doing a comparison paper between some methods of classification of nc-RNA. I've been encharged of ploting the ROC curves' graphs. But I'm new on working with R and I'm having some difficulty with the prediction-class. I don't get where the values of ROCR.simple$predictions, for example, came from

Is it possible to use ROCR to plot a simple recall at p plot? I.e., a plot where the x-axis is the position into the ranked test set, and the y-axis is the recall, so you can see what's the recall in the top 10% of the ranked results. I searched through the performance() manual but found nothing. (Not a cutoff-vs-recall graph, since the cutoff is the probability estimate returned by the

Hello list, I am trying to perform ROC analysis and count the AUC in order to validate my results. I use package ROCR. I would like to count the AUC not under the cutoff found by "performance" but to use another cutoff that I calculate. How could I change the following command in order to get what I want? perform=performance(pred,measure="auc",x.measure="cutoff"),

Dear List, I am trying to assess the prediction accuracy of an ordinal model fit with LRM in the Design package. I used predict.lrm to predict on an independent dataset and am now attempting to assess the accuracy of these predictions. >From what I have read, the AUC is good for this because it is threshold independent. I obtained the AUC for the fit model output from the c score (c =

Hello, I've come across a strange error... Here is what happens: model <- svm(traindata,trainlabels, type="C-classification", kernel="radial", cost=10, class.weights=c("win"=3,"lose"=1), scale=FALSE, probability = TRUE) predictions <- predict(model, traindata) pred <- prediction(predictions, trainlabels) This returns an error: Error in

Dear R-help and ROCR developers (Tobias Sing and Oliver Sander) - I think my first question is generic and could apply to many methods, which is why I'm directing this initially to R-help as well as Tobias and Oliver. Question 1. The plot function in ROCR will average your cross validation data if asked. I'd like to use that averaged data to find a "best" cutoff but I