R help - Aug 2010 - ROCR predictions

Hi everybody,

I am having a problem building a ROC curve with my data using the ROCR
package.

I have 10 lists of proteins such as attached (proteinlist.xls). each of the
lists was calculated with a different p-value.
The goal is to find the optimal p-value for the highest number of true
positives as well as lowaest number of false positives.

As far as I understood the explanations from the vignette of ROCR, my data
of TP and FP are the labels of the prediction function. But I don't know how
to assign the right predictions to these labels.

BTW, Is there a way of finding the optimum in the curve? I mean to find the
exact value in the ROC curve (see sheet 2 in the excel file for the ROC
curve).

I would like to thank for any help in advance

Assa

Dear Assa,
> 
> I am having a problem building a ROC curve with my data using the ROCR
> package.
> 
> I have 10 lists of proteins such as attached (proteinlist.xls). each of the
your file didn't make it to the list.

> lists was calculated with a different p-value.
> The goal is to find the optimal p-value for the highest number of true
> positives as well as lowaest number of false positives.
> As far as I understood the explanations from the vignette of ROCR, my data
> of TP and FP are the labels of the prediction function. But I don't
know how
> to assign the right predictions to these labels.
I assume the p-values are different cutoffs that you use for
"hardening" (=
making yes/no predictions) from some soft (= continuous class membership) output
of your classifier.

Usually, ROCR calculates the curves as function of the cutoff/threshold itself 
from the continuos predictions. If you have these soft predictions, let ROCR do 
the calculation for you.

If you don't have them, ROCR can calculate your characteristics (sens, spec,
precision, recall, whatever) for each of the p-values. While you could combine 
the results "by hand" into a ROCR-performance object and let ROCR do
the
plotting, it is then probably easier if you plot directly yourself.

Don't be shy to look into the prediction and performance objects, I find
them
pretty obvious. Maybe start with the objects produced by the examples.

Also, note ROCR works with binary validation data only. If your data has more 
than one class, you need to make two-class-problems first (e.g. protein xy ./. 
not protein xy).

> BTW, Is there a way of finding the optimum in the curve? I mean to find the
> exact value in the ROC curve (see sheet 2 in the excel file for the ROC
> curve).
Someone asked for optimum on ROC a couple of months ago, RSiteSearch on the 
mailing list with ROC and optimal or optimum should get you answers.

> I would like to thank for any help in advance
You're welcome.

Claudia

-- 
Claudia Beleites
Dipartimento dei Materiali e delle Risorse Naturali
Universit? degli Studi di Trieste
Via Alfonso Valerio 6/a
I-34127 Trieste

phone: +39 0 40 5 58-37 68
email: cbeleites at units.it