R help - Feb 2012 - Custom caret metric based on prob-predictions/rankings

I'm dealing with classification problems, and I'm trying to specify a
custom scoring metric (recall at p, ROC, etc.) that depends on not just
the class output but the probability estimates, so that caret::train
can choose the optimal tuning parameters based on this metric.

However, when I supply a trainControl summaryFunction, the data given
to it contains only class predictions, so the only metrics possible
are things like accuracy, kappa, etc.

Is there any way to do this that I'm looking?  If not, could I put
this in as a feature request?  Thanks!

-- 
Yang Zhang
http://yz.mit.edu/

Oops, found trainControl's classProbs right after I sent!

On Thu, Feb 9, 2012 at 4:30 PM, Yang Zhang <yanghatespam at gmail.com>
wrote:
> I'm dealing with classification problems, and I'm trying to specify
a
> custom scoring metric (recall at p, ROC, etc.) that depends on not just
> the class output but the probability estimates, so that caret::train
> can choose the optimal tuning parameters based on this metric.
>
> However, when I supply a trainControl summaryFunction, the data given
> to it contains only class predictions, so the only metrics possible
> are things like accuracy, kappa, etc.
>
> Is there any way to do this that I'm looking? ?If not, could I put
> this in as a feature request? ?Thanks!
>
> --
> Yang Zhang
> http://yz.mit.edu/


-- 
Yang Zhang
http://yz.mit.edu/