Hello, I'm attempting to evaluate the accuracy of the probability predictions for my model. As previously discussed here, the AUC is not a good measure as I'm not concerned with classification accuracy but probability accurcy. It was suggested to me that the loess function would be a good measure to look at. I can see some libraries (Design) will plot the loess function as a curve of the resulting model. That's nice to look at, but I need a way to feed in a test set of data and measure the accuracy of my predicted probabilities. I read the help page for loess and it looks like a learner of its own. I already have a model trained with SVM and can apply the test data for a result of output that is predicted probabilities and true labels. How can I feed this to the loess function to both see a curve and get the mean absolute error among other measures. Ultimately what I am trying to do is develop a model with high accuracy of predicted probabilities. I'm testing many different learning functions, kernels, parameters, etc. I'mm looking of a single "performance measure" that summarized the probability accuracy for a given model. That way I can track my experiment's results and pick the best one. Any suggestions along these lines would be greatly appreciated. Thank You, -Noah
Hello, I'm attempting to evaluate the accuracy of the probability predictions for my model. As previously discussed here, the AUC is not a good measure as I'm not concerned with classification accuracy but probability accurcy. It was suggested to me that the loess function would be a good measure to look at. I can see some libraries (Design) will plot the loess function as a curve of the resulting model. That's nice to look at, but I need a way to feed in a test set of data and measure the accuracy of my predicted probabilities. I read the help page for loess and it looks like a learner of its own. I already have a model trained with SVM and can apply the test data for a result of output that is predicted probabilities and true labels. How can I feed this to the loess function to both see a curve and get the mean absolute error among other measures. Ultimately what I am trying to do is develop a model with high accuracy of predicted probabilities. I'm testing many different learning functions, kernels, parameters, etc. I'mm looking of a single "performance measure" that summarized the probability accuracy for a given model. That way I can track my experiment's results and pick the best one. Any suggestions along these lines would be greatly appreciated. Thank You, -Noah