Dear R Helpers
I am having difficulty understanding how to use the penalty matrix for the
nomROC function in package 'nonbinROC'.
The documentation says that the values of the penalty matrix code the
penalty function L[i,j] in which 0 <= L[i,j] <= 1 for
j > i. It gives an example that if we have an ordered response with 4
categories, then we might wish to penalise larger misclassifications more - so
there is (for example) 0 penalty for correct classifications, 0.25 penalty for
misclassifying by one category, 0.5 penalty for misclassifying by two categories
and 1.0 penalty for misclassifying by 3 categories. I wanted to use this sort of
penalty - but with equal distances between the 4 categories (0, 1/3, 2/3, 1).
But, I found that if I simply re-scale the penalty matrix, while maintaining
equal distances between categories, then the estimate of overall accuracy
increases. In effect I can achieve any value for accuracy - including unity - by
re-scaling the penalty matrix. So, I'd like to ask what, if any are the
contraints on the scaling process?
Here is a working code that illustrates my difficulty:-
set.seed(1); gldstd=round(runif(200)*4); gldstd[gldstd==0]=4; table(gldstd)
pred1=gldstd*rnorm(200, mean=1, sd=2/gldstd); boxplot(pred1~gldstd)
library(nonbinROC); gldstd=ordered(gldstd)
ordered_penalty = matrix(c(0,0,0,0,1/3,0,0,0,2/3,1/3,0,0,1,2/3,1/3,0), nrow = 4)
constant_penalty = matrix(c(0,0,0,0,1,0,0,0,1,1,0,0,1,1,1,0), nrow = 4)
# first using the constant_penalty (default) matrix
ordROC(gldstd,pred1, penalty=constant_penalty)
# now re-scaling the constant_penalty matrix by 1/2
ordROC(gldstd,pred1, penalty=constant_penalty/2)
# now using an ordered penalty matrix
ordROC(gldstd,pred1,penalty=ordered_penalty)
# now re-scaling the ordered_penalty matrix by 1/2
ordROC(gldstd,pred1,penalty=ordered_penalty/2)
Here is the resulting output:-> # first using the constant_penalty (default) matrix
> ordROC(gldstd,pred1, penalty=constant_penalty)
$`Pairwise Accuracy`
Pair Estimate Standard.Error
1 1 vs 2 0.6420973 0.05667060
2 1 vs 3 0.7083172 0.05293275
3 1 vs 4 0.8500507 0.04149018
4 2 vs 3 0.6081169 0.05438183
5 2 vs 4 0.7950680 0.04736247
6 3 vs 4 0.7025974 0.05371362
$`Penalty Matrix`
1 2 3 4
1 0 1 1 1
2 0 0 1 1
3 0 0 0 1
4 0 0 0 0
$`Overall Accuracy`
Estimate Standard.Error
1 0.7074935 0.02764736
> # now re-scaling the constant_penalty matrix by 1/2
> ordROC(gldstd,pred1, penalty=constant_penalty/2)
$`Pairwise Accuracy`
Pair Estimate Standard.Error
1 1 vs 2 0.6420973 0.05667060
2 1 vs 3 0.7083172 0.05293275
3 1 vs 4 0.8500507 0.04149018
4 2 vs 3 0.6081169 0.05438183
5 2 vs 4 0.7950680 0.04736247
6 3 vs 4 0.7025974 0.05371362
$`Penalty Matrix`
1 2 3 4
1 0 0.5 0.5 0.5
2 0 0.0 0.5 0.5
3 0 0.0 0.0 0.5
4 0 0.0 0.0 0.0
$`Overall Accuracy`
Estimate Standard.Error
1 0.8537467 0.01382368 <=========== larger estimate of overall accuracy
(cf 0.707 for original constant_penalty matrix)
> # now using an ordered penalty matrix
> ordROC(gldstd,pred1,penalty=ordered_penalty)
$`Pairwise Accuracy`
Pair Estimate Standard.Error
1 1 vs 2 0.6420973 0.05667060
2 1 vs 3 0.7083172 0.05293275
3 1 vs 4 0.8500507 0.04149018
4 2 vs 3 0.6081169 0.05438183
5 2 vs 4 0.7950680 0.04736247
6 3 vs 4 0.7025974 0.05371362
$`Penalty Matrix`
1 2 3 4
1 0 0.3333333 0.6666667 1.0000000
2 0 0.0000000 0.3333333 0.6666667
3 0 0.0000000 0.0000000 0.3333333
4 0 0.0000000 0.0000000 0.0000000
$`Overall Accuracy`
Estimate Standard.Error
1 0.8616933 0.01622384
> # now re-scaling the ordered_penalty matrix by 1/2
> ordROC(gldstd,pred1,penalty=ordered_penalty/2)
$`Pairwise Accuracy`
Pair Estimate Standard.Error
1 1 vs 2 0.6420973 0.05667060
2 1 vs 3 0.7083172 0.05293275
3 1 vs 4 0.8500507 0.04149018
4 2 vs 3 0.6081169 0.05438183
5 2 vs 4 0.7950680 0.04736247
6 3 vs 4 0.7025974 0.05371362
$`Penalty Matrix`
1 2 3 4
1 0 0.1666667 0.3333333 0.5000000
2 0 0.0000000 0.1666667 0.3333333
3 0 0.0000000 0.0000000 0.1666667
4 0 0.0000000 0.0000000 0.0000000
$`Overall Accuracy`
Estimate Standard.Error
1 0.9308467 0.00811192 <=========== larger estimate of overall accuracy
(cf 0.862 for original ordered_penalty matrix)
I can see that penalising differences between categories less might produce
better overall accuracy. However, if I use a constant penalty matrix with all
off-diagonal values very close to zero, then the overall accuracy approaches 1.
It seems counter-intuitive to me that the estimate of overall accuracy for an
ordinal gold standard should depend on the absolute values of the penalty
matrix.
So, I would like to ask, (a) should the penalty matrix always contain at least
one penalty with a value of 1 and/or (b) should there be any other constraint on
the sum of penalties in the matrix (e.g. should the matrix sum to some multiple
of the number of categories), or (c) is one free to use arbitrarily-scaled
penalty matrices for estimates of the accuracy of an ordinal gold standard?
Thanks, in advance, for your help,
Jonathan Williams
[[alternative HTML version deleted]]