R help - Dec 2006 - cohen kappa for two-way table

Greetings,

I am a bit confused by the results returned by the functions:
cohen.kappa {concord}
classAgreement {e1071}

when using a two-way table.


for example, if I have an matrix A, and a similar matrix B (same
dimensions), then:

matrix A and B can be found:
http://casoilresource.lawr.ucdavis.edu/drupal/files/a_40.txt
http://casoilresource.lawr.ucdavis.edu/drupal/files/b_40.txt

A <- matrix(unlist( read.table('a_40.txt'), use.names=FALSE),
ncol=14)
B <- matrix(unlist( read.table('b_40.txt'), use.names=FALSE),
ncol=14)

# compute cohen's kappa, default settings:
cohen.kappa(table(A,B))
Kappa test for nominally classified data
9 categories - 90 methods
kappa (Cohen) = 0.97353 , Z = 45.4465 , p = 0
kappa (Siegel) = -0.00744097 , Z = -0.0794501 , p = 0.531663
kappa (2*PA-1) = 0.947061


# compute cohen's kappa - type = counts
cohen.kappa(table(A,B), type='counts')

Different row sums, a no-classification category was added.

Kappa test for nominally classified data
91 categories - 22 methods
kappa (Siegel) = 0.168593 , Z = 2.50298 , p = 0.00615762
kappa (2*PA-1) = 0.71485

it seems like the second method (type='counts') is the correct way to
use a contingency table... but am i correct?

Secondly, when using the classAgreements() function I get different numbers:

classAgreement(table(A,B))
$diag
[1] 0.03296703

$kappa
[1] 0.02180419

$rand
[1] 0.9874325

$crand
[1] 0.7648124



Perhaps I am mis-reading the relevant manual pages. Can anyone shed
some light on the proper use, and therfore interpretation of these two
methods - when using a contingency table as input?

Thanks in advance,

Dylan

Dylan Beaudette wrote:
> Greetings,
> 
> I am a bit confused by the results returned by the functions:
> cohen.kappa {concord}
> classAgreement {e1071}
> 
> when using a two-way table.
> 
> 
> for example, if I have an matrix A, and a similar matrix B (same
> dimensions), then:
> 
> matrix A and B can be found:
> http://casoilresource.lawr.ucdavis.edu/drupal/files/a_40.txt
> http://casoilresource.lawr.ucdavis.edu/drupal/files/b_40.txt
> 
> A <- matrix(unlist( read.table('a_40.txt'), use.names=FALSE),
ncol=14)
> B <- matrix(unlist( read.table('b_40.txt'), use.names=FALSE),
ncol=14)
> 
If I interpret this correctly, you are considering the numbers in A and 
B to be nominal variables representing levels of some data attribute.
What cohen.kappa wants is:

1(type=counts) a matrix of counts where each cell represents the number 
of methods (rows are different methods) that classified the data objects 
into levels (columns are different levels).
2(type=scores) a matrix of scores where each cell represents the level 
assigned by a method (rows again) to a data object (columns are data 
objects here).

If your numbers are counts, you can get a sensible kappa for _each_ matrix.
If your numbers are scores, you can get a sensible kappa by specifying 
type=scores for _each_ matrix
As far as I can see, you are computing a sparsely populated table of the 
two matrices and passing that, and I don't see how that can give a 
meaningful result. I suspect that the numbers might be counts in the 
first place, but they are being used as scores.
> Secondly, when using the classAgreements() function I get different
numbers:
> 
I'll let someone who knows more about the classAgreement function deal 
with this.

Jim