R help - Apr 2012 - Text mining: Narrowing a field of 27, 855 predictors using semi-partial correlations or some other means

Hello Everyone,

Trying to learn a little bit about data mining. I'm working on a text mining
project that will attempt to predict whether cancer patients got a particular
type of genetic testing. A subsequent stage then will be aimed at predicting
what the results of that testing were.
?
I've used the tm package to prepare my data and am planning to use rattle to
do the actual data mining. The tm package has proved to be a great help so far.
I've managed to perform a variety of transformations of my data. I've
also managed to create a document-term matrix that has a row for each of my
patients and columns for each of the terms in my patient medical records.
?
Because I'm not yet a particularly good R programmer, I've converted my
document-term matrix to a data frame and then added information about the
genetic testing.
?
So here's the thing. The tm package has a feature that would allow me to
drop words that occur infrequently in patient medical records. However, I've
been asked not to use it because it's believed that even infrequently
occurring terms may be highly diagnostic. The consequence is that my data frame
has a large number of columns for the various words. In fact, over 27,000 of
them.
?
So my question is how to reduce this to some more manageable number. One thought
has been to look at semi-partial correlations. Here these would be between
tested(y/n) and each predictor, controlling for length of medical record. The
idea would be to use only those predictors that were significant in the actual
data mining.
?
Is this likely to be a good approach? Or is there likely to be a better way of
doing it?
?
If it is a good approach, I?m wondering how to go about obtaining the necessary
results. I?ve managed to figure out how to compute semi-partial correlations
using the spcor.test() function in the ppcor package, as in:
?
> spcor.test(as.numeric(Tested$TestStatus=="Yes"),
Tested$predictor, Tested $nchar_record)
?
?? estimate????? p.value statistic?? n gp? Method
1 0.3853547 2.307562e-08? 5.587203 182? 1 pearson
?
This is fine for a single pair of variables. What I?d need though is to combine
a whole series of such outputs, one for each of my predictors. After that, I?d
need to be able to determine which semi-partial correlations were significant
(or perhaps substantial) and to create a list that I could use to eliminate a
lot of the predictors from my data frame. I?m just beginning to use R in my
day-to-day work. So it?s not clear to me how to do this.
?
Thanks,
?
Paul

?