I'm using text miner (the "tm" package) to process large numbers
of blog and message board postings (about 245,000). Does anyone have any advice
for how to efficiently extract the meta data from a corpus of this size?
TM does a great job of using MPI for many functions (e.g. tmMap) which greatly
speed up the processing. However, the "meta" function that I need does
not take advantage of MPI.
I have two ideas:
1) Find a way of running the meta function in parallel mode. Specifically, the
code that I'm running is:
urllist <- lapply(workingcorpus, meta, tag = "FeedUrl")
Unfortunately, I receive the following error message when I try to use the
command "parLapply"
"Error in checkCluster(cl) : not a valid cluster
Calls: parLapply ... is.vector -> clusterApply -> staticClusterApply ->
checkCluster"
2) Alternatively, I wonder if there might be a way of extracting all of the meta
data into a data.frame that would be faster for processing?
Thanks for any suggestions or ideas!
Shad
shad thomas | president | glass box research company | +1 (312) 451-3611 tel |
shad.thomas@glassboxresearch.com | www.glassboxresearch.com
[[alternative HTML version deleted]]
