Sun Shine
2015-Mar-02 07:36 UTC
[R] Using a text file as a removeWord dictionary in tm_map
Thanks Jim. I thought that I was passing a vector, not realising I had converted this to a list object. I haven't come across the scan() function so far, so this is good to know. Good explanation - I'll give this a go when I can get back to that piece of work later today. Thanks again. Regards, Sun On 01/03/15 21:13, jim holtman wrote:> The 'read.table' was creating a data.frame (not a vector) and applying > 'c' to it converted it to a list. You should alway look at the object > you are creating. You probably want to use 'scan'. > > =====================>> testFile <- "Although,this,query,applies,specifically,to,the,tm,package" >> # read in with read.table create a data.frame >> df_words <- read.table(text = testFile, sep = ',') >> df_words # not a vector > V1 V2 V3 V4 V5 V6 V7 V8 V9 > 1 Although this query applies specifically to the tm package >> c(df_words) # this results in a list > $V1 > [1] Although > Levels: Although > $V2 > [1] this > Levels: this > $V3 > [1] query > Levels: query > $V4 > [1] applies > Levels: applies > $V5 > [1] specifically > Levels: specifically > $V6 > [1] to > Levels: to > $V7 > [1] the > Levels: the > $V8 > [1] tm > Levels: tm > $V9 > [1] package > Levels: package >> # now read with 'scan' >> scan_words <- scan(text = testFile, what = '', sep = ',') > Read 9 items >> scan_words > [1] "Although" "this" "query" "applies" > "specifically" "to" > [7] "the" "tm" "package" >> > Jim Holtman > Data Munger Guru > > What is the problem that you are trying to solve? > Tell me what you want to do, not how you want to do it. > > > On Sat, Feb 28, 2015 at 8:46 AM, Sun Shine <phaedrusv at gmail.com> wrote: >> Hi list >> >> Although this query applies specifically to the tm package, perhaps it's >> something that others might be able to lend a thought to. >> >> Using tm to do some initial text mining, I want to include an external (to >> R) generated dictionary of words that I want removed from the corpus. >> >> I have created a comma separated list of terms in " " marks in a >> stopList.txt plain UTF-8 file. I want to read this into R, so do: >> >>> stopDict <- read.table('~/path/to/file/stopList.txt', sep=',') >> When I want to load it as part of the removeWords function in tm, I do: >> >>> docs <- tm_map(docs, removeWords, stopDict) >> which has no effect. Neither does: >> >>> docs <- tm_map(docs, removeWords, c(stopDict)) >> What am I not seeing/ doing? >> >> How do I pass a text file with pre-defined terms to the removeWords >> transform of tm? >> >> Thanks for any ideas. >> >> Cheers >> >> Sun >> >> ______________________________________________ >> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code.
Sun Shine
2015-Mar-03 14:43 UTC
[R] Using a text file as a removeWord dictionary in tm_map
Hi again I've now had the chance to try this out, and using scan() doesn't seem to work either. This is what I used: 1) I generated a plain text file called stopDict.txt. This file is of the format: "a, bunch, of, words, to, use" 2) I invoked scan(), like this: > userStopList <- scan(text = '~/path/to/stopDict.txt', what = " ", sep = ",") 3) Then I used the externally generated list as stop words: > docs <- tm_map(docs, removeWords, userStopList) 3) When I go to inspect the document, at least two of the user-defined stop words are in the text Is there a further argument I should be passing to scan(), or is the stopDict.txt file not set up the correct way? I tried each term separated by ' ' and ',', (e.g. 'all', 'the', 'text') but that didn't work, neither does it seem to work when the whole list is enclosed within quotes (e.g. "all, the, text"). While not critical to have the capacity to read in an externally generated list, it sure would be helpful. Thanks. Sun On 02/03/15 07:36, Sun Shine wrote:> Thanks Jim. > > I thought that I was passing a vector, not realising I had converted > this to a list object. > > I haven't come across the scan() function so far, so this is good to > know. > > Good explanation - I'll give this a go when I can get back to that > piece of work later today. > > Thanks again. > > Regards, > > Sun > > > On 01/03/15 21:13, jim holtman wrote: >> The 'read.table' was creating a data.frame (not a vector) and applying >> 'c' to it converted it to a list. You should alway look at the object >> you are creating. You probably want to use 'scan'. >> >> =====================>>> testFile <- >>> "Although,this,query,applies,specifically,to,the,tm,package" >>> # read in with read.table create a data.frame >>> df_words <- read.table(text = testFile, sep = ',') >>> df_words # not a vector >> V1 V2 V3 V4 V5 V6 V7 V8 V9 >> 1 Although this query applies specifically to the tm package >>> c(df_words) # this results in a list >> $V1 >> [1] Although >> Levels: Although >> $V2 >> [1] this >> Levels: this >> $V3 >> [1] query >> Levels: query >> $V4 >> [1] applies >> Levels: applies >> $V5 >> [1] specifically >> Levels: specifically >> $V6 >> [1] to >> Levels: to >> $V7 >> [1] the >> Levels: the >> $V8 >> [1] tm >> Levels: tm >> $V9 >> [1] package >> Levels: package >>> # now read with 'scan' >>> scan_words <- scan(text = testFile, what = '', sep = ',') >> Read 9 items >>> scan_words >> [1] "Although" "this" "query" "applies" >> "specifically" "to" >> [7] "the" "tm" "package" >>> >> Jim Holtman >> Data Munger Guru >> >> What is the problem that you are trying to solve? >> Tell me what you want to do, not how you want to do it. >> >> >> On Sat, Feb 28, 2015 at 8:46 AM, Sun Shine <phaedrusv at gmail.com> wrote: >>> Hi list >>> >>> Although this query applies specifically to the tm package, perhaps >>> it's >>> something that others might be able to lend a thought to. >>> >>> Using tm to do some initial text mining, I want to include an >>> external (to >>> R) generated dictionary of words that I want removed from the corpus. >>> >>> I have created a comma separated list of terms in " " marks in a >>> stopList.txt plain UTF-8 file. I want to read this into R, so do: >>> >>>> stopDict <- read.table('~/path/to/file/stopList.txt', sep=',') >>> When I want to load it as part of the removeWords function in tm, I do: >>> >>>> docs <- tm_map(docs, removeWords, stopDict) >>> which has no effect. Neither does: >>> >>>> docs <- tm_map(docs, removeWords, c(stopDict)) >>> What am I not seeing/ doing? >>> >>> How do I pass a text file with pre-defined terms to the removeWords >>> transform of tm? >>> >>> Thanks for any ideas. >>> >>> Cheers >>> >>> Sun >>> >>> ______________________________________________ >>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >>> https://stat.ethz.ch/mailman/listinfo/r-help >>> PLEASE do read the posting guide >>> http://www.R-project.org/posting-guide.html >>> and provide commented, minimal, self-contained, reproducible code. >
Jeff Newmiller
2015-Mar-03 15:47 UTC
[R] Using a text file as a removeWord dictionary in tm_map
You seem to be conflating the data input operation with your data processing.
You need to stop and examine the in-memory representation of your data
{"userStopList"), and compare it with the expectations of your data
processing operation ("tm_map"). Then adjust your input data by
choosing a different input function or change the parameters you are using, or
add some manipulation step in between that "fixes" the data so it is
suitable to the task.
Real world data analysis is rarely handled cleanly by one or two function calls.
Take responsibility for insuring the quality of your data by looking at it.
I am not familiar with tm_map, but scan tends to be quite literal with your
specification. I suspect that the elements in your userStopList have spaces in
them because scan is not removing them, and tm_map is then only looking for
instances in your text where spaces are present.
---------------------------------------------------------------------------
Jeff Newmiller The ..... ..... Go Live...
DCN:<jdnewmil at dcn.davis.ca.us> Basics: ##.#. ##.#. Live
Go...
Live: OO#.. Dead: OO#.. Playing
Research Engineer (Solar/Batteries O.O#. #.O#. with
/Software/Embedded Controllers) .OO#. .OO#. rocks...1k
---------------------------------------------------------------------------
Sent from my phone. Please excuse my brevity.
On March 3, 2015 6:43:39 AM PST, Sun Shine <phaedrusv at gmail.com>
wrote:>Hi again
>
>I've now had the chance to try this out, and using scan() doesn't
seem
>to work either.
>
>This is what I used:
>
>1) I generated a plain text file called stopDict.txt. This file is of
>the format: "a, bunch, of, words, to, use"
>
>2) I invoked scan(), like this:
>> userStopList <- scan(text = '~/path/to/stopDict.txt', what =
" ", sep
>
>= ",")
>
>3) Then I used the externally generated list as stop words:
> > docs <- tm_map (docs, removeWords, userStopList)
>
>3) When I go to inspect the document, at least two of the user-defined
>stop words are in the text
>
>Is there a further argument I should be passing to scan(), or is the
>stopDict.txt file not set up the correct way? I tried each term
>separated by ' ' and ',', (e.g. 'all',
'the', 'text') but that didn't
>work, neither does it seem to work when the whole list is enclosed
>within quotes (e.g. "all, the, text").
>
>While not critical to have the capacity to read in an externally
>generated list, it sure would be helpful.
>
>Thanks.
>
>Sun
>
>
>On 02/03/15 07:36, Sun Shine wrote:
>> Thanks Jim.
>>
>> I thought that I was passing a vector, not realising I had converted
>> this to a list object.
>>
>> I haven't come across the scan() function so far, so this is good
to
>> know.
>>
>> Good explanation - I'll give this a go when I can get back to that
>> piece of work later today.
>>
>> Thanks again.
>>
>> Regards,
>>
>> Sun
>>
>>
>> On 01/03/15 21:13, jim holtman wrote:
>>> The 'read.table' was creating a data.frame (not a vector)
and
>applying
>>> 'c' to it converted it to a list. You should alway look at
the
>object
>>> you are creating. You probably want to use 'scan'.
>>>
>>> =====================>>>> testFile <-
>>>>
"Although,this,query,applies,specifically,to,the,tm,package"
>>>> # read in with read.table create a data.frame
>>>> df_words <- read.table(text = testFile, sep = ',')
>>>> df_words # not a vector
>>> V1 V2 V3 V4 V5 V6 V7 V8 V9
>>> 1 Although this query applies specifically to the tm package
>>>> c(df_words) # this results in a list
>>> $V1
>>> [1] Although
>>> Levels: Although
>>> $V2
>>> [1] this
>>> Levels: this
>>> $V3
>>> [1] query
>>> Levels: query
>>> $V4
>>> [1] applies
>>> Levels: applies
>>> $V5
>>> [1] specifically
>>> Levels: specifically
>>> $V6
>>> [1] to
>>> Levels: to
>>> $V7
>>> [1] the
>>> Levels: the
>>> $V8
>>> [1] tm
>>> Levels: tm
>>> $V9
>>> [1] package
>>> Levels: package
>>>> # now read with 'scan'
>>>> scan_words <- scan(text = testFile, what = '', sep =
',')
>>> Read 9 items
>>>> scan_words
>>> [1] "Although" "this"
"query" "applies"
>>> "specifically" "to"
>>> [7] "the" "tm"
"package"
>>>>
>>> Jim Holtman
>>> Data Munger Guru
>>>
>>> What is the problem that you are trying to solve?
>>> Tell me what you want to do, not how you want to do it.
>>>
>>>
>>> On Sat, Feb 28, 2015 at 8:46 AM, Sun Shine <phaedrusv at
gmail.com>
>wrote:
>>>> Hi list
>>>>
>>>> Although this query applies specifically to the tm package,
perhaps
>
>>>> it's
>>>> something that others might be able to lend a thought to.
>>>>
>>>> Using tm to do some initial text mining, I want to include an
>>>> external (to
>>>> R) generated dictionary of words that I want removed from the
>corpus.
>>>>
>>>> I have created a comma separated list of terms in " "
marks in a
>>>> stopList.txt plain UTF-8 file. I want to read this into R, so
do:
>>>>
>>>>> stopDict <-
read.table('~/path/to/file/stopList.txt', sep=',')
>>>> When I want to load it as part of the removeWords function in
tm, I
>do:
>>>>
>>>>> docs <- tm_map(docs, removeWords, stopDict)
>>>> which has no effect. Neither does:
>>>>
>>>>> docs <- tm_map(docs, removeWords, c(stopDict))
>>>> What am I not seeing/ doing?
>>>>
>>>> How do I pass a text file with pre-defined terms to the
removeWords
>>>> transform of tm?
>>>>
>>>> Thanks for any ideas.
>>>>
>>>> Cheers
>>>>
>>>> Sun
>>>>
>>>> ______________________________________________
>>>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and
more, see
>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>> PLEASE do read the posting guide
>>>> http://www.R-project.org/posting-guide.html
>>>> and provide commented, minimal, self-contained, reproducible
code.
>>
>
>______________________________________________
>R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.