thr3ads.net - R help - [R] Memory usage in R grows considerably while calculating word frequencies [Sep 2012]

If this information is useful, please help other people find it:
Share via:

mcelis

2012-Sep-24 23:29 UTC

[R] Memory usage in R grows considerably while calculating word frequencies

I am working with some large text files (up to 16 GBytes).  I am interested
in extracting the words and counting each time each word appears in the
text. I have written a very simple R program by following some suggestions
and examples I found online.  

If my input file is 1 GByte, I see that R uses up to 11 GBytes of memory
when executing the program on
a 64-bit system running CentOS 6.3. Why is R using so much memory? Is there
a better way to do this that will
minimize memory usage.

I am very new to R, so I would appreciate some tips on how to improve my
program or a better way to do it.

R program:
# Read in the entire file and convert all words in text to lower case
words.txt<-tolower(scan("text_file","character",sep="\n"))

# Extract words
pattern <- "(\\b[A-Za-z]+\\b)"
match <- gregexpr(pattern,words.txt)
words.txt <- regmatches(words.txt,match)

# Create a vector from the list of words
words.txt<-unlist(words.txt)

# Calculate word frequencies
words.txt<-table(words.txt,dnn="words")

# Sort by frequency, not alphabetically
words.txt<-sort(words.txt,decreasing=TRUE)

# Put into some readable form, "Name of word" and "Number of
times it
occurs"
words.txt<-paste(names(words.txt),words.txt,sep="\t")

# Results to a file
cat("Word\tFREQ",words.txt,file="frequencies",sep="\n")



--
View this message in context:
http://r.789695.n4.nabble.com/Memory-usage-in-R-grows-considerably-while-calculating-word-frequencies-tp4644053.html
Sent from the R help mailing list archive at Nabble.com.

arun

2012-Sep-25 02:23 UTC

head link

[R] Memory usage in R grows considerably while calculating word frequencies

HI,
In a text file of 6834 words, I compared your program with a modified program.
sapply(strsplit(txt1," "),length)
#[1] 6834

#your program
system.time({
txt1<-tolower(scan("text_file","character",sep="\n"))
pattern <- "(\\b[A-Za-z]+\\b)"
match <- gregexpr(pattern,txt1)
words.txt <- regmatches(txt1,match)
words.txt<-unlist(words.txt)
words.txt<-table(words.txt,dnn="words")
words.txt<-sort(words.txt,decreasing=TRUE)
words.txt<-paste(names(words.txt),words.txt,sep="\t")
cat("Word\tFREQ",words.txt,file="frequencies",sep="\n")
})

#?? user? system elapsed 
?# 0.208?? 0.000?? 0.206 

#Modified code

system.time({
txt1<-tolower(scan("text_file","character",sep="\n"))
?words.txt<-sort(table(strsplit(tolower(txt1),"\\s")),decreasing=TRUE)
?words.txt<-paste(names(words.txt),words.txt,sep="\t")
?cat("Word\tFREQ",words.txt,file="frequencies",sep="\n")
})
#? user? system elapsed 
?# 0.016?? 0.000?? 0.014? 

A.K.









----- Original Message -----
From: mcelis <mcelis at lightminersystems.com>
To: r-help at r-project.org
Cc: 
Sent: Monday, September 24, 2012 7:29 PM
Subject: [R] Memory usage in R grows considerably while calculating word
frequencies

I am working with some large text files (up to 16 GBytes).? I am interested
in extracting the words and counting each time each word appears in the
text. I have written a very simple R program by following some suggestions
and examples I found online.? 

If my input file is 1 GByte, I see that R uses up to 11 GBytes of memory
when executing the program on
a 64-bit system running CentOS 6.3. Why is R using so much memory? Is there
a better way to do this that will
minimize memory usage.

I am very new to R, so I would appreciate some tips on how to improve my
program or a better way to do it.

R program:
# Read in the entire file and convert all words in text to lower case
words.txt<-tolower(scan("text_file","character",sep="\n"))

# Extract words
pattern <- "(\\b[A-Za-z]+\\b)"
match <- gregexpr(pattern,words.txt)
words.txt <- regmatches(words.txt,match)

# Create a vector from the list of words
words.txt<-unlist(words.txt)

# Calculate word frequencies
words.txt<-table(words.txt,dnn="words")

# Sort by frequency, not alphabetically
words.txt<-sort(words.txt,decreasing=TRUE)

# Put into some readable form, "Name of word" and "Number of
times it
occurs"
words.txt<-paste(names(words.txt),words.txt,sep="\t")

# Results to a file
cat("Word\tFREQ",words.txt,file="frequencies",sep="\n")



--
View this message in context:
http://r.789695.n4.nabble.com/Memory-usage-in-R-grows-considerably-while-calculating-word-frequencies-tp4644053.html
Sent from the R help mailing list archive at Nabble.com.

______________________________________________
R-help at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

arun

2012-Sep-25 02:59 UTC

head link

[R] Memory usage in R grows considerably while calculating word frequencies

HI,

In the previous email, I forgot to add unlist().
With four paragraphs,
sapply(strsplit(txt1," "),length)
#[1] 4850 9072 6400 2071


#Your code:
system.time({
txt1<-tolower(scan("text_file","character",sep="\n"))
pattern <- "(\\b[A-Za-z]+\\b)"
match <- gregexpr(pattern,txt1)
words.txt <- regmatches(txt1,match)
words.txt<-unlist(words.txt)
words.txt<-table(words.txt,dnn="words")
words.txt<-sort(words.txt,decreasing=TRUE)
words.txt<-paste(names(words.txt),words.txt,sep="\t")
cat("Word\tFREQ",words.txt,file="frequencies",sep="\n")
})

#Read 4 items
#?? user? system elapsed 
# 11.781?? 0.004? 11.799 


#Modified code:
system.time({
txt1<-tolower(scan("text_file","character",sep="\n"))
?words.txt<-sort(table(unlist(strsplit(tolower(txt1),"\\s"))),decreasing=TRUE)
?words.txt<-paste(names(words.txt),words.txt,sep="\t")
?cat("Word\tFREQ",words.txt,file="frequencies",sep="\n")
})
#Read 4 items
?#user? system elapsed 
?# 0.036?? 0.008?? 0.043 


A.K.




----- Original Message -----
From: mcelis <mcelis at lightminersystems.com>
To: r-help at r-project.org
Cc: 
Sent: Monday, September 24, 2012 7:29 PM
Subject: [R] Memory usage in R grows considerably while calculating word
frequencies

I am working with some large text files (up to 16 GBytes).? I am interested
in extracting the words and counting each time each word appears in the
text. I have written a very simple R program by following some suggestions
and examples I found online.? 

If my input file is 1 GByte, I see that R uses up to 11 GBytes of memory
when executing the program on
a 64-bit system running CentOS 6.3. Why is R using so much memory? Is there
a better way to do this that will
minimize memory usage.

I am very new to R, so I would appreciate some tips on how to improve my
program or a better way to do it.

R program:
# Read in the entire file and convert all words in text to lower case
words.txt<-tolower(scan("text_file","character",sep="\n"))

# Extract words
pattern <- "(\\b[A-Za-z]+\\b)"
match <- gregexpr(pattern,words.txt)
words.txt <- regmatches(words.txt,match)

# Create a vector from the list of words
words.txt<-unlist(words.txt)

# Calculate word frequencies
words.txt<-table(words.txt,dnn="words")

# Sort by frequency, not alphabetically
words.txt<-sort(words.txt,decreasing=TRUE)

# Put into some readable form, "Name of word" and "Number of
times it
occurs"
words.txt<-paste(names(words.txt),words.txt,sep="\t")

# Results to a file
cat("Word\tFREQ",words.txt,file="frequencies",sep="\n")



--
View this message in context:
http://r.789695.n4.nabble.com/Memory-usage-in-R-grows-considerably-while-calculating-word-frequencies-tp4644053.html
Sent from the R help mailing list archive at Nabble.com.

______________________________________________
R-help at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Milan Bouchet-Valat

2012-Sep-25 12:08 UTC

head link

[R] Memory usage in R grows considerably while calculating word frequencies

Le lundi 24 septembre 2012 ? 16:29 -0700, mcelis a ?crit
:> I am working with some large text files (up to 16 GBytes).  I am interested
> in extracting the words and counting each time each word appears in the
> text. I have written a very simple R program by following some suggestions
> and examples I found online.  
> 
> If my input file is 1 GByte, I see that R uses up to 11 GBytes of memory
> when executing the program on
> a 64-bit system running CentOS 6.3. Why is R using so much memory? Is there
> a better way to do this that will
> minimize memory usage.
> 
> I am very new to R, so I would appreciate some tips on how to improve my
> program or a better way to do it.First, I think you should have a look at the tm package by Ingo
Feinerer. It will help you to import the texts, optionally run
processing steps on it, and then extract the words and create a
document-term matrix counting their frequencies. No need to reinvent the
wheel.

Second, there's nothing wrong with using RAM as long as it's available.
If other programs need it, the Linux will reclaim it. There's a problem
only if R's memory use does not reduce at that point. Use gc() to check
whether the RAM allocated to R is really in use. But tm should improve
the efficiency of the computations.


My two cents
> R program:
> # Read in the entire file and convert all words in text to lower case
>
words.txt<-tolower(scan("text_file","character",sep="\n"))
> 
> # Extract words
> pattern <- "(\\b[A-Za-z]+\\b)"
> match <- gregexpr(pattern,words.txt)
> words.txt <- regmatches(words.txt,match)
> 
> # Create a vector from the list of words
> words.txt<-unlist(words.txt)
> 
> # Calculate word frequencies
> words.txt<-table(words.txt,dnn="words")
> 
> # Sort by frequency, not alphabetically
> words.txt<-sort(words.txt,decreasing=TRUE)
> 
> # Put into some readable form, "Name of word" and "Number of
times it
> occurs"
> words.txt<-paste(names(words.txt),words.txt,sep="\t")
> 
> # Results to a file
>
cat("Word\tFREQ",words.txt,file="frequencies",sep="\n")
> 
> 
> 
> --
> View this message in context:
http://r.789695.n4.nabble.com/Memory-usage-in-R-grows-considerably-while-calculating-word-frequencies-tp4644053.html
> Sent from the R help mailing list archive at Nabble.com.
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

Martin Maechler

2012-Sep-25 13:07 UTC

head link

[R] Memory usage in R grows considerably while calculating word frequencies

>>>>> arun  <smartpink111 at yahoo.com>
>>>>>     on Mon, 24 Sep 2012 19:59:35 -0700 writes:
    > HI,
    > In the previous email, I forgot to add unlist().
    > With four paragraphs,
    > sapply(strsplit(txt1," "),length)
    > #[1] 4850 9072 6400 2071


    > #Your code:
    > system.time({
    >
txt1<-tolower(scan("text_file","character",sep="\n"))
    > pattern <- "(\\b[A-Za-z]+\\b)"
    > match <- gregexpr(pattern,txt1)
    > words.txt <- regmatches(txt1,match)
    > words.txt<-unlist(words.txt)
    > words.txt<-table(words.txt,dnn="words")
    > words.txt<-sort(words.txt,decreasing=TRUE)
    > words.txt<-paste(names(words.txt),words.txt,sep="\t")
    >
cat("Word\tFREQ",words.txt,file="frequencies",sep="\n")
    > })

    > #Read 4 items
    > #?? user? system elapsed 
    > # 11.781?? 0.004? 11.799 


    > #Modified code:
    > system.time({
    >
txt1<-tolower(scan("text_file","character",sep="\n"))
    >
?words.txt<-sort(table(unlist(strsplit(tolower(txt1),"\\s"))),decreasing=TRUE)
    > ?words.txt<-paste(names(words.txt),words.txt,sep="\t")
    >
?cat("Word\tFREQ",words.txt,file="frequencies",sep="\n")
    > })
    > #Read 4 items
    > ?#user? system elapsed 
    > ?# 0.036?? 0.008?? 0.043 


    > A.K.

Well, dear A.K., your definition of "word" is really different,
and in my view clearly much too simplistic, compared to what the
OP (= original-poster) asked from.

E.g., from the above paragraph, your method will get words such as
 "A.K.,"   "different,"  or  "(="  
clearly wrongly.

Martin Maechler, ETH Zurich



    > ----- Original Message -----
    > From: mcelis <mcelis at lightminersystems.com>
    > To: r-help at r-project.org
    > Cc: 
    > Sent: Monday, September 24, 2012 7:29 PM
    > Subject: [R] Memory usage in R grows considerably while calculating
word frequencies

    > I am working with some large text files (up to 16 GBytes).? I am
interested
    > in extracting the words and counting each time each word appears in the
    > text. I have written a very simple R program by following some
suggestions
    > and examples I found online.? 

    > If my input file is 1 GByte, I see that R uses up to 11 GBytes of
memory
    > when executing the program on
    > a 64-bit system running CentOS 6.3. Why is R using so much memory? Is
there
    > a better way to do this that will
    > minimize memory usage.

    > I am very new to R, so I would appreciate some tips on how to improve
my
    > program or a better way to do it.

    > R program:
    > # Read in the entire file and convert all words in text to lower case
    >
words.txt<-tolower(scan("text_file","character",sep="\n"))

    > # Extract words
    > pattern <- "(\\b[A-Za-z]+\\b)"
    > match <- gregexpr(pattern,words.txt)
    > words.txt <- regmatches(words.txt,match)

    > # Create a vector from the list of words
    > words.txt<-unlist(words.txt)

    > # Calculate word frequencies
    > words.txt<-table(words.txt,dnn="words")

    > # Sort by frequency, not alphabetically
    > words.txt<-sort(words.txt,decreasing=TRUE)

    > # Put into some readable form, "Name of word" and
"Number of times it
    > occurs"
    > words.txt<-paste(names(words.txt),words.txt,sep="\t")

    > # Results to a file
    >
cat("Word\tFREQ",words.txt,file="frequencies",sep="\n")



    > --
    > View this message in context:
http://r.789695.n4.nabble.com/Memory-usage-in-R-grows-considerably-while-calculating-word-frequencies-tp4644053.html
    > Sent from the R help mailing list archive at Nabble.com.

    > ______________________________________________
    > R-help at r-project.org mailing list
    > https://stat.ethz.ch/mailman/listinfo/r-help
    > PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
    > and provide commented, minimal, self-contained, reproducible code.


    > ______________________________________________
    > R-help at r-project.org mailing list
    > https://stat.ethz.ch/mailman/listinfo/r-help
    > PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
    > and provide commented, minimal, self-contained, reproducible code.

Rainer M Krug

2012-Sep-26 07:58 UTC

head link

[R] Memory usage in R grows considerably while calculating word frequencies

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 25/09/12 01:29, mcelis wrote:> I am working with some large text files (up to 16 GBytes).  I am interested
in extracting the
> words and counting each time each word appears in the text. I have written
a very simple R
> program by following some suggestions and examples I found online.
Just an idea (I have no experience with what you want to do, so it might not
work):

What about putting the text in a database (sqlite comes to mind) where each word
is one entry.
Then you could use sql to query the database, which should need much less
memory.

In addition, it should make further processing much easier.

Cheers,

Rainer
> 
> If my input file is 1 GByte, I see that R uses up to 11 GBytes of memory
when executing the
> program on a 64-bit system running CentOS 6.3. Why is R using so much
memory? Is there a
> better way to do this that will minimize memory usage.
> 
> I am very new to R, so I would appreciate some tips on how to improve my
program or a better
> way to do it.
> 
> R program: # Read in the entire file and convert all words in text to lower
case
>
words.txt<-tolower(scan("text_file","character",sep="\n"))
> 
> # Extract words pattern <- "(\\b[A-Za-z]+\\b)" match <-
gregexpr(pattern,words.txt) words.txt
> <- regmatches(words.txt,match)
> 
> # Create a vector from the list of words words.txt<-unlist(words.txt)
> 
> # Calculate word frequencies
words.txt<-table(words.txt,dnn="words")
> 
> # Sort by frequency, not alphabetically
words.txt<-sort(words.txt,decreasing=TRUE)
> 
> # Put into some readable form, "Name of word" and "Number of
times it occurs"
> words.txt<-paste(names(words.txt),words.txt,sep="\t")
> 
> # Results to a file
cat("Word\tFREQ",words.txt,file="frequencies",sep="\n")
> 
> 
> 
> -- View this message in context: 
>
http://r.789695.n4.nabble.com/Memory-usage-in-R-grows-considerably-while-calculating-word-frequencies-tp4644053.html
>
>
> 
Sent from the R help mailing list archive at Nabble.com.> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlBitboACgkQoYgNqgF2egr1pgCgjHxE/E1qIwUbrYzB30qIk9cK
z/oAoILCYn66+c9CF5tzkWeQH3E2utwi
=ahI5
-----END PGP SIGNATURE-----

Reasonably Related Threads

Search for more apparently analagous threads

R help - Sep 2012 - Memory usage in R grows considerably while calculating word frequencies

[R] Memory usage in R grows considerably while calculating word frequencies

[R] Memory usage in R grows considerably while calculating word frequencies

[R] Memory usage in R grows considerably while calculating word frequencies

[R] Memory usage in R grows considerably while calculating word frequencies

[R] Memory usage in R grows considerably while calculating word frequencies

[R] Memory usage in R grows considerably while calculating word frequencies

Reasonably Related Threads