Dear all,
This is the first time I am sending mail to the mailing list, so I
hope I do not make a mistake...
The last months I have been working on my MSc thesis project on
performing data mining techniques on user logs of a
software-as-a-service application. The main problem I am experiencing
is how to process the huge amount of data. More specifically:
I am using R 2.10.1 in a laptop with Windows 7 - 32bit system, 2GB RAM
and CPU Intel Core Duo 2GHz.
The user logs data come from a query Crystal report (.rpt file) which
I transform with some Java code into a tab separated file.
Although with a small subset of my data everything manages to run,
when I increase the data set I get several problems:
The first problem is with the use of read.delim(). When I try to read
a big amount of data (over 2.400.000 rows and 18 attributes at each
row) it doesn't seem to transform all table into a data frame. In
particular, the data frame returned has 1.220.987 rows.
Furthermore, as one of the data attributes is DataTime, when I try to
split this column into two columns (one with Data and one with the
Time), the returned result is quite strange, as the two new columns
appear to have more rows than the data frame:
applicLog.dat <- read.delim("file.txt")
#Process the syscreated column (Date time --> Date + time)
copyDate <- applicLog.dat[["?..syscreated"]]
copyDate <- as.character(copyDate)
splitDate <- strsplit(copyDate, " ")
splitDate <- unlist(splitDate)
splitDateIndex <- c(1:length(splitDate))
sysCreatedDate <- splitDate[splitDateIndex %% 2 == 1]
sysCreatedTime <- splitDate[splitDateIndex %% 2 == 0]
sysCreatedDate <- strptime(sysCreatedDate, format="%Y-%m-%d")
op <- options(digits.secs = 3)
sysCreatedTime <- strptime(sysCreatedTime, format ="%H:%M:%OS")
applicLog.dat[["?..syscreated"]] <- NULL
applicLog.dat <- cbind (sysCreatedDate,sysCreatedTime,applicLog.dat)
Then I get the error: Error in data.frame(..., check.names = FALSE) :
arguments imply differing number of rows: 1221063, 1221062, 1220987
Finally, another problem I have is when I perform association mining
on the data set using the package arules: I turn the data frame into
transactions table and then run the apriori algorithm. When I put too
low support in order to manage to find the rules I need, the vector of
rules becomes too big and I get problems with the memory such as:
Error: cannot allocate vector of size 923.1 Mb
In addition: Warning messages:
1: In items(x) : Reached total allocation of 153Mb: see help(memory.size)
Could you please help me with how I could allocate more RAM? Or, do
you think there is a way to process the data by loading them into a
document instead of loading all into RAM? Do you know how I could
manage to read all my data set?
I would really appreciate your help.
Kind regards,
Stella Pachidi
PS: Do you know any text editor that can read huge .txt files?
--
Stella Pachidi
Master in Business Informatics student
Utrecht University
Stella,
A few brief words of advice:
1. Work through your code a line at a time, making sure that each is what
you would expect. I think some of your later problems are a result of
something
early not being as expected. For example, if the read.delim() is in fact
not
giving you what you expect, stop there before moving onwards. I suspect
some funny character(s) or character encodings might be a problem.
2. 32-bit Windows can be limiting. With 2 GB of RAM, you're probably not
going to be able to work effectively in native R with objects over 200-300
MB,
and the error indicates that something (you or a package you're using)
simply
have run out of memory. So...
3. Consider more RAM (and preferably with 64-bit R). Other solutions might
be possible, such as using a database to hand the data transition into R.
2.5 million rows by 18 columns is apt to be around 360 MB. Although you
can afford 1 (or a few) copies of this, it doesn't leave you much room for
the memory overhead of working with such an object.
Part of the oringal message below.
Jay
-------------------------------------------------------------
Message: 80
Date: Mon, 19 Apr 2010 22:07:03 +0200
From: Stella Pachidi <stella.pachidi@gmail.com>
To: r-help@stat.math.ethz.ch
Subject: [R] Huge data sets and RAM problems
Message-ID:
<g2j133363581004191307t2a48c1bfqd9d57cf0d6c629a6@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Dear all,
....
I am using R 2.10.1 in a laptop with Windows 7 - 32bit system, 2GB RAM
and CPU Intel Core Duo 2GHz.
.....
Finally, another problem I have is when I perform association mining
on the data set using the package arules: I turn the data frame into
transactions table and then run the apriori algorithm. When I put too
low support in order to manage to find the rules I need, the vector of
rules becomes too big and I get problems with the memory such as:
Error: cannot allocate vector of size 923.1 Mb
In addition: Warning messages:
1: In items(x) : Reached total allocation of 153Mb: see help(memory.size)
Could you please help me with how I could allocate more RAM? Or, do
you think there is a way to process the data by loading them into a
document instead of loading all into RAM? Do you know how I could
manage to read all my data set?
I would really appreciate your help.
Kind regards,
Stella Pachidi
--
John W. Emerson (Jay)
Associate Professor of Statistics
Department of Statistics
Yale University
http://www.stat.yale.edu/~jay
[[alternative HTML version deleted]]
You set records to NULL perhaps (delete, shift up). Perhaps your system is
susceptible to butterflies on the other side of the world.
Your code may have 'worked' on a small section of data, but the data
used
did not include all of the cases needed to fully test your code. So... test
your code!
scan(), used with 'nlines', 'skip', 'sep', and
'what' will cut your read
time by at least half while taking less RAM memory to do it, do most of your
post processing, and give you something to better test your code. Or, don't
use 'nlines' and lose your time/memory benefits over read.table().
'skip'
will get you "right to the point" before where things failed. That
would be
an interesting small segment of data to test with.
wordpad can read your file (and then some). Eventually.
Sincerely,
KeithC.
-----Original Message-----
From: Stella Pachidi [mailto:stella.pachidi at gmail.com]
Sent: Monday, April 19, 2010 2:07 PM
To: r-help at stat.math.ethz.ch
Subject: [R] Huge data sets and RAM problems
Dear all,
This is the first time I am sending mail to the mailing list, so I hope I do
not make a mistake...
The last months I have been working on my MSc thesis project on performing
data mining techniques on user logs of a software-as-a-service application.
The main problem I am experiencing is how to process the huge amount of
data. More specifically:
I am using R 2.10.1 in a laptop with Windows 7 - 32bit system, 2GB RAM and
CPU Intel Core Duo 2GHz.
The user logs data come from a query Crystal report (.rpt file) which I
transform with some Java code into a tab separated file.
Although with a small subset of my data everything manages to run, when I
increase the data set I get several problems:
The first problem is with the use of read.delim(). When I try to read a big
amount of data (over 2.400.000 rows and 18 attributes at each
row) it doesn't seem to transform all table into a data frame. In
particular, the data frame returned has 1.220.987 rows.
Furthermore, as one of the data attributes is DataTime, when I try to split
this column into two columns (one with Data and one with the Time), the
returned result is quite strange, as the two new columns appear to have more
rows than the data frame:
applicLog.dat <- read.delim("file.txt")
#Process the syscreated column (Date time --> Date + time) copyDate <-
applicLog.dat[["?..syscreated"]] copyDate <- as.character(copyDate)
splitDate <- strsplit(copyDate, " ") splitDate <-
unlist(splitDate)
splitDateIndex <- c(1:length(splitDate)) sysCreatedDate <-
splitDate[splitDateIndex %% 2 == 1] sysCreatedTime <-
splitDate[splitDateIndex %% 2 == 0] sysCreatedDate <-
strptime(sysCreatedDate, format="%Y-%m-%d") op <-
options(digits.secs = 3)
sysCreatedTime <- strptime(sysCreatedTime, format ="%H:%M:%OS")
applicLog.dat[["?..syscreated"]] <- NULL applicLog.dat <- cbind
(sysCreatedDate,sysCreatedTime,applicLog.dat)
Then I get the error: Error in data.frame(..., check.names = FALSE) :
arguments imply differing number of rows: 1221063, 1221062, 1220987
Finally, another problem I have is when I perform association mining on the
data set using the package arules: I turn the data frame into transactions
table and then run the apriori algorithm. When I put too low support in
order to manage to find the rules I need, the vector of rules becomes too
big and I get problems with the memory such as:
Error: cannot allocate vector of size 923.1 Mb In addition: Warning
messages:
1: In items(x) : Reached total allocation of 153Mb: see help(memory.size)
Could you please help me with how I could allocate more RAM? Or, do you
think there is a way to process the data by loading them into a document
instead of loading all into RAM? Do you know how I could manage to read all
my data set?
I would really appreciate your help.
Kind regards,
Stella Pachidi
PS: Do you know any text editor that can read huge .txt files?
--
Stella Pachidi
Master in Business Informatics student
Utrecht University