R help - Jun 2009 - Exporting large datasets

Hi all,

I am currently working on a particularly large (data wise) project, the
problem I am having is with exporting the data (for use later on). My
current sample dataset contains about 400million entries (approximately
20000 columns  x 20000 rows). I tried using the write.table command, but 8
hours later its still going. I will be trying write.matrix (MASS) next as it
is supposed to handle things a bit quicker, but I thought I would ask if
anyone has had experience with exporting large datasets and what is the
quickest method to do so.

Thanks in advance

-Dmitri
-- 
View this message in context:
http://www.nabble.com/Exporting-large-datasets-tp24249149p24249149.html
Sent from the R help mailing list archive at Nabble.com.

What do you want to export it to and in what format?  You will probably have
a problem trying to write that large an object out at one time.  Try instead
opening a connecting and the writing much smaller pieces to the file:

x <- matrix(runif(10000), 100)
x[,1] <- 1:100  # put some labels for testing
x.out <- file('/test.csv', 'w')
# not efficient, just for demonstration
for (i in 1:100) write.table(x[i,, drop=FALSE], file=x.out,
    sep=',', col.names=FALSE, row.names=FALSE)
close(x.out)



On Mon, Jun 29, 2009 at 2:51 AM, DrDimo
<dmitri.mouradov@ludwig.edu.au>wrote:
>
> Hi all,
>
> I am currently working on a particularly large (data wise) project, the
> problem I am having is with exporting the data (for use later on). My
> current sample dataset contains about 400million entries (approximately
> 20000 columns  x 20000 rows). I tried using the write.table command, but 8
> hours later its still going. I will be trying write.matrix (MASS) next as
> it
> is supposed to handle things a bit quicker, but I thought I would ask if
> anyone has had experience with exporting large datasets and what is the
> quickest method to do so.
>
> Thanks in advance
>
> -Dmitri
> --
> View this message in context:
> http://www.nabble.com/Exporting-large-datasets-tp24249149p24249149.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
>
http://www.R-project.org/posting-guide.html<http://www.r-project.org/posting-guide.html>
> and provide commented, minimal, self-contained, reproducible code.
>


-- 
Jim Holtman
Cincinnati, OH
+1 513 646 9390

What is the problem that you are trying to solve?

	[[alternative HTML version deleted]]

Depends on what you mean by "export".  I like save().  Check out the 
options in the help.  It seems to be very efficient and I had no 
problems working with the 50,000 x 15,000 data sets from the 2009 KDD 
Cup challenge.  Maybe your disk is just slow (network file storage)?

Allan.

On 29/06/09 07:51, DrDimo wrote:
> Hi all,
>
> I am currently working on a particularly large (data wise) project, the
> problem I am having is with exporting the data (for use later on). My
> current sample dataset contains about 400million entries (approximately
> 20000 columns  x 20000 rows). I tried using the write.table command, but 8
> hours later its still going. I will be trying write.matrix (MASS) next as
it
> is supposed to handle things a bit quicker, but I thought I would ask if
> anyone has had experience with exporting large datasets and what is the
> quickest method to do so.
>
> Thanks in advance
>
> -Dmitri
>    
	[[alternative HTML version deleted]]