R help - Oct 2013 - How to save very large matrix?

Hello!
I have a very large matrix of results: 50000x100000. I saved it as RDS, but I
would also need to save it as txt or csv. Is there a way to do it? Now, with
write.table I am receiving an error:
Error in .External2(C_writetable, x, file, nrow(x), p, rnames, sep, eol,  : 
  long vectors not supported yet: io.c:1116

Please, help! Many thanks!

PM

Have you tried write.csv() or write.matrix()?  I really don't know, but
they may be more efficient than write.table() with large matrices.

Jean


On Tue, Oct 29, 2013 at 2:27 PM, Petar Milin <pmilin@gmail.com> wrote:
> Hello!
> I have a very large matrix of results: 50000x100000. I saved it as RDS,
> but I would also need to save it as txt or csv. Is there a way to do it?
> Now, with write.table I am receiving an error:
> Error in .External2(C_writetable, x, file, nrow(x), p, rnames, sep, eol,  :
>   long vectors not supported yet: io.c:1116
>
> Please, help! Many thanks!
>
> PM
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
	[[alternative HTML version deleted]]

Hello,

You can use the argument to write.csv or write.table  append = TRUE to 
write the matrix in chunks. Something like the following.



bigwrite <- function(x, file, rows = 1000L, ...){
	passes <- NROW(x) %/% rows
	remaining <- NROW(x) %% rows
	k <- 1L
	write.table(x[k:rows, ], file, row.names = FALSE, ...)
	k <- k + rows
	for(i in seq_len(passes)[-1]){
		write.table(x[k:(rows*i), ], file, append = TRUE, row.names = FALSE, 
col.names = FALSE, ...)
		k <- k + rows
	}
	if(remaining > 0)
		write.table(x[k:NROW(x), ], file, append = TRUE, row.names = FALSE, 
col.names = FALSE, ...)
}

f <- "temp"
m <- matrix(0, 50012, 10)

bigwrite(m, f, sep = ",")  # Use 'sep' to get a csv file



Hope this helps,

Rui Barradas


Em 29-10-2013 19:27, Petar Milin escreveu:
> Hello!
> I have a very large matrix of results: 50000x100000. I saved it as RDS, but
I would also need to save it as txt or csv. Is there a way to do it? Now, with
write.table I am receiving an error:
> Error in .External2(C_writetable, x, file, nrow(x), p, rnames, sep, eol,  :
>    long vectors not supported yet: io.c:1116
>
> Please, help! Many thanks!
>
> PM
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>