R help - Dec 2001 - write.table and large datasets

Hi,

I'll continue the discussion about the write.table() and problems with large
datasets.

The databases I have to work with are quite huge, 7500 obs x 1200 vars were
on of the smallest of them.  I usually write a perl script to preprocess
them line-by-line and extract only the variables which I need later.  This
results into quite a manageable size but I have to have the dataset in ASCII
form.

As it appears, R could easily read in a dataset of that size (it was in
STATA format) but the conversion to ascii was quite complicated.  I
remember, I have done it with a larger dataset, using SPSS and a rougly
equal computer (128M memory).  It did not took much time, perhaps a minute
or so.  That's why I was surprised that apparently similar function in R was
unable to save the dataset at all.

I still think that it would be a good idea to be able to transform big
datasets to ascii, it is perhaps the most simple and open standard which I
think everybody else can use too.  Perhaps it is more meaningful to make a
less-general function (in package external) which can save at least numeric
and character variables?  I have made some attempts myself but these are
still in very early stage.

So, what is the general opinion -- is such kind of a conversion utility
needed in R?

Regards,

Ott Toomet


-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

There already is such a function, write.matrix in package MASS. As from
the next release it can work in blocks of rows as well.  It's already
cross-referenced in R-patched and R-devel.

BDR

On Thu, 27 Dec 2001, Ott Toomet wrote:
> Hi,
>
> I'll continue the discussion about the write.table() and problems with
large
> datasets.
>
> The databases I have to work with are quite huge, 7500 obs x 1200 vars were
> on of the smallest of them.  I usually write a perl script to preprocess
> them line-by-line and extract only the variables which I need later.  This
> results into quite a manageable size but I have to have the dataset in
ASCII
> form.
>
> As it appears, R could easily read in a dataset of that size (it was in
> STATA format) but the conversion to ascii was quite complicated.  I
> remember, I have done it with a larger dataset, using SPSS and a rougly
> equal computer (128M memory).  It did not took much time, perhaps a minute
> or so.  That's why I was surprised that apparently similar function in
R was
> unable to save the dataset at all.
>
> I still think that it would be a good idea to be able to transform big
> datasets to ascii, it is perhaps the most simple and open standard which I
> think everybody else can use too.  Perhaps it is more meaningful to make a
> less-general function (in package external) which can save at least numeric
> and character variables?  I have made some attempts myself but these are
> still in very early stage.
>
> So, what is the general opinion -- is such kind of a conversion utility
> needed in R?
-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272860 (secr)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._