R help - Jul 2002 - arrays in binary files

Dear all,

In the writing of functions for a package have to deal with (very)
 large arrays and I was considering that
having them in files rather than in memory would help me a bit
on the memory side of the things. I was thinking of accessing the
elements in the arrays by 'seek' and friends in C. 
The package RObjectTables seems to provide the framework to do this
kind of thing, but I have questions:

- The speed side of the things worries me. Is the idea totally
stupid ?

- Does anybody out there have experience in doing similar things
(and why not dream about it, doing the exact same thing ;) ) with
R ?


Thanking any kind of help/pointer in advance,


L.


PS: I know of the package 'rhdf5'. It provides what I am looking for,
but I'd like to offer a poor man's solution (should I say 'lazy
man')...
the package should work 'out of the box'
 

 


-- 
--------------------------------------------------------------
Laurent Gautier			CBS, Building 208, DTU
PhD. Student			DK-2800 Lyngby,Denmark	
tel: +45 45 25 24 85		http://www.cbs.dtu.dk/laurent
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

>In the writing of functions for a package have to deal with (very)
> large arrays and I was considering that
>having them in files rather than in memory would help me a bit
>on the memory side of the things. I was thinking of accessing the
>elements in the arrays by 'seek' and friends in C. 
My gut feeling is that this is a bad thing to do.  Unless you go to a
lot of trouble to optimize, it's generally better to let the operating
system create "virtual memory" and handle the seeking and such.

The main case where this is not true is the case of large static
arrays that change little from run to run, of which you only use a
small proportion of the entries.  Unless you can do memory mapped
files, you'll have to read the whole file into virtual memory on each
run, and that's slow.  If you're accessing every element anyway, then
it's not such a penalty to read the whole thing at the start; if they
are temporary for the run, or change frequently, then you'll be
writing the whole thing every time anyway.

Duncan Murdoch
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._