David Mosberger <david_mosberger@hp.com> writes:
> Hi Peter,
>
> Hope you don't mind my asking you this directly (I wasn't sure if
it
> would be appropriate to post this question to R-devel;
It would.
> if you think it
> is, please feel free to forward it).
Done.
> I'm wondering whether there are any plans to extend R so it could
> handle arbitrarily large objects. For the particular application I
> have in mind, the objects would be several hundred MB in size (in
> uncompressed form).
Some ideas involving "virtual objects" and database interfaces have
been vented on various occasions, but there are no immediate plans. It
*should* happen at some point, I think, but just now we're busy trying
to get the documentation in sync with the implementation and getting
rid of bugs.
> One way I think this could be handled is to leave the basic data types
> presented by R to the user unchanged, but to offer different
> implementation choices for those user data types. For example, an
> "array of structures" is presently loaded into memory in its
entirety
> when accessed. For large data objects, this isn't ideal. An
> alternative implementation would be to store such a big array on disk
> and load into memory only the parts that are really needed. In other
> words, the incore representation would simply be a cache of the entire
> data object. Of course, once this is done you could also vary the
> external representation of objects. For example, instead of storing
> each array element next to each other, it often could be advantageous
> to store the fields of the array next to each other (so that
> operations like "compute the average of the .age field" could be
> performed efficiently). Yet another variation might be to add
> on-the-fly compression/decompression to minimize the size of the
> external data file.
>
> If this approach were taken, I'd imagine that R would continue to use
> the "store entirely in memory" approach by default to maintain
> backwards compatibility. At the same time, a few new functions could
> be introduced that would allow precise control over how the object is
> implemented. So when the user wants to deal with a large object, it
> would create the object, set its implementation to something suitable
> (e.g., cache-only, field-sequential layout, on-the-fly compression)
> and then continue to use the object as usual.
>
> Since I'm not familiar with the internals of R, I have no idea how
> easy/hard this would be and I'd therefore appreciate hearing your
> opinion on whether you think this would be a valuable and doable
> extension.
>
> In any case, thanks for working on R! I was excited to find that I
> now have to option to use the S language on my Linux systems!
>
> Cheers,
>
> --david
>
> --
> David Mosberger, Ph.D; HP Labs; 1501 Page Mill Rd MS 1U17; Palo Alto, CA
94304
> davidm@hpl.hp.com voice (650) 236-2575 fax
857-5100
>
--
O__ ---- Peter Dalgaard Blegdamsvej 3
c/ /'_ --- Dept. of Biostatistics 2200 Cph. N
(*) \(*) -- University of Copenhagen Denmark Ph: (+45) 35327918
~~~~~~~~~~ - (p.dalgaard@biostat.ku.dk) FAX: (+45) 35327907
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !) To:
r-devel-request@stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
