R devel - Aug 2012 - Arrays Partial unserialization

Hi all,

I'm working with some huge array in R and I need to load several ones to 
apply some functions that requires to have all my arrays values for each 
cell...

To make it possible, I would like to load only a part (for example 100 
cells) of all my arrays, apply my function, delete all cells loaded, 
loaded following cells and so on.

Is it possible to unserialize (or load) only a defined part of an R array ?
Do you know some tools that might help me?

Finally, I did lot of research to find the way array (and all other R 
object) are serialized into binary object, but I found nothing 
explaining really algorithms involved. If someone has some information 
on this topic, I'm interesting in.

Hoping my request is understandable,

All the best,

Damien.G

1. I believe this is an R-Help, not an R-devel topic. Post there.

2. It is not clear to me from your post how the arrays are stored -- as
.Rdata files or in some original tabular or database format. I believe the
answer would depend on clarifying that point -- or on others understanding
what I do not.

-- Bert

On Fri, Aug 31, 2012 at 6:47 AM, Damien Georges
<damien.georges2@gmail.com>wrote:
> Hi all,
>
> I'm working with some huge array in R and I need to load several ones
to
> apply some functions that requires to have all my arrays values for each
> cell...
>
> To make it possible, I would like to load only a part (for example 100
> cells) of all my arrays, apply my function, delete all cells loaded, loaded
> following cells and so on.
>
> Is it possible to unserialize (or load) only a defined part of an R array ?
> Do you know some tools that might help me?
>
> Finally, I did lot of research to find the way array (and all other R
> object) are serialized into binary object, but I found nothing explaining
> really algorithms involved. If someone has some information on this topic,
> I'm interesting in.
>
> Hoping my request is understandable,
>
> All the best,
>
> Damien.G
>
> ______________________________**________________
> R-devel@r-project.org mailing list
>
https://stat.ethz.ch/mailman/**listinfo/r-devel<https://stat.ethz.ch/mailman/listinfo/r-devel>
>


-- 

Bert Gunter
Genentech Nonclinical Biostatistics

Internal Contact Info:
Phone: 467-7374
Website:
http://pharmadevelopment.roche.com/index/pdb/pdb-functional-groups/pdb-biostatistics/pdb-ncb-home.htm

	[[alternative HTML version deleted]]

On 31/08/2012 9:47 AM, Damien Georges wrote:
> Hi all,
>
> I'm working with some huge array in R and I need to load several ones
to
> apply some functions that requires to have all my arrays values for each
> cell...
>
> To make it possible, I would like to load only a part (for example 100
> cells) of all my arrays, apply my function, delete all cells loaded,
> loaded following cells and so on.
>
> Is it possible to unserialize (or load) only a defined part of an R array ?
> Do you know some tools that might help me?
I don't know of any tools to do that, but there are tools to maintain 
large objects in files, and load only parts of them at a time, e.g. the 
ff package.  Or you could simply use readBin and writeBin to do the same 
yourself.
>
> Finally, I did lot of research to find the way array (and all other R
> object) are serialized into binary object, but I found nothing
> explaining really algorithms involved. If someone has some information
> on this topic, I'm interesting in.
You can read the source for this; it is in src/main/serialize.c.

Duncan Murdoch
>
> Hoping my request is understandable,
>
> All the best,
>
> Damien.G
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

I'm not an expert in its use, but I believe the bigmemory package offers
the functionality you are looking for (or at least similar functionality
that can be co-opted for your use-case). See the sub.big.matrix function.
Depending on what you mean by "huge" it may offer other benefits as
well.

HTH,
~G

On Fri, Aug 31, 2012 at 6:47 AM, Damien Georges
<damien.georges2@gmail.com>wrote:
> Hi all,
>
> I'm working with some huge array in R and I need to load several ones
to
> apply some functions that requires to have all my arrays values for each
> cell...
>
> To make it possible, I would like to load only a part (for example 100
> cells) of all my arrays, apply my function, delete all cells loaded, loaded
> following cells and so on.
>
> Is it possible to unserialize (or load) only a defined part of an R array ?
> Do you know some tools that might help me?
>
> Finally, I did lot of research to find the way array (and all other R
> object) are serialized into binary object, but I found nothing explaining
> really algorithms involved. If someone has some information on this topic,
> I'm interesting in.
>
> Hoping my request is understandable,
>
> All the best,
>
> Damien.G
>
> ______________________________**________________
> R-devel@r-project.org mailing list
>
https://stat.ethz.ch/mailman/**listinfo/r-devel<https://stat.ethz.ch/mailman/listinfo/r-devel>
>


-- 
Gabriel Becker
Graduate Student
Statistics Department
University of California, Davis

	[[alternative HTML version deleted]]