R help - Jun 2014 - big matrix reading and writing

All,

Got a tricky situation and unfortunately because it's a big file I can't
exactly provide an example, so I'll describe this as best I can for
everyone.

I have a distance matrix that we are using for a modeling calculation in
space for multiple days.  Since the matrix is never going to change for
different dates, I want to keep the matrix in a file and refer to that so I
don't have to repeat the calculation over and over again for that.  The
problem is it's a 32000 X 32000 matrix and roughly works out to 15GB of
storage.  This makes it a trick to read the file back into R, but it leaves
me with two questions for the group.

Is there anyway to have R write this out so that it takes up less space?  I
know R primarily treats numbers as doubles, but I'm trying to find a way to
get R to write the values as floats or singles.

with how big it is, it may not be wise to save it as an object in R when
read in, so I'm wondering is there anyway to have R do the calculation it
needs to do without saving the matrix as an object in R?  Basically can I
have it run the calculation off the file itself?

Thanks!

Adrienne


-- 
Adrienne Wootten
Graduate Research Assistant
State Climate Office of North Carolina
Department of Marine, Earth and Atmospheric Sciences
North Carolina State University

	[[alternative HTML version deleted]]

How are you writing it out now?  Are you using 'save' which will
compress
the file?  What are the range of numbers in the matrix?  Can you scale them
to integers (what is the range of the numbers) which might save some space?
 You did not provide enough information to make a definitive solution.


Jim Holtman
Data Munger Guru

What is the problem that you are trying to solve?
Tell me what you want to do, not how you want to do it.


On Thu, Jun 5, 2014 at 9:26 AM, Adrienne Wootten <amwootte@ncsu.edu>
wrote:
> All,
>
> Got a tricky situation and unfortunately because it's a big file I
can't
> exactly provide an example, so I'll describe this as best I can for
> everyone.
>
> I have a distance matrix that we are using for a modeling calculation in
> space for multiple days.  Since the matrix is never going to change for
> different dates, I want to keep the matrix in a file and refer to that so I
> don't have to repeat the calculation over and over again for that.  The
> problem is it's a 32000 X 32000 matrix and roughly works out to 15GB of
> storage.  This makes it a trick to read the file back into R, but it leaves
> me with two questions for the group.
>
> Is there anyway to have R write this out so that it takes up less space?  I
> know R primarily treats numbers as doubles, but I'm trying to find a
way to
> get R to write the values as floats or singles.
>
> with how big it is, it may not be wise to save it as an object in R when
> read in, so I'm wondering is there anyway to have R do the calculation
it
> needs to do without saving the matrix as an object in R?  Basically can I
> have it run the calculation off the file itself?
>
> Thanks!
>
> Adrienne
>
>
> --
> Adrienne Wootten
> Graduate Research Assistant
> State Climate Office of North Carolina
> Department of Marine, Earth and Atmospheric Sciences
> North Carolina State University
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
	[[alternative HTML version deleted]]