R help - Apr 2023 - Matrix scalar operation that saves memory?

Hi all,

I am currently working with a quite large matrix that takes 300G of 
memory. My computer only has 512G of memory. I would need to do a few 
arithmetic on it with a scalar value. My current code looks like this:

mat <- 100 - mat

However such code quickly uses up all of the remaining memory and got 
the R script killed by OOM killer.

Are there any more memory-efficient way of doing such operation?

Thanks,

S. Zhang

	[[alternative HTML version deleted]]

The example given does not leave room for even a single copy of your matrix
so, yes, you need alternatives.

Your example was fairly trivial as all you wanted to do is subtract each
value from 100 and replace it. Obviously something like squaring a matrix
has no trivial way to do without multiple copies out there that won't fit.

One technique that might work is a nested loop that changes one cell of the
matrix at a time and in-place. A variant of this might be a singe loop that
changes a single row (or column) at a time and in place.

Another odd concept is to save your matrix in a file with some format you
can read back in such as a line or row at a time, and then do the
subtraction from 100 and write it back to disk in another file. If you need
it again, I assume you can read it in but perhaps you should consider how to
increase some aspects of your "memory".

Is your matrix a real matrix type or something like a list of lists or a
data.frame? You may do better with some data structures that are more
efficient than others.

Some OS allow you to use virtual memory that is mapped in and out from the
disk that allows larger things to be done, albeit often much slower. I also
note that you can remove some things you are not using and hope garbage
collection happens soon enough.

-----Original Message-----
From: R-help <r-help-bounces at r-project.org> On Behalf Of Shunran Zhang
Sent: Tuesday, April 11, 2023 10:21 PM
To: r-help at r-project.org
Subject: [R] Matrix scalar operation that saves memory?

Hi all,

I am currently working with a quite large matrix that takes 300G of 
memory. My computer only has 512G of memory. I would need to do a few 
arithmetic on it with a scalar value. My current code looks like this:

mat <- 100 - mat

However such code quickly uses up all of the remaining memory and got 
the R script killed by OOM killer.

Are there any more memory-efficient way of doing such operation?

Thanks,

S. Zhang

	[[alternative HTML version deleted]]

______________________________________________
R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

I doubt that R's basic matrix capabilities can handle this, but have a look
at the Matrix package, especially if your matrix is some special form.

Bert

On Tue, Apr 11, 2023, 19:21 Shunran Zhang <szhang at
ngs.gen-info.osaka-u.ac.jp>
wrote:
> Hi all,
>
> I am currently working with a quite large matrix that takes 300G of
> memory. My computer only has 512G of memory. I would need to do a few
> arithmetic on it with a scalar value. My current code looks like this:
>
> mat <- 100 - mat
>
> However such code quickly uses up all of the remaining memory and got
> the R script killed by OOM killer.
>
> Are there any more memory-efficient way of doing such operation?
>
> Thanks,
>
> S. Zhang
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
	[[alternative HTML version deleted]]