R devel - Apr 2021 - [External] memory consumption of nested (un)serialize of sys.frames()

Hi,

please consider the following minimal reproducible example:

Create a new R package which just contains the following two (exported) objects:


crash_dumps <- new.env()

f <- function() {
  x <- runif(1e5)
  dump <- lapply(1:2, function(i) unserialize(serialize(sys.frames(), NULL)))
  assign("last.dump", dump, crash_dumps)
}


WARNING: the following will probably eat all your RAM!

Attach this package and run:

for (i in 1:100) {
  print(i)
  f()
}

You will notice that with each iteration the execution of f() slows down
significantly while the memory consumption of the R process (v4.0.5 on Linux)
quickly explodes.

I am having a hard time to understand what exactly is happening here. Something
w.r.t. too deeply nested environments? Could someone please enlighten me?
Thanks!

Regards,
Andreas


Background:
In an R package I store crash dumps on error in a parallel processes in a way
similar to what I have just shown (hence the (un)serialize(), which happens as
part of returning the objects to the parent process). The first 2 or 3 times I
do so in a session everything is fine, but afterwards it takes very long and I
soon run out of memory.

Some more observations:
- If I omit `x <- runif(1e5)`, the issues seem to be less pronounced.
- If I assign to .GlobalEnv instead of crash_dumps, there seems to be no issue -
probably because .GlobalEnv is not included in sys.frames(), while crash_dumps
is indirectly via the namespace of the package being the parent.env of some of
the sys.frames()!?
- If I omit the lapply(...), i.e. use `dump <-
unserialize(serialize(sys.frames(), NULL))` directly, there seems to be no
issue. The immediate consequence is that there are less sys.frames and - in
particular - there is no frame which has the base namespace as its parent.env.
- If I make crash_dumps a list and use assignInMyNamespace() to store the dump
in it, there also seems to be no issue. I will probably use this as a
workaround:

crash_dumps <- list()

f <- function() {
  x <- runif(1e5)
  dump <- lapply(1:2, function(i) unserialize(serialize(sys.frames(), NULL)))
  crash_dumps[["last.dump"]] <- dump
  assignInMyNamespace("crash_dumps", crash_dumps)
}

On Wed, 7 Apr 2021, Andreas Kersting wrote:
> Hi,
>
> please consider the following minimal reproducible example:
>
> Create a new R package which just contains the following two (exported)
objects:
I would not expect this behavior and I don't see it when I make such a
package (in R 4.0.3 or R-devel on Ubuntu).  You will need to provide a
more complete reproducible example if you want help with what you are
trying to do; also sessionInfo() would help.

Best,

luke
>
>
> crash_dumps <- new.env()
>
> f <- function() {
>  x <- runif(1e5)
>  dump <- lapply(1:2, function(i) unserialize(serialize(sys.frames(),
NULL)))
>  assign("last.dump", dump, crash_dumps)
> }
>
>
> WARNING: the following will probably eat all your RAM!
>
> Attach this package and run:
>
> for (i in 1:100) {
>  print(i)
>  f()
> }
>
> You will notice that with each iteration the execution of f() slows down
significantly while the memory consumption of the R process (v4.0.5 on Linux)
quickly explodes.
>
> I am having a hard time to understand what exactly is happening here.
Something w.r.t. too deeply nested environments? Could someone please enlighten
me? Thanks!
>
> Regards,
> Andreas
>
>
> Background:
> In an R package I store crash dumps on error in a parallel processes in a
way similar to what I have just shown (hence the (un)serialize(), which happens
as part of returning the objects to the parent process). The first 2 or 3 times
I do so in a session everything is fine, but afterwards it takes very long and I
soon run out of memory.
>
> Some more observations:
> - If I omit `x <- runif(1e5)`, the issues seem to be less pronounced.
> - If I assign to .GlobalEnv instead of crash_dumps, there seems to be no
issue - probably because .GlobalEnv is not included in sys.frames(), while
crash_dumps is indirectly via the namespace of the package being the parent.env
of some of the sys.frames()!?
> - If I omit the lapply(...), i.e. use `dump <-
unserialize(serialize(sys.frames(), NULL))` directly, there seems to be no
issue. The immediate consequence is that there are less sys.frames and - in
particular - there is no frame which has the base namespace as its parent.env.
> - If I make crash_dumps a list and use assignInMyNamespace() to store the
dump in it, there also seems to be no issue. I will probably use this as a
workaround:
>
> crash_dumps <- list()
>
> f <- function() {
>  x <- runif(1e5)
>  dump <- lapply(1:2, function(i) unserialize(serialize(sys.frames(),
NULL)))
>  crash_dumps[["last.dump"]] <- dump
>  assignInMyNamespace("crash_dumps", crash_dumps)
> }
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
-- 
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa                  Phone:             319-335-3386
Department of Statistics and        Fax:               319-335-3017
    Actuarial Science
241 Schaeffer Hall                  email:   luke-tierney at uiowa.edu
Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu