Dirk Eddelbuettel
2021-Apr-07 14:20 UTC
[Rd] [External] memory consumption of nested (un)serialize of sys.frames()
On 7 April 2021 at 16:06, Andreas Kersting wrote:
| Hi Luke,
|
| Please see https://github.com/akersting/dumpTest for the package.
|
| Here a session showing my issue:
|
| > library(dumpTest)
| > sessionInfo()
| R version 4.0.5 (2021-03-31)
| Platform: x86_64-pc-linux-gnu (64-bit)
| Running under: Debian GNU/Linux 10 (buster)
|
| Matrix products: default
| BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.8.0
| LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.8.0
|
| locale:
| [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
| [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
| [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
| [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
| [9] LC_ADDRESS=C LC_TELEPHONE=C
| [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
|
| attached base packages:
| [1] stats graphics grDevices utils datasets methods base
|
| other attached packages:
| [1] dumpTest_0.1.0
|
| loaded via a namespace (and not attached):
| [1] compiler_4.0.5
| > for (i in 1:100) {
| + print(i)
| + print(system.time(f()))
| + }
| [1] 1
| user system elapsed
| 0.028 0.004 0.034
| [1] 2
| user system elapsed
| 0.067 0.008 0.075
| [1] 3
| user system elapsed
| 0.176 0.000 0.176
| [1] 4
| user system elapsed
| 0.335 0.012 0.349
| [1] 5
| user system elapsed
| 0.745 0.023 0.770
| [1] 6
| user system elapsed
| 1.495 0.060 1.572
| [1] 7
| user system elapsed
| 2.902 0.136 3.040
| [1] 8
| user system elapsed
| 5.753 0.272 6.034
| [1] 9
| user system elapsed
| 11.807 0.708 12.597
| [1] 10
| ^C
| Timing stopped at: 6.638 0.549 7.214
|
| I had to interrupt in iteration 10 because I was running low on RAM.
No issue here. Ubuntu 20.10, R 4.0.5 'from CRAN' i.e. Michael's PPA
build
off my Debian package, hence instrumentation as in the Debian package.
edd at rob:~$ installGithub.r akersting/dumpTest
Using github PAT from envvar GITHUB_PAT
Downloading GitHub repo akersting/dumpTest at HEAD
? checking for file
?/tmp/remotes3f9af733166ccd/akersting-dumpTest-3bed8e2/DESCRIPTION? ...
? preparing ?dumpTest?:
? checking DESCRIPTION meta-information ...
? checking for LF line-endings in source and make files and shell scripts
? checking for empty or unneeded directories
? building ?dumpTest_0.1.0.tar.gz?
Installing package into ?/usr/local/lib/R/site-library?
(as ?lib? is unspecified)
* installing *source* package ?dumpTest? ...
** using staged installation
** R
** byte-compile and prepare package for lazy loading
** help
No man pages found in package ?dumpTest?
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (dumpTest)
edd at rob:~$ Rscript -e 'system.time({for (i in 1:100) dumpTest::f()})'
user system elapsed
0.481 0.019 0.500
edd at rob:~$
(I also ran the variant you showed with the dual print statements, it just
consumes more screen real estate and ends on
[...]
[1] 97
user system elapsed
0.004 0.000 0.005
[1] 98
user system elapsed
0.004 0.000 0.005
[1] 99
user system elapsed
0.004 0.000 0.004
[1] 100
user system elapsed
0.005 0.000 0.005
edd at rob:~$ )
Dirk
--
https://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
Andreas Kersting
2021-Apr-07 15:09 UTC
[Rd] [External] memory consumption of nested (un)serialize of sys.frames()
Hi Dirk, hi Luke, Thanks for checking! I could narrow it down further. I have the issue only if I install --with-keep.source, i.e. R CMD INSTALL --with-keep.source dumpTest Since this is the default in RStudio when clicking "Install and Restart", I was always having the issue - also from base R. If I install using e.g. devtools::install_github() directly it is also fine for me. Could you please confirm? Thanks! Regards, Andreas 2021-04-07 16:20 GMT+02:00 "Dirk Eddelbuettel" <edd at debian.org>:> > On 7 April 2021 at 16:06, Andreas Kersting wrote: > | Hi Luke, > | > | Please see https://github.com/akersting/dumpTest for the package. > | > | Here a session showing my issue: > | > | > library(dumpTest) > | > sessionInfo() > | R version 4.0.5 (2021-03-31) > | Platform: x86_64-pc-linux-gnu (64-bit) > | Running under: Debian GNU/Linux 10 (buster) > | > | Matrix products: default > | BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.8.0 > | LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.8.0 > | > | locale: > | [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C > | [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 > | [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 > | [7] LC_PAPER=en_US.UTF-8 LC_NAME=C > | [9] LC_ADDRESS=C LC_TELEPHONE=C > | [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C > | > | attached base packages: > | [1] stats graphics grDevices utils datasets methods base > | > | other attached packages: > | [1] dumpTest_0.1.0 > | > | loaded via a namespace (and not attached): > | [1] compiler_4.0.5 > | > for (i in 1:100) { > | + print(i) > | + print(system.time(f())) > | + } > | [1] 1 > | user system elapsed > | 0.028 0.004 0.034 > | [1] 2 > | user system elapsed > | 0.067 0.008 0.075 > | [1] 3 > | user system elapsed > | 0.176 0.000 0.176 > | [1] 4 > | user system elapsed > | 0.335 0.012 0.349 > | [1] 5 > | user system elapsed > | 0.745 0.023 0.770 > | [1] 6 > | user system elapsed > | 1.495 0.060 1.572 > | [1] 7 > | user system elapsed > | 2.902 0.136 3.040 > | [1] 8 > | user system elapsed > | 5.753 0.272 6.034 > | [1] 9 > | user system elapsed > | 11.807 0.708 12.597 > | [1] 10 > | ^C > | Timing stopped at: 6.638 0.549 7.214 > | > | I had to interrupt in iteration 10 because I was running low on RAM. > > No issue here. Ubuntu 20.10, R 4.0.5 'from CRAN' i.e. Michael's PPA build > off my Debian package, hence instrumentation as in the Debian package. > > edd at rob:~$ installGithub.r akersting/dumpTest > Using github PAT from envvar GITHUB_PAT > Downloading GitHub repo akersting/dumpTest at HEAD > ? checking for file ?/tmp/remotes3f9af733166ccd/akersting-dumpTest-3bed8e2/DESCRIPTION? ... > ? preparing ?dumpTest?: > ? checking DESCRIPTION meta-information ... > ? checking for LF line-endings in source and make files and shell scripts > ? checking for empty or unneeded directories > ? building ?dumpTest_0.1.0.tar.gz? > > Installing package into ?/usr/local/lib/R/site-library? > (as ?lib? is unspecified) > * installing *source* package ?dumpTest? ... > ** using staged installation > ** R > ** byte-compile and prepare package for lazy loading > ** help > No man pages found in package ?dumpTest? > *** installing help indices > ** building package indices > ** testing if installed package can be loaded from temporary location > ** testing if installed package can be loaded from final location > ** testing if installed package keeps a record of temporary installation path > * DONE (dumpTest) > edd at rob:~$ Rscript -e 'system.time({for (i in 1:100) dumpTest::f()})' > user system elapsed > 0.481 0.019 0.500 > edd at rob:~$ > > (I also ran the variant you showed with the dual print statements, it just > consumes more screen real estate and ends on > > [...] > [1] 97 > user system elapsed > 0.004 0.000 0.005 > [1] 98 > user system elapsed > 0.004 0.000 0.005 > [1] 99 > user system elapsed > 0.004 0.000 0.004 > [1] 100 > user system elapsed > 0.005 0.000 0.005 > edd at rob:~$ ) > > Dirk > > -- > https://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org >