Andreas Kersting
2021-Apr-07 15:09 UTC
[Rd] [External] memory consumption of nested (un)serialize of sys.frames()
Hi Dirk, hi Luke, Thanks for checking! I could narrow it down further. I have the issue only if I install --with-keep.source, i.e. R CMD INSTALL --with-keep.source dumpTest Since this is the default in RStudio when clicking "Install and Restart", I was always having the issue - also from base R. If I install using e.g. devtools::install_github() directly it is also fine for me. Could you please confirm? Thanks! Regards, Andreas 2021-04-07 16:20 GMT+02:00 "Dirk Eddelbuettel" <edd at debian.org>:> > On 7 April 2021 at 16:06, Andreas Kersting wrote: > | Hi Luke, > | > | Please see https://github.com/akersting/dumpTest for the package. > | > | Here a session showing my issue: > | > | > library(dumpTest) > | > sessionInfo() > | R version 4.0.5 (2021-03-31) > | Platform: x86_64-pc-linux-gnu (64-bit) > | Running under: Debian GNU/Linux 10 (buster) > | > | Matrix products: default > | BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.8.0 > | LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.8.0 > | > | locale: > | [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C > | [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 > | [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 > | [7] LC_PAPER=en_US.UTF-8 LC_NAME=C > | [9] LC_ADDRESS=C LC_TELEPHONE=C > | [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C > | > | attached base packages: > | [1] stats graphics grDevices utils datasets methods base > | > | other attached packages: > | [1] dumpTest_0.1.0 > | > | loaded via a namespace (and not attached): > | [1] compiler_4.0.5 > | > for (i in 1:100) { > | + print(i) > | + print(system.time(f())) > | + } > | [1] 1 > | user system elapsed > | 0.028 0.004 0.034 > | [1] 2 > | user system elapsed > | 0.067 0.008 0.075 > | [1] 3 > | user system elapsed > | 0.176 0.000 0.176 > | [1] 4 > | user system elapsed > | 0.335 0.012 0.349 > | [1] 5 > | user system elapsed > | 0.745 0.023 0.770 > | [1] 6 > | user system elapsed > | 1.495 0.060 1.572 > | [1] 7 > | user system elapsed > | 2.902 0.136 3.040 > | [1] 8 > | user system elapsed > | 5.753 0.272 6.034 > | [1] 9 > | user system elapsed > | 11.807 0.708 12.597 > | [1] 10 > | ^C > | Timing stopped at: 6.638 0.549 7.214 > | > | I had to interrupt in iteration 10 because I was running low on RAM. > > No issue here. Ubuntu 20.10, R 4.0.5 'from CRAN' i.e. Michael's PPA build > off my Debian package, hence instrumentation as in the Debian package. > > edd at rob:~$ installGithub.r akersting/dumpTest > Using github PAT from envvar GITHUB_PAT > Downloading GitHub repo akersting/dumpTest at HEAD > ? checking for file ?/tmp/remotes3f9af733166ccd/akersting-dumpTest-3bed8e2/DESCRIPTION? ... > ? preparing ?dumpTest?: > ? checking DESCRIPTION meta-information ... > ? checking for LF line-endings in source and make files and shell scripts > ? checking for empty or unneeded directories > ? building ?dumpTest_0.1.0.tar.gz? > > Installing package into ?/usr/local/lib/R/site-library? > (as ?lib? is unspecified) > * installing *source* package ?dumpTest? ... > ** using staged installation > ** R > ** byte-compile and prepare package for lazy loading > ** help > No man pages found in package ?dumpTest? > *** installing help indices > ** building package indices > ** testing if installed package can be loaded from temporary location > ** testing if installed package can be loaded from final location > ** testing if installed package keeps a record of temporary installation path > * DONE (dumpTest) > edd at rob:~$ Rscript -e 'system.time({for (i in 1:100) dumpTest::f()})' > user system elapsed > 0.481 0.019 0.500 > edd at rob:~$ > > (I also ran the variant you showed with the dual print statements, it just > consumes more screen real estate and ends on > > [...] > [1] 97 > user system elapsed > 0.004 0.000 0.005 > [1] 98 > user system elapsed > 0.004 0.000 0.005 > [1] 99 > user system elapsed > 0.004 0.000 0.004 > [1] 100 > user system elapsed > 0.005 0.000 0.005 > edd at rob:~$ ) > > Dirk > > -- > https://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org >
Andreas Kersting
2021-Apr-08 08:45 UTC
[Rd] [External] memory consumption of nested (un)serialize of sys.frames()
Hi,
For (hopefully) full reproducibility:
docker run rocker/tidyverse:4.0.5 Rscript -e
'devtools::install_github("akersting/dumpTest", INSTALL_opts =
"--with-keep.source"); library(dumpTest); for (i in 1:100) {print(i);
print(system.time(f()))}'
Regards,
Andreas
2021-04-07 17:09 GMT+02:00 "Andreas Kersting" <r-devel at
akersting.de>:> Hi Dirk, hi Luke,
>
> Thanks for checking!
>
> I could narrow it down further. I have the issue only if I install
--with-keep.source, i.e.
>
> R CMD INSTALL --with-keep.source dumpTest
>
> Since this is the default in RStudio when clicking "Install and
Restart", I was always having the issue - also from base R. If I install
using e.g. devtools::install_github() directly it is also fine for me.
>
> Could you please confirm? Thanks!
>
> Regards,
> Andreas
>
> 2021-04-07 16:20 GMT+02:00 "Dirk Eddelbuettel" <edd at
debian.org>:
>>
>> On 7 April 2021 at 16:06, Andreas Kersting wrote:
>> | Hi Luke,
>> |
>> | Please see https://github.com/akersting/dumpTest for the package.
>> |
>> | Here a session showing my issue:
>> |
>> | > library(dumpTest)
>> | > sessionInfo()
>> | R version 4.0.5 (2021-03-31)
>> | Platform: x86_64-pc-linux-gnu (64-bit)
>> | Running under: Debian GNU/Linux 10 (buster)
>> |
>> | Matrix products: default
>> | BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.8.0
>> | LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.8.0
>> |
>> | locale:
>> | [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
>> | [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
>> | [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
>> | [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
>> | [9] LC_ADDRESS=C LC_TELEPHONE=C
>> | [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>> |
>> | attached base packages:
>> | [1] stats graphics grDevices utils datasets methods base
>> |
>> | other attached packages:
>> | [1] dumpTest_0.1.0
>> |
>> | loaded via a namespace (and not attached):
>> | [1] compiler_4.0.5
>> | > for (i in 1:100) {
>> | + print(i)
>> | + print(system.time(f()))
>> | + }
>> | [1] 1
>> | user system elapsed
>> | 0.028 0.004 0.034
>> | [1] 2
>> | user system elapsed
>> | 0.067 0.008 0.075
>> | [1] 3
>> | user system elapsed
>> | 0.176 0.000 0.176
>> | [1] 4
>> | user system elapsed
>> | 0.335 0.012 0.349
>> | [1] 5
>> | user system elapsed
>> | 0.745 0.023 0.770
>> | [1] 6
>> | user system elapsed
>> | 1.495 0.060 1.572
>> | [1] 7
>> | user system elapsed
>> | 2.902 0.136 3.040
>> | [1] 8
>> | user system elapsed
>> | 5.753 0.272 6.034
>> | [1] 9
>> | user system elapsed
>> | 11.807 0.708 12.597
>> | [1] 10
>> | ^C
>> | Timing stopped at: 6.638 0.549 7.214
>> |
>> | I had to interrupt in iteration 10 because I was running low on RAM.
>>
>> No issue here. Ubuntu 20.10, R 4.0.5 'from CRAN' i.e.
Michael's PPA build
>> off my Debian package, hence instrumentation as in the Debian package.
>>
>> edd at rob:~$ installGithub.r akersting/dumpTest
>> Using github PAT from envvar GITHUB_PAT
>> Downloading GitHub repo akersting/dumpTest at HEAD
>> ? checking for file
?/tmp/remotes3f9af733166ccd/akersting-dumpTest-3bed8e2/DESCRIPTION? ...
>> ? preparing ?dumpTest?:
>> ? checking DESCRIPTION meta-information ...
>> ? checking for LF line-endings in source and make files and shell
scripts
>> ? checking for empty or unneeded directories
>> ? building ?dumpTest_0.1.0.tar.gz?
>>
>> Installing package into ?/usr/local/lib/R/site-library?
>> (as ?lib? is unspecified)
>> * installing *source* package ?dumpTest? ...
>> ** using staged installation
>> ** R
>> ** byte-compile and prepare package for lazy loading
>> ** help
>> No man pages found in package ?dumpTest?
>> *** installing help indices
>> ** building package indices
>> ** testing if installed package can be loaded from temporary location
>> ** testing if installed package can be loaded from final location
>> ** testing if installed package keeps a record of temporary
installation path
>> * DONE (dumpTest)
>> edd at rob:~$ Rscript -e 'system.time({for (i in 1:100)
dumpTest::f()})'
>> user system elapsed
>> 0.481 0.019 0.500
>> edd at rob:~$
>>
>> (I also ran the variant you showed with the dual print statements, it
just
>> consumes more screen real estate and ends on
>>
>> [...]
>> [1] 97
>> user system elapsed
>> 0.004 0.000 0.005
>> [1] 98
>> user system elapsed
>> 0.004 0.000 0.005
>> [1] 99
>> user system elapsed
>> 0.004 0.000 0.004
>> [1] 100
>> user system elapsed
>> 0.005 0.000 0.005
>> edd at rob:~$ )
>>
>> Dirk
>>
>> --
>> https://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
>>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
iuke-tier@ey m@iii@g oii uiow@@edu
2021-Apr-08 14:15 UTC
[Rd] [External] memory consumption of nested (un)serialize of sys.frames()
I see that now also. Not sure yet what is going on. One work-around that may work for you is to create a fresh crash dump in a .onLoad function; somehting like crash_dumps <- NULL .onLoad <- function(...) crash_dumps <<- new.env() Best, luke On Wed, 7 Apr 2021, Andreas Kersting wrote:> Hi Dirk, hi Luke, > > Thanks for checking! > > I could narrow it down further. I have the issue only if I install --with-keep.source, i.e. > > R CMD INSTALL --with-keep.source dumpTest > > Since this is the default in RStudio when clicking "Install and Restart", I was always having the issue - also from base R. If I install using e.g. devtools::install_github() directly it is also fine for me. > > Could you please confirm? Thanks! > > Regards, > Andreas > > 2021-04-07 16:20 GMT+02:00 "Dirk Eddelbuettel" <edd at debian.org>: >> >> On 7 April 2021 at 16:06, Andreas Kersting wrote: >> | Hi Luke, >> | >> | Please see https://github.com/akersting/dumpTest for the package. >> | >> | Here a session showing my issue: >> | >> | > library(dumpTest) >> | > sessionInfo() >> | R version 4.0.5 (2021-03-31) >> | Platform: x86_64-pc-linux-gnu (64-bit) >> | Running under: Debian GNU/Linux 10 (buster) >> | >> | Matrix products: default >> | BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.8.0 >> | LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.8.0 >> | >> | locale: >> | [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C >> | [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 >> | [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 >> | [7] LC_PAPER=en_US.UTF-8 LC_NAME=C >> | [9] LC_ADDRESS=C LC_TELEPHONE=C >> | [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C >> | >> | attached base packages: >> | [1] stats graphics grDevices utils datasets methods base >> | >> | other attached packages: >> | [1] dumpTest_0.1.0 >> | >> | loaded via a namespace (and not attached): >> | [1] compiler_4.0.5 >> | > for (i in 1:100) { >> | + print(i) >> | + print(system.time(f())) >> | + } >> | [1] 1 >> | user system elapsed >> | 0.028 0.004 0.034 >> | [1] 2 >> | user system elapsed >> | 0.067 0.008 0.075 >> | [1] 3 >> | user system elapsed >> | 0.176 0.000 0.176 >> | [1] 4 >> | user system elapsed >> | 0.335 0.012 0.349 >> | [1] 5 >> | user system elapsed >> | 0.745 0.023 0.770 >> | [1] 6 >> | user system elapsed >> | 1.495 0.060 1.572 >> | [1] 7 >> | user system elapsed >> | 2.902 0.136 3.040 >> | [1] 8 >> | user system elapsed >> | 5.753 0.272 6.034 >> | [1] 9 >> | user system elapsed >> | 11.807 0.708 12.597 >> | [1] 10 >> | ^C >> | Timing stopped at: 6.638 0.549 7.214 >> | >> | I had to interrupt in iteration 10 because I was running low on RAM. >> >> No issue here. Ubuntu 20.10, R 4.0.5 'from CRAN' i.e. Michael's PPA build >> off my Debian package, hence instrumentation as in the Debian package. >> >> edd at rob:~$ installGithub.r akersting/dumpTest >> Using github PAT from envvar GITHUB_PAT >> Downloading GitHub repo akersting/dumpTest at HEAD >> ? checking for file ?/tmp/remotes3f9af733166ccd/akersting-dumpTest-3bed8e2/DESCRIPTION? ... >> ? preparing ?dumpTest?: >> ? checking DESCRIPTION meta-information ... >> ? checking for LF line-endings in source and make files and shell scripts >> ? checking for empty or unneeded directories >> ? building ?dumpTest_0.1.0.tar.gz? >> >> Installing package into ?/usr/local/lib/R/site-library? >> (as ?lib? is unspecified) >> * installing *source* package ?dumpTest? ... >> ** using staged installation >> ** R >> ** byte-compile and prepare package for lazy loading >> ** help >> No man pages found in package ?dumpTest? >> *** installing help indices >> ** building package indices >> ** testing if installed package can be loaded from temporary location >> ** testing if installed package can be loaded from final location >> ** testing if installed package keeps a record of temporary installation path >> * DONE (dumpTest) >> edd at rob:~$ Rscript -e 'system.time({for (i in 1:100) dumpTest::f()})' >> user system elapsed >> 0.481 0.019 0.500 >> edd at rob:~$ >> >> (I also ran the variant you showed with the dual print statements, it just >> consumes more screen real estate and ends on >> >> [...] >> [1] 97 >> user system elapsed >> 0.004 0.000 0.005 >> [1] 98 >> user system elapsed >> 0.004 0.000 0.005 >> [1] 99 >> user system elapsed >> 0.004 0.000 0.004 >> [1] 100 >> user system elapsed >> 0.005 0.000 0.005 >> edd at rob:~$ ) >> >> Dirk >> >> -- >> https://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org >>-- Luke Tierney Ralph E. Wareham Professor of Mathematical Sciences University of Iowa Phone: 319-335-3386 Department of Statistics and Fax: 319-335-3017 Actuarial Science 241 Schaeffer Hall email: luke-tierney at uiowa.edu Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu