thr3ads.net - R help - [R] Maximum # of DLLs reached, or, how to clean up after yourself? [Sep 2016]

If this information is useful, please help other people find it:
Share via:

Alexander Shenkin

2016-Sep-14 08:49 UTC

[R] Maximum # of DLLs reached, or, how to clean up after yourself?

Hi Henrik,

Thanks for your reply.  I didn't realize that floating DLLs were an 
issue (good to know).  My query is actually a bit more basic.  I'm 
actually wondering how folks manage their loading and unloading of 
packages when calling scripts within scripts.

Quick example:
Script1:
	library(package1)
	source("script2.r")
	# do stuff reliant on package1
	detach("package:package1", unload=TRUE)

Script2:
	library(package1)
	library(package2)
	# do stuff reliant on package1 and package2
	detach("package:package1", unload=TRUE)
	detach("package:package2", unload=TRUE)

Script2 breaks Script1 by unloading package1 (though unloading package2 
is ok).  I will have to test whether each package is loaded in Script2 
before loading it, and use that list when unloading at the end of the 
Script2.  *Unless there's a better way to do it* (which is my question - 
is there?).  I'm possibly just pushing the whole procedural scripting 
thing too far, but I also think that this likely isn't uncommon in R.

Any thoughts greatly appreciated!

Thanks,
Allie

On 9/13/2016 7:23 PM, Henrik Bengtsson wrote:> In R.utils (>= 2.4.0), which I hope to submitted to CRAN today or
> tomorrow, you can simply call:
>
>    R.utils::gcDLLs()
>
> It will look at base::getLoadedDLLs() and its content and compare to
> loadedNamespaces() and unregister any "stray" DLLs that remain
after
> corresponding packages have been unloaded.
>
> Until the new version is on CRAN, you can install it via
>
>     source("http://callr.org/install#HenrikBengtsson/R.utils at
develop")
>
> or alternatively just source() the source file:
>
>    
source("https://raw.githubusercontent.com/HenrikBengtsson/R.utils/develop/R/gcDLLs.R")
>
>
> DISCUSSION:
> (this might be better suited for R-devel; feel free to move it there)
>
> As far as I understand the problem, running into this error / limit is
> _not_ the fault of the user.  Instead, I'd argue that it is the
> responsibility of package developers to make sure to unregister any
> registered DLLs of theirs when the package is unloaded.  A developer
> can do this by adding the following to their package:
>
> .onUnload <- function(libpath) {
>     library.dynam.unload(utils::packageName(), libpath)
>  }
>
> That should be all - then the DLL will be unloaded as soon as the
> package is unloaded.
>
> I would like to suggest that 'R CMD check' would include a check
that
> asserts when a package is unloaded it does not leave any registered
> DLLs behind, e.g.
>
> * checking whether the namespace can be unloaded cleanly ... WARNING
>   Unloading the namespace does not unload DLL
> * checking loading without being on the library search path ... OK
>
> For further details on my thoughts on this, see
> https://github.com/HenrikBengtsson/Wishlist-for-R/issues/29.
>
> Hope this helps
>
> Henrik
>
> On Tue, Sep 13, 2016 at 6:05 AM, Alexander Shenkin <ashenkin at
ufl.edu> wrote:
>> Hello all,
>>
>> I have a number of analyses that call bunches of sub-scripts, and in
the
>> end, I get the "maximal number of DLLs reached" error.  This
has been asked
>> before (e.g.
>>
http://stackoverflow.com/questions/36974206/r-maximal-number-of-dlls-reached),
>> and the general answer is, "just clean up after yourself".
>>
>> Assuming there are no plans to raise this 100-DLL limit in the near
future,
>> my question becomes, what is best practice for cleaning up (detaching)
>> loaded packages in scripts, when those scripts are sometimes called
from
>> other scripts?  One can detach all packages at the end of a script that
were
>> loaded at the beginning of the script.  However, if a package is
required in
>> a calling script, one should really make sure it hadn't been loaded
prior to
>> sub-script invocation before detaching it.
>>
>> I could write a custom function that pushes and pops package names from
a
>> global list, in order to keep track, but maybe there's a better way
out
>> there...
>>
>> Thanks for any thoughts.
>>
>> Allie
>>
>> ______________________________________________
>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.

Jeff Newmiller

2016-Sep-14 13:41 UTC

head link

[R] Maximum # of DLLs reached, or, how to clean up after yourself?

I never detach packages. I rarely load more than 6 or 7 packages directly before
restarting R. I frequently re-run my scripts in new R sessions to confirm
reproducibility.
-- 
Sent from my phone. Please excuse my brevity.

On September 14, 2016 1:49:55 AM PDT, Alexander Shenkin <ashenkin at
ufl.edu> wrote:>Hi Henrik,
>
>Thanks for your reply.  I didn't realize that floating DLLs were an 
>issue (good to know).  My query is actually a bit more basic.  I'm 
>actually wondering how folks manage their loading and unloading of 
>packages when calling scripts within scripts.
>
>Quick example:
>Script1:
>	library(package1)
>	source("script2.r")
>	# do stuff reliant on package1
>	detach("package:package1", unload=TRUE)
>
>Script2:
>	library(package1)
>	library(package2)
>	# do stuff reliant on package1 and package2
>	detach("package:package1", unload=TRUE)
>	detach("package:package2", unload=TRUE)
>
>Script2 breaks Script1 by unloading package1 (though unloading package2
>
>is ok).  I will have to test whether each package is loaded in Script2 
>before loading it, and use that list when unloading at the end of the 
>Script2.  *Unless there's a better way to do it* (which is my question
>- 
>is there?).  I'm possibly just pushing the whole procedural scripting 
>thing too far, but I also think that this likely isn't uncommon in R.
>
>Any thoughts greatly appreciated!
>
>Thanks,
>Allie
>
>On 9/13/2016 7:23 PM, Henrik Bengtsson wrote:
>> In R.utils (>= 2.4.0), which I hope to submitted to CRAN today or
>> tomorrow, you can simply call:
>>
>>    R.utils::gcDLLs()
>>
>> It will look at base::getLoadedDLLs() and its content and compare to
>> loadedNamespaces() and unregister any "stray" DLLs that
remain after
>> corresponding packages have been unloaded.
>>
>> Until the new version is on CRAN, you can install it via
>>
>>    
>source("http://callr.org/install#HenrikBengtsson/R.utils at
develop")
>>
>> or alternatively just source() the source file:
>>
>>    
>source("https://raw.githubusercontent.com/HenrikBengtsson/R.utils/develop/R/gcDLLs.R")
>>
>>
>> DISCUSSION:
>> (this might be better suited for R-devel; feel free to move it there)
>>
>> As far as I understand the problem, running into this error / limit
>is
>> _not_ the fault of the user.  Instead, I'd argue that it is the
>> responsibility of package developers to make sure to unregister any
>> registered DLLs of theirs when the package is unloaded.  A developer
>> can do this by adding the following to their package:
>>
>> .onUnload <- function(libpath) {
>>     library.dynam.unload(utils::packageName(), libpath)
>>  }
>>
>> That should be all - then the DLL will be unloaded as soon as the
>> package is unloaded.
>>
>> I would like to suggest that 'R CMD check' would include a
check that
>> asserts when a package is unloaded it does not leave any registered
>> DLLs behind, e.g.
>>
>> * checking whether the namespace can be unloaded cleanly ... WARNING
>>   Unloading the namespace does not unload DLL
>> * checking loading without being on the library search path ... OK
>>
>> For further details on my thoughts on this, see
>> https://github.com/HenrikBengtsson/Wishlist-for-R/issues/29.
>>
>> Hope this helps
>>
>> Henrik
>>
>> On Tue, Sep 13, 2016 at 6:05 AM, Alexander Shenkin <ashenkin at
ufl.edu>
>wrote:
>>> Hello all,
>>>
>>> I have a number of analyses that call bunches of sub-scripts, and
in
>the
>>> end, I get the "maximal number of DLLs reached" error. 
This has
>been asked
>>> before (e.g.
>>>
>http://stackoverflow.com/questions/36974206/r-maximal-number-of-dlls-reached),
>>> and the general answer is, "just clean up after
yourself".
>>>
>>> Assuming there are no plans to raise this 100-DLL limit in the near
>future,
>>> my question becomes, what is best practice for cleaning up
>(detaching)
>>> loaded packages in scripts, when those scripts are sometimes called
>from
>>> other scripts?  One can detach all packages at the end of a script
>that were
>>> loaded at the beginning of the script.  However, if a package is
>required in
>>> a calling script, one should really make sure it hadn't been
loaded
>prior to
>>> sub-script invocation before detaching it.
>>>
>>> I could write a custom function that pushes and pops package names
>from a
>>> global list, in order to keep track, but maybe there's a better
way
>out
>>> there...
>>>
>>> Thanks for any thoughts.
>>>
>>> Allie
>>>
>>> ______________________________________________
>>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more,
see
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>
>______________________________________________
>R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.

Henrik Bengtsson

2016-Sep-14 18:42 UTC

head link

[R] Maximum # of DLLs reached, or, how to clean up after yourself?

As Jeff says, I think the common use case is to run/rerun in fresh R sessions.

But, yes, if you'd like to have each script clean up after itself,
then you need to check with pkgs0 <- loadedNamespaces() to see what
packages are loaded when the script starts (not just attached) and
then unload the ones added at the end by pkgsDiff <-
setdiff(loadedNamespaces(), pkgs0).   However, it's not as simple as
calling unloadNamespace(pkgsDiff), because they need to be unloaded in
an order that is compatible with the package dependencies.   One way
is to too use while(length(pkgDiffs) > 0) loop over with a
try(unloadNamespace(pkg)) until all are unloaded.   At the end, run
R.utils::gcDLLs() too (now on CRAN).

unloadNamespace("foo") should result in the same as
detach("package::foo", unload=TRUE) [anyone correct me if I'm
wrong].

Hope this helps

Henrik

On Wed, Sep 14, 2016 at 6:41 AM, Jeff Newmiller
<jdnewmil at dcn.davis.ca.us> wrote:> I never detach packages. I rarely load more than 6 or 7 packages directly
before restarting R. I frequently re-run my scripts in new R sessions to confirm
reproducibility.
> --
> Sent from my phone. Please excuse my brevity.
>
> On September 14, 2016 1:49:55 AM PDT, Alexander Shenkin <ashenkin at
ufl.edu> wrote:
>>Hi Henrik,
>>
>>Thanks for your reply.  I didn't realize that floating DLLs were an
>>issue (good to know).  My query is actually a bit more basic.  I'm
>>actually wondering how folks manage their loading and unloading of
>>packages when calling scripts within scripts.
>>
>>Quick example:
>>Script1:
>>       library(package1)
>>       source("script2.r")
>>       # do stuff reliant on package1
>>       detach("package:package1", unload=TRUE)
>>
>>Script2:
>>       library(package1)
>>       library(package2)
>>       # do stuff reliant on package1 and package2
>>       detach("package:package1", unload=TRUE)
>>       detach("package:package2", unload=TRUE)
>>
>>Script2 breaks Script1 by unloading package1 (though unloading package2
>>
>>is ok).  I will have to test whether each package is loaded in Script2
>>before loading it, and use that list when unloading at the end of the
>>Script2.  *Unless there's a better way to do it* (which is my
question
>>-
>>is there?).  I'm possibly just pushing the whole procedural
scripting
>>thing too far, but I also think that this likely isn't uncommon in
R.
>>
>>Any thoughts greatly appreciated!
>>
>>Thanks,
>>Allie
>>
>>On 9/13/2016 7:23 PM, Henrik Bengtsson wrote:
>>> In R.utils (>= 2.4.0), which I hope to submitted to CRAN today
or
>>> tomorrow, you can simply call:
>>>
>>>    R.utils::gcDLLs()
>>>
>>> It will look at base::getLoadedDLLs() and its content and compare
to
>>> loadedNamespaces() and unregister any "stray" DLLs that
remain after
>>> corresponding packages have been unloaded.
>>>
>>> Until the new version is on CRAN, you can install it via
>>>
>>>
>>source("http://callr.org/install#HenrikBengtsson/R.utils at
develop")
>>>
>>> or alternatively just source() the source file:
>>>
>>>
>>source("https://raw.githubusercontent.com/HenrikBengtsson/R.utils/develop/R/gcDLLs.R")
>>>
>>>
>>> DISCUSSION:
>>> (this might be better suited for R-devel; feel free to move it
there)
>>>
>>> As far as I understand the problem, running into this error / limit
>>is
>>> _not_ the fault of the user.  Instead, I'd argue that it is the
>>> responsibility of package developers to make sure to unregister any
>>> registered DLLs of theirs when the package is unloaded.  A
developer
>>> can do this by adding the following to their package:
>>>
>>> .onUnload <- function(libpath) {
>>>     library.dynam.unload(utils::packageName(), libpath)
>>>  }
>>>
>>> That should be all - then the DLL will be unloaded as soon as the
>>> package is unloaded.
>>>
>>> I would like to suggest that 'R CMD check' would include a
check that
>>> asserts when a package is unloaded it does not leave any registered
>>> DLLs behind, e.g.
>>>
>>> * checking whether the namespace can be unloaded cleanly ...
WARNING
>>>   Unloading the namespace does not unload DLL
>>> * checking loading without being on the library search path ... OK
>>>
>>> For further details on my thoughts on this, see
>>> https://github.com/HenrikBengtsson/Wishlist-for-R/issues/29.
>>>
>>> Hope this helps
>>>
>>> Henrik
>>>
>>> On Tue, Sep 13, 2016 at 6:05 AM, Alexander Shenkin <ashenkin at
ufl.edu>
>>wrote:
>>>> Hello all,
>>>>
>>>> I have a number of analyses that call bunches of sub-scripts,
and in
>>the
>>>> end, I get the "maximal number of DLLs reached"
error.  This has
>>been asked
>>>> before (e.g.
>>>>
>>http://stackoverflow.com/questions/36974206/r-maximal-number-of-dlls-reached),
>>>> and the general answer is, "just clean up after
yourself".
>>>>
>>>> Assuming there are no plans to raise this 100-DLL limit in the
near
>>future,
>>>> my question becomes, what is best practice for cleaning up
>>(detaching)
>>>> loaded packages in scripts, when those scripts are sometimes
called
>>from
>>>> other scripts?  One can detach all packages at the end of a
script
>>that were
>>>> loaded at the beginning of the script.  However, if a package
is
>>required in
>>>> a calling script, one should really make sure it hadn't
been loaded
>>prior to
>>>> sub-script invocation before detaching it.
>>>>
>>>> I could write a custom function that pushes and pops package
names
>>from a
>>>> global list, in order to keep track, but maybe there's a
better way
>>out
>>>> there...
>>>>
>>>> Thanks for any thoughts.
>>>>
>>>> Allie
>>>>
>>>> ______________________________________________
>>>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and
more, see
>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>> PLEASE do read the posting guide
>>http://www.R-project.org/posting-guide.html
>>>> and provide commented, minimal, self-contained, reproducible
code.
>>
>>______________________________________________
>>R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>>https://stat.ethz.ch/mailman/listinfo/r-help
>>PLEASE do read the posting guide
>>http://www.R-project.org/posting-guide.html
>>and provide commented, minimal, self-contained, reproducible code.
>

R help - Sep 2016 - Maximum # of DLLs reached, or, how to clean up after yourself?

[R] Maximum # of DLLs reached, or, how to clean up after yourself?

[R] Maximum # of DLLs reached, or, how to clean up after yourself?

[R] Maximum # of DLLs reached, or, how to clean up after yourself?