R help - Sep 2016 - Maximum # of DLLs reached, or, how to clean up after yourself?

Hello all,

I have a number of analyses that call bunches of sub-scripts, and in the 
end, I get the "maximal number of DLLs reached" error.  This has been 
asked before (e.g. 
http://stackoverflow.com/questions/36974206/r-maximal-number-of-dlls-reached), 
and the general answer is, "just clean up after yourself".

Assuming there are no plans to raise this 100-DLL limit in the near 
future, my question becomes, what is best practice for cleaning up 
(detaching) loaded packages in scripts, when those scripts are sometimes 
called from other scripts?  One can detach all packages at the end of a 
script that were loaded at the beginning of the script.  However, if a 
package is required in a calling script, one should really make sure it 
hadn't been loaded prior to sub-script invocation before detaching it.

I could write a custom function that pushes and pops package names from 
a global list, in order to keep track, but maybe there's a better way 
out there...

Thanks for any thoughts.

Allie

In R.utils (>= 2.4.0), which I hope to submitted to CRAN today or
tomorrow, you can simply call:

   R.utils::gcDLLs()

It will look at base::getLoadedDLLs() and its content and compare to
loadedNamespaces() and unregister any "stray" DLLs that remain after
corresponding packages have been unloaded.

Until the new version is on CRAN, you can install it via

    source("http://callr.org/install#HenrikBengtsson/R.utils at
develop")

or alternatively just source() the source file:

   
source("https://raw.githubusercontent.com/HenrikBengtsson/R.utils/develop/R/gcDLLs.R")


DISCUSSION:
(this might be better suited for R-devel; feel free to move it there)

As far as I understand the problem, running into this error / limit is
_not_ the fault of the user.  Instead, I'd argue that it is the
responsibility of package developers to make sure to unregister any
registered DLLs of theirs when the package is unloaded.  A developer
can do this by adding the following to their package:

.onUnload <- function(libpath) {
    library.dynam.unload(utils::packageName(), libpath)
 }

That should be all - then the DLL will be unloaded as soon as the
package is unloaded.

I would like to suggest that 'R CMD check' would include a check that
asserts when a package is unloaded it does not leave any registered
DLLs behind, e.g.

* checking whether the namespace can be unloaded cleanly ... WARNING
  Unloading the namespace does not unload DLL
* checking loading without being on the library search path ... OK

For further details on my thoughts on this, see
https://github.com/HenrikBengtsson/Wishlist-for-R/issues/29.

Hope this helps

Henrik

On Tue, Sep 13, 2016 at 6:05 AM, Alexander Shenkin <ashenkin at ufl.edu>
wrote:
> Hello all,
>
> I have a number of analyses that call bunches of sub-scripts, and in the
> end, I get the "maximal number of DLLs reached" error.  This has
been asked
> before (e.g.
>
http://stackoverflow.com/questions/36974206/r-maximal-number-of-dlls-reached),
> and the general answer is, "just clean up after yourself".
>
> Assuming there are no plans to raise this 100-DLL limit in the near future,
> my question becomes, what is best practice for cleaning up (detaching)
> loaded packages in scripts, when those scripts are sometimes called from
> other scripts?  One can detach all packages at the end of a script that
were
> loaded at the beginning of the script.  However, if a package is required
in
> a calling script, one should really make sure it hadn't been loaded
prior to
> sub-script invocation before detaching it.
>
> I could write a custom function that pushes and pops package names from a
> global list, in order to keep track, but maybe there's a better way out
> there...
>
> Thanks for any thoughts.
>
> Allie
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

Hi Henrik,

Thanks for your reply.  I didn't realize that floating DLLs were an 
issue (good to know).  My query is actually a bit more basic.  I'm 
actually wondering how folks manage their loading and unloading of 
packages when calling scripts within scripts.

Quick example:
Script1:
	library(package1)
	source("script2.r")
	# do stuff reliant on package1
	detach("package:package1", unload=TRUE)

Script2:
	library(package1)
	library(package2)
	# do stuff reliant on package1 and package2
	detach("package:package1", unload=TRUE)
	detach("package:package2", unload=TRUE)

Script2 breaks Script1 by unloading package1 (though unloading package2 
is ok).  I will have to test whether each package is loaded in Script2 
before loading it, and use that list when unloading at the end of the 
Script2.  *Unless there's a better way to do it* (which is my question - 
is there?).  I'm possibly just pushing the whole procedural scripting 
thing too far, but I also think that this likely isn't uncommon in R.

Any thoughts greatly appreciated!

Thanks,
Allie

On 9/13/2016 7:23 PM, Henrik Bengtsson wrote:
> In R.utils (>= 2.4.0), which I hope to submitted to CRAN today or
> tomorrow, you can simply call:
>
>    R.utils::gcDLLs()
>
> It will look at base::getLoadedDLLs() and its content and compare to
> loadedNamespaces() and unregister any "stray" DLLs that remain
after
> corresponding packages have been unloaded.
>
> Until the new version is on CRAN, you can install it via
>
>     source("http://callr.org/install#HenrikBengtsson/R.utils at
develop")
>
> or alternatively just source() the source file:
>
>    
source("https://raw.githubusercontent.com/HenrikBengtsson/R.utils/develop/R/gcDLLs.R")
>
>
> DISCUSSION:
> (this might be better suited for R-devel; feel free to move it there)
>
> As far as I understand the problem, running into this error / limit is
> _not_ the fault of the user.  Instead, I'd argue that it is the
> responsibility of package developers to make sure to unregister any
> registered DLLs of theirs when the package is unloaded.  A developer
> can do this by adding the following to their package:
>
> .onUnload <- function(libpath) {
>     library.dynam.unload(utils::packageName(), libpath)
>  }
>
> That should be all - then the DLL will be unloaded as soon as the
> package is unloaded.
>
> I would like to suggest that 'R CMD check' would include a check
that
> asserts when a package is unloaded it does not leave any registered
> DLLs behind, e.g.
>
> * checking whether the namespace can be unloaded cleanly ... WARNING
>   Unloading the namespace does not unload DLL
> * checking loading without being on the library search path ... OK
>
> For further details on my thoughts on this, see
> https://github.com/HenrikBengtsson/Wishlist-for-R/issues/29.
>
> Hope this helps
>
> Henrik
>
> On Tue, Sep 13, 2016 at 6:05 AM, Alexander Shenkin <ashenkin at
ufl.edu> wrote:
>> Hello all,
>>
>> I have a number of analyses that call bunches of sub-scripts, and in
the
>> end, I get the "maximal number of DLLs reached" error.  This
has been asked
>> before (e.g.
>>
http://stackoverflow.com/questions/36974206/r-maximal-number-of-dlls-reached),
>> and the general answer is, "just clean up after yourself".
>>
>> Assuming there are no plans to raise this 100-DLL limit in the near
future,
>> my question becomes, what is best practice for cleaning up (detaching)
>> loaded packages in scripts, when those scripts are sometimes called
from
>> other scripts?  One can detach all packages at the end of a script that
were
>> loaded at the beginning of the script.  However, if a package is
required in
>> a calling script, one should really make sure it hadn't been loaded
prior to
>> sub-script invocation before detaching it.
>>
>> I could write a custom function that pushes and pops package names from
a
>> global list, in order to keep track, but maybe there's a better way
out
>> there...
>>
>> Thanks for any thoughts.
>>
>> Allie
>>
>> ______________________________________________
>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.