This is a request to increase MAX_NUM_DLLS in Rdynload.c in from 100 to 500. On line 131 of Rdynload.c, changing #define MAX_NUM_DLLS 100 to #define MAX_NUM_DLLS 500 In development of the mlr package, there have been several episodes in the past where we have had to break up unit tests because of the "maximum number of DLLs reached" error. This error has been an inconvenience that is going to keep happening as the package continues to grow. Is there more than meets the eye with this error or would everything be okay if the above line changes? Would that have a larger effect in other parts of R? As R grows, we are likely to see more 'meta-packages' such as the Hadley-verse, caret, mlr, etc. need an increasing amount of DLLs loaded at any point in time to conduct effective unit tests. If MAX_NUM_DLLS is set to 100 for a very particular reason than I apologize, but if it is possible to increase MAX_NUM_DLLS it would at least make the testing at mlr much easier. I understand you are all very busy and thank you for your time. Regards, Steve Bronder Website: stevebronder.com Phone: 412-719-1282 Email: sbronder at stevebronder.com [[alternative HTML version deleted]]
Henrik Bengtsson
2016-Dec-20 06:04 UTC
[Rd] Request: Increasing MAX_NUM_DLLS in Rdynload.c
On reason for hitting the MAX_NUM_DLLS (= 100) limit is because some packages don't unload their DLLs when they being unloaded themselves. In other words, there may be left-over DLLs just sitting there doing nothing but occupying space. You can remove these, using: R.utils::gcDLLs() Maybe that will help you get through your tests (as long as you're unloading packages). gcDLLs() will look at base::getLoadedDLLs() and its content and compare to loadedNamespaces() and unregister any "stray" DLLs that remain after corresponding packages have been unloaded. I think it would be useful if R CMD check would also check that DLLs are unregistered when a package is unloaded (https://github.com/HenrikBengtsson/Wishlist-for-R/issues/29), but of course, someone needs to write the code / a patch for this to happen. /Henrik On Mon, Dec 19, 2016 at 6:01 PM, Steve Bronder <sbronder at stevebronder.com> wrote:> This is a request to increase MAX_NUM_DLLS in Rdynload.c in from 100 to 500. > > On line 131 of Rdynload.c, changing > > #define MAX_NUM_DLLS 100 > > to > > #define MAX_NUM_DLLS 500 > > > In development of the mlr package, there have been several episodes in the > past where we have had to break up unit tests because of the "maximum > number of DLLs reached" error. This error has been an inconvenience that is > going to keep happening as the package continues to grow. Is there more > than meets the eye with this error or would everything be okay if the above > line changes? Would that have a larger effect in other parts of R? > > As R grows, we are likely to see more 'meta-packages' such as the > Hadley-verse, caret, mlr, etc. need an increasing amount of DLLs loaded at > any point in time to conduct effective unit tests. If MAX_NUM_DLLS is set > to 100 for a very particular reason than I apologize, but if it is possible > to increase MAX_NUM_DLLS it would at least make the testing at mlr much > easier. > > I understand you are all very busy and thank you for your time. > > > Regards, > > Steve Bronder > Website: stevebronder.com > Phone: 412-719-1282 > Email: sbronder at stevebronder.com > > [[alternative HTML version deleted]] > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel
Thanks Henrik this is very helpful! I will try this out on our tests and see if gcDLLs() has a positive effect. mlr currently has tests broken down by learner type such as classification, regression, forecasting, clustering, etc.. There are 83 classifiers alone so even when loading and unloading across learner types we can still hit the MAX_NUM_DLLS error, meaning we'll have to break them down further (or maybe we can be clever with gcDLLs()?). I'm CC'ing Lars Kotthoff and Bernd Bischl to make sure I am representing the issue well. Regards, Steve Bronder Website: stevebronder.com Phone: 412-719-1282 Email: sbronder at stevebronder.com On Tue, Dec 20, 2016 at 1:04 AM, Henrik Bengtsson < henrik.bengtsson at gmail.com> wrote:> On reason for hitting the MAX_NUM_DLLS (= 100) limit is because some > packages don't unload their DLLs when they being unloaded themselves. > In other words, there may be left-over DLLs just sitting there doing > nothing but occupying space. You can remove these, using: > > R.utils::gcDLLs() > > Maybe that will help you get through your tests (as long as you're > unloading packages). gcDLLs() will look at base::getLoadedDLLs() and > its content and compare to loadedNamespaces() and unregister any > "stray" DLLs that remain after corresponding packages have been > unloaded. > > I think it would be useful if R CMD check would also check that DLLs > are unregistered when a package is unloaded > (https://github.com/HenrikBengtsson/Wishlist-for-R/issues/29), but of > course, someone needs to write the code / a patch for this to happen. > > /Henrik > > On Mon, Dec 19, 2016 at 6:01 PM, Steve Bronder > <sbronder at stevebronder.com> wrote: > > This is a request to increase MAX_NUM_DLLS in Rdynload.c in from 100 to > 500. > > > > On line 131 of Rdynload.c, changing > > > > #define MAX_NUM_DLLS 100 > > > > to > > > > #define MAX_NUM_DLLS 500 > > > > > > In development of the mlr package, there have been several episodes in > the > > past where we have had to break up unit tests because of the "maximum > > number of DLLs reached" error. This error has been an inconvenience that > is > > going to keep happening as the package continues to grow. Is there more > > than meets the eye with this error or would everything be okay if the > above > > line changes? Would that have a larger effect in other parts of R? > > > > As R grows, we are likely to see more 'meta-packages' such as the > > Hadley-verse, caret, mlr, etc. need an increasing amount of DLLs loaded > at > > any point in time to conduct effective unit tests. If MAX_NUM_DLLS is > set > > to 100 for a very particular reason than I apologize, but if it is > possible > > to increase MAX_NUM_DLLS it would at least make the testing at mlr much > > easier. > > > > I understand you are all very busy and thank you for your time. > > > > > > Regards, > > > > Steve Bronder > > Website: stevebronder.com > > Phone: 412-719-1282 > > Email: sbronder at stevebronder.com > > > > [[alternative HTML version deleted]] > > > > ______________________________________________ > > R-devel at r-project.org mailing list > > https://stat.ethz.ch/mailman/listinfo/r-devel >[[alternative HTML version deleted]]
On Tue, Dec 20, 2016 at 7:04 AM, Henrik Bengtsson <henrik.bengtsson at gmail.com> wrote:> On reason for hitting the MAX_NUM_DLLS (= 100) limit is because some > packages don't unload their DLLs when they being unloaded themselves.I am surprised by this. Why does R not do this automatically? What is the case for keeping the DLL loaded after the package has been unloaded? What happens if you reload another version of the same package from a different library after unloading?