R devel - May 2016 - Is it possible to increase MAX_NUM_DLLS in future R releases?

Hi,

I?m working on a Shiny app for statistical analysis. I ran into this
"maximal number of DLLs reached" issue recently because my app
requires importing many other packages.

I?ve posted my question on stackoverflow
(http://stackoverflow.com/questions/36974206/r-maximal-number-of-dlls-reached
<http://stackoverflow.com/questions/36974206/r-maximal-number-of-dlls-reached>).

I?m just wondering is there any reason to set the maximal number of DLLs to be
100, and is there any plan to increase it/not hardcoding it in the future? It
seems many people are also running into this problem. I know I can work around
this problem by modifying the source, but since my package is going to be used
by other people, I don?t think this is a feasible solution.

Any suggestions would be appreciated. Thanks!

Qin
	[[alternative HTML version deleted]]

>>>>> Qin Zhu <qinzhu at outlook.com>
>>>>>     on Mon, 2 May 2016 16:19:44 -0400 writes:
    > Hi,
    > I?m working on a Shiny app for statistical analysis. I ran into this
"maximal number of DLLs reached" issue recently because my app
requires importing many other packages.

    > I?ve posted my question on stackoverflow
(http://stackoverflow.com/questions/36974206/r-maximal-number-of-dlls-reached
<http://stackoverflow.com/questions/36974206/r-maximal-number-of-dlls-reached>).

    > I?m just wondering is there any reason to set the maximal number of
DLLs to be 100, and is there any plan to increase it/not hardcoding it in the
future? It seems many people are also running into this problem. I know I can
work around this problem by modifying the source, but since my package is going
to be used by other people, I don?t think this is a feasible solution.

    > Any suggestions would be appreciated. Thanks!
    > Qin

Increasing that number is of course "possible"... but it also
costs a bit (adding to the fixed memory footprint of R). 

I did not set that limit, but I'm pretty sure it was also meant
as reminder for the useR to "clean up" a bit in her / his R
session, i.e., not load package namespaces unnecessarily. I
cannot yet imagine that you need > 100 packages | namespaces
loaded in your R session. OTOH, some packages nowadays have a
host of dependencies, so I agree that this at least may happen
accidentally more frequently than in the past. 

The real solution of course would be a code improvement that
starts with a relatively small number of "DLLinfo" structures
(say 32), and then allocates more batches (of size say 32) if
needed. 

Patches to the R sources (development trunk in subversion at
https://svn.r-project.org/R/trunk/ ) are very welcome! 

Martin Maechler
ETH Zurich  &  R Core Team

On 04/05/2016 08:44, Martin Maechler wrote:
>>>>>> Qin Zhu <qinzhu at outlook.com>
>>>>>>      on Mon, 2 May 2016 16:19:44 -0400 writes:
>
>      > Hi,
>      > I?m working on a Shiny app for statistical analysis. I ran into
this "maximal number of DLLs reached" issue recently because my app
requires importing many other packages.
>
>      > I?ve posted my question on stackoverflow
(http://stackoverflow.com/questions/36974206/r-maximal-number-of-dlls-reached
<http://stackoverflow.com/questions/36974206/r-maximal-number-of-dlls-reached>).
>
>      > I?m just wondering is there any reason to set the maximal number
of DLLs to be 100, and is there any plan to increase it/not hardcoding it in the
future? It seems many people are also running into this problem. I know I can
work around this problem by modifying the source, but since my package is going
to be used by other people, I don?t think this is a feasible solution.
>
>      > Any suggestions would be appreciated. Thanks!
>      > Qin
>
> Increasing that number is of course "possible"... but it also
> costs a bit (adding to the fixed memory footprint of R).
And not only that.  At the time this was done (and it was once 50) the 
main cost was searching DLLs for symbols.  That is still an issue, and 
few packages exclude their DLL from symbol search so if symbols have to 
searched for a lot of DLLs will be searched.  (Registering all the 
symbols needed in a package avoids a search, and nowadays by default 
searches from a namespace are restricted to that namespace.)

See 
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Registering-native-routines
for some further details about the search mechanism.
> I did not set that limit, but I'm pretty sure it was also meant
> as reminder for the useR to "clean up" a bit in her / his R
> session, i.e., not load package namespaces unnecessarily. I
> cannot yet imagine that you need > 100 packages | namespaces
> loaded in your R session. OTOH, some packages nowadays have a
> host of dependencies, so I agree that this at least may happen
> accidentally more frequently than in the past.
I am not convinced that it is needed.  The OP says he imports many 
packages, and I doubt that more than a few are required at any one time. 
  Good practice is to load namespaces as required, using requireNamespace.
> The real solution of course would be a code improvement that
> starts with a relatively small number of "DLLinfo" structures
> (say 32), and then allocates more batches (of size say 32) if
> needed.
The problem of course is that such code will rarely be exercised, and 
people have made errors on the boundaries (here multiples of 32) many 
times in the past.  (Note too that DLLs can be removed as well as added, 
another point of coding errors.)
> Patches to the R sources (development trunk in subversion at
> https://svn.r-project.org/R/trunk/ ) are very welcome!
>
> Martin Maechler
> ETH Zurich  &  R Core Team


-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Emeritus Professor of Applied Statistics, University of Oxford