R devel - Aug 2021 - Is it a good choice to increase the NCONNECTION value?

At least in 2015, a github user, tobigithub, submit an
[issue](https://github.com/sneumann/xcms/issues/20) about the error "Error
in file(con, "w") : all connections are in use"
Nowadays, since AMD have really cool CPUs which increases the thread numbers to
128 or even 256 on a single server, we found that the NCONNECTIONS variable
could prevent us from utilizing all the 128 threads.
It might be a good choice to increase its value.


the variable is defined in `R-4.1.1/src/main/connections.c: 17`
I have tested that, increase it to 1024 generates no error and all the clusters
(I tried with 256 clusters on my 16 threads Laptop) works fine.


Is it possible increase the size of NCONNECTION?


	[[alternative HTML version deleted]]

>>>>> qweytr1--- via R-devel 
>>>>>     on Tue, 24 Aug 2021 00:51:31 +0800 (GMT+08:00) writes:
    > At least in 2015, a github user, tobigithub, submit an
    > [issue](https://github.com/sneumann/xcms/issues/20) about
    > the error "Error in file(con, "w") : all connections are
    > in use" Nowadays, since AMD have really cool CPUs which
    > increases the thread numbers to 128 or even 256 on a
    > single server, we found that the NCONNECTIONS variable
    > could prevent us from utilizing all the 128 threads.  It
    > might be a good choice to increase its value.


    > the variable is defined in
    > `R-4.1.1/src/main/connections.c: 17` I have tested that,
    > increase it to 1024 generates no error and all the
    > clusters (I tried with 256 clusters on my 16 threads
    > Laptop) works fine.

    > Is it possible increase the size of NCONNECTION?

Yes, of course, it is possible.
The question is how much it costs  and to which number it should
be increased.

A quick look at the source connections.c --> src/R_ext/include/Connections.h
reveals that the Rconnection* <--> Rconn is a struct with about
200 chars and ca 30 int-like plus another 20 pointers .. which
would amount to a rough 400 bytes per connection.
Adding 1024-128 = 896 new ones  would then amount to increase
the R executable by about 360 kB .. all the above being rough.
So personally, I guess that's  "about ok" --
are there other things to consider?

Ideally, of course, the number of possible connections could be
increased dynamically only when needed

Martin