R devel - Aug 2021 - Is it a good choice to increase the NCONNECTION value?

RConnection is a pointer to a Rconn structure. The Rconn structure must be
allocated independently (e.g. by malloc() in R_new_custom_connection).

Therefore, increasing NCONNECTION to 1024 should only use 8 kilobytes on 64-bits
platforms and 4 kilobytes on 32 bits platforms.

Ideally, it should be dynamically allocated : either as a linked list or as a
dynamic array (malloc/realloc). However, a simple change of NCONNECTION to 1024
should be enough for most uses.


--

Sincerely

Andr? GILLIBERT


________________________________
De : R-devel <r-devel-bounces at r-project.org> de la part de Martin
Maechler <maechler at stat.math.ethz.ch>
Envoy? : mardi 24 ao?t 2021 10:44
? : qweytr1 at mail.ustc.edu.cn
Cc : R-devel
Objet : Re: [Rd] Is it a good choice to increase the NCONNECTION value?

>>>>> qweytr1--- via R-devel
>>>>>     on Tue, 24 Aug 2021 00:51:31 +0800 (GMT+08:00) writes:
    > At least in 2015, a github user, tobigithub, submit an
    > [issue](https://github.com/sneumann/xcms/issues/20) about
    > the error "Error in file(con, "w") : all connections are
    > in use" Nowadays, since AMD have really cool CPUs which
    > increases the thread numbers to 128 or even 256 on a
    > single server, we found that the NCONNECTIONS variable
    > could prevent us from utilizing all the 128 threads.  It
    > might be a good choice to increase its value.


    > the variable is defined in
    > `R-4.1.1/src/main/connections.c: 17` I have tested that,
    > increase it to 1024 generates no error and all the
    > clusters (I tried with 256 clusters on my 16 threads
    > Laptop) works fine.

    > Is it possible increase the size of NCONNECTION?

Yes, of course, it is possible.
The question is how much it costs  and to which number it should
be increased.

A quick look at the source connections.c --> src/R_ext/include/Connections.h
reveals that the Rconnection* <--> Rconn is a struct with about
200 chars and ca 30 int-like plus another 20 pointers .. which
would amount to a rough 400 bytes per connection.
Adding 1024-128 = 896 new ones  would then amount to increase
the R executable by about 360 kB .. all the above being rough.
So personally, I guess that's  "about ok" --
are there other things to consider?

Ideally, of course, the number of possible connections could be
increased dynamically only when needed

Martin

______________________________________________
R-devel at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel


	[[alternative HTML version deleted]]

>>>>> GILLIBERT, Andre 
>>>>>     on Tue, 24 Aug 2021 09:49:52 +0000 writes:
  > RConnection is a pointer to a Rconn structure. The Rconn
  > structure must be allocated independently (e.g. by
  > malloc() in R_new_custom_connection).  Therefore,
  > increasing NCONNECTION to 1024 should only use 8
  > kilobytes on 64-bits platforms and 4 kilobytes on 32
  > bits platforms.

You are right indeed, and I was wrong.

  > Ideally, it should be dynamically allocated : either as
  > a linked list or as a dynamic array
  > (malloc/realloc). However, a simple change of
  > NCONNECTION to 1024 should be enough for most uses.

There is one important other problem I've been made aware
(similarly to the number of open DLL libraries, an issue 1-2
years ago) :

The OS itself has limits on the number of open files
(yes, I know that there are other connections than files) and
these limits may quite differ from platform to platform.

On my Linux laptop, in a shell, I see

  $ ulimit -n
  1024

which is barely conformant with your proposed 1024 NCONNECTION.

Now if NCONNCECTION is larger than the max allowed number of
open files and if R opens more files than the OS allowed, the
user may get quite unpleasant behavior, e.g. R being terminated brutally
(or behaving crazily) without good R-level warning / error messages.

It's also not at all sufficient to check for the open files
limit at compile time, but rather at R process startup time 

So this may need considerably more work than you / we have
hoped, and it's probably hard to find a safe number that is
considerably larger than 128  and less than the smallest of all
non-crazy platforms' {number of open files limit}.

  > Sincerely
  > Andr? GILLIBERT

  [............]