Martin Maechler
2021-Aug-24 08:44 UTC
[Rd] Is it a good choice to increase the NCONNECTION value?
>>>>> qweytr1--- via R-devel >>>>> on Tue, 24 Aug 2021 00:51:31 +0800 (GMT+08:00) writes:> At least in 2015, a github user, tobigithub, submit an > [issue](https://github.com/sneumann/xcms/issues/20) about > the error "Error in file(con, "w") : all connections are > in use" Nowadays, since AMD have really cool CPUs which > increases the thread numbers to 128 or even 256 on a > single server, we found that the NCONNECTIONS variable > could prevent us from utilizing all the 128 threads. It > might be a good choice to increase its value. > the variable is defined in > `R-4.1.1/src/main/connections.c: 17` I have tested that, > increase it to 1024 generates no error and all the > clusters (I tried with 256 clusters on my 16 threads > Laptop) works fine. > Is it possible increase the size of NCONNECTION? Yes, of course, it is possible. The question is how much it costs and to which number it should be increased. A quick look at the source connections.c --> src/R_ext/include/Connections.h reveals that the Rconnection* <--> Rconn is a struct with about 200 chars and ca 30 int-like plus another 20 pointers .. which would amount to a rough 400 bytes per connection. Adding 1024-128 = 896 new ones would then amount to increase the R executable by about 360 kB .. all the above being rough. So personally, I guess that's "about ok" -- are there other things to consider? Ideally, of course, the number of possible connections could be increased dynamically only when needed Martin
GILLIBERT, Andre
2021-Aug-24 09:49 UTC
[Rd] Is it a good choice to increase the NCONNECTION value?
RConnection is a pointer to a Rconn structure. The Rconn structure must be allocated independently (e.g. by malloc() in R_new_custom_connection). Therefore, increasing NCONNECTION to 1024 should only use 8 kilobytes on 64-bits platforms and 4 kilobytes on 32 bits platforms. Ideally, it should be dynamically allocated : either as a linked list or as a dynamic array (malloc/realloc). However, a simple change of NCONNECTION to 1024 should be enough for most uses. -- Sincerely Andr? GILLIBERT ________________________________ De : R-devel <r-devel-bounces at r-project.org> de la part de Martin Maechler <maechler at stat.math.ethz.ch> Envoy? : mardi 24 ao?t 2021 10:44 ? : qweytr1 at mail.ustc.edu.cn Cc : R-devel Objet : Re: [Rd] Is it a good choice to increase the NCONNECTION value?>>>>> qweytr1--- via R-devel >>>>> on Tue, 24 Aug 2021 00:51:31 +0800 (GMT+08:00) writes:> At least in 2015, a github user, tobigithub, submit an > [issue](https://github.com/sneumann/xcms/issues/20) about > the error "Error in file(con, "w") : all connections are > in use" Nowadays, since AMD have really cool CPUs which > increases the thread numbers to 128 or even 256 on a > single server, we found that the NCONNECTIONS variable > could prevent us from utilizing all the 128 threads. It > might be a good choice to increase its value. > the variable is defined in > `R-4.1.1/src/main/connections.c: 17` I have tested that, > increase it to 1024 generates no error and all the > clusters (I tried with 256 clusters on my 16 threads > Laptop) works fine. > Is it possible increase the size of NCONNECTION? Yes, of course, it is possible. The question is how much it costs and to which number it should be increased. A quick look at the source connections.c --> src/R_ext/include/Connections.h reveals that the Rconnection* <--> Rconn is a struct with about 200 chars and ca 30 int-like plus another 20 pointers .. which would amount to a rough 400 bytes per connection. Adding 1024-128 = 896 new ones would then amount to increase the R executable by about 360 kB .. all the above being rough. So personally, I guess that's "about ok" -- are there other things to consider? Ideally, of course, the number of possible connections could be increased dynamically only when needed Martin ______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel [[alternative HTML version deleted]]