similar to: parallel PSOCK connection latency is greater on Linux?

On Mon, 2 Nov 2020 at 02:22, Simon Urbanek <simon.urbanek at r-project.org> wrote: > > It looks like R sockets on Linux could do with TCP_NODELAY -- without (status quo): How many network packets are generated with and without it? If there are many small writes and thus setting TCP_NODELAY causes many small packets to be sent, it might make more sense to set TCP_QUICKACK instead.

I'm not sure the user would know ;). This is very system-specific issue just because the Linux network stack behaves so differently from other OSes (for purely historical reasons). That makes it hard to abstract as a "feature" for the R sockets that are supposed to be platform-independent. At least TCP_NODELAY is actually part of POSIX so it is on better footing, and disabling

It looks like R sockets on Linux could do with TCP_NODELAY -- without (status quo): Unit: microseconds expr min lq mean median uq max clusterEvalQ(cl, iris) 1449.997 43991.99 43975.21 43997.1 44001.91 48027.83 neval 1000 exactly the same machine + R but with TCP_NODELAY enabled in R_SockConnect(): Unit: microseconds expr

Could TCP_NODELAY and TCP_QUICKACK be exposed to the R user so that they might determine what is best for their potentially latency- or throughput-sensitive application? Best, Jeff On Mon, Nov 2, 2020 at 14:05, I?aki Ucar <iucar at fedoraproject.org> wrote: > On Mon, 2 Nov 2020 at 02:22, Simon Urbanek > <simon.urbanek at r-project.org> wrote: >> >> It looks

Please, check a tcpdump session on localhost while running the following script: library(parallel) library(tictoc) cl <- makeCluster(1) Sys.sleep(1) for (i in 1:10) { tic() x <- clusterEvalQ(cl, iris) toc() } The initialization phase comprises 7 packets. Then, the 1-second sleep will help you see where the evaluation starts. Each clusterEvalQ generates 6 packets: 1. main ->

Hi, I'd like to test whether a (localhost) PSOCK cluster node is still running or not by its PID, e.g. it may have crashed / core dumped. I'm ok with getting false-positive results due to *another* process with the same PID has since started. I can the PID of each cluster nodes by querying them for their Sys.getpid(), e.g. pids <- parallel::clusterEvalQ(cl, Sys.getpid()) Is there

Hi all, I am not able to export an ALTREP object when `gctorture` is on in the worker. The package simplemmap can be used to reproduce the problem. See the example below ``` ## Create a temporary file filePath <- tempfile() con <- file(filePath, "wrb") writeBin(rep(0.0,10),con) close(con) library(simplemmap) library(parallel) cl <- makeCluster(1) x <- mmap(filePath,

This Index: src/main/altrep.c =================================================================== --- src/main/altrep.c (revision 79385) +++ src/main/altrep.c (working copy) @@ -275,10 +275,11 @@ SEXP psym = ALTREP_SERIALIZED_CLASS_PKGSYM(info); SEXP class = LookupClass(csym, psym); if (class == NULL) { - SEXP pname = ScalarString(PRINTNAME(psym)); + SEXP pname =

James A. Peltier <jpeltier at sfu.ca> wrote: > While I can't assure you that upgrading to 7 will fix the problem, > the likeliness that will work is at least 50% better being as 7 is > based on a newer kernel. I've installed CentOS 7 with the latest kernel from elrepo (4.11.8), and I can't get Grub to install. I think I'll have to conclude that it really

Dear R-users, we are using library(snow) for computation on a linux cluster with RMPI. We have a problem with clusterEvalQ: after launching clusterEvalQ it seems loading the required library on each node but if we type a function belonging to the loaded package R doesn't find it. > library(snow) # making cluster with 3 nodes > cl <- makeCluster(3, type = "MPI") Loading

Hi all, Is it possible to use the 'renice' option together with parallel clusters of type 'PSOCK'? The help page for parallel::makeCluster is not specific about which options are supported on which types and I am getting the following message when passing renice = 19 : > cl <- parallel::makeCluster(2, renice = 19) nice: ?+19?: No such file or directory Kind regards,

I have a new system with an AMD Ryzen 1600 CPU and I'm trying to run CentOS, so far without success. The only information I could find was a post reporting that CentOS 7 would crash during installation on a Ryzen system: https://www.centos.org/forums/viewtopic.php?t=61831 . I tried updating to the latest elrepo "mainline stable" kernel, version 4.11 (by booting from a live CD and

Looks like a bug to me due to wrong assumptions about 'nice' arguments, but could be because a "non-standard" 'nice' is used. If we do: > trace(system, tracer = quote(print(command))) Tracing function "system" in package "base" we see that the system call used is: > cl <- parallel::makePSOCKcluster(2L, renice = 19) Tracing system(cmd, wait

On Fri, Aug 31, 2018 at 2:51 PM Tomas Kalibera <tomas.kalibera at gmail.com> wrote: [...] > kill(sig=0) is specified by POSIX but indeed as you say there is a race > condition due to PID-reuse. In principle, detecting that a worker > process is still alive cannot be done correctly outside base R. I am not sure why you think so. > At user-level I would probably consider some

Hi Henrik, Thanks for the detailed in fast reply! My guess would be that the confusion comes from the different use of nice and renice. The workraund you provided work fine! Thanks a lot. Best, Andreas Henrik Bengtsson <henrik.bengtsson at gmail.com> writes: > Looks like a bug to me due to wrong assumptions about 'nice' > arguments, but could be because a

The following program is whittled down from a much larger program that always works on Intel, and always works on AMD's threadripper with lapply but not mclappy. With mclapply on AMD, all processes go into "suspend" mode and the program then hangs. This bug is replicable on an AMD Ryzen Threadripper 2950X 16-Core Processor (128GB RAM), running latest ubuntu 18.04. The R version

OK, AMD has announced it's new line of server and desktop processors. What level of CentOS has been tested on them? OK then, when will CentOS be tested on them? Or do we wait for Red Hat? -- Eugene Poole Woodstock, Georgia

hi, our current servers are suffering from a weird hardware issue that forces us to start over. in short we have two servers with 15 disks at 6TB each, divided into three raid5 arrays for three bricks per server at 22TB per brick. each brick on one server is replicated to a brick on the second server. the hardware issue is that somewhere in the backplane random I/O errors happen when the system

Will 5.0 have scheduler, reg alloc etc. bits for explicit support of AMD's 1st gen Zen (Ryzen, Epyc) arch? Is it safe to assume that the changes for Zen's 2nd gen will land prior to the hardware release, once the 1st gen is added? I think zen2 will arrive next year.

Hi, amdgpu + IOMMU + SME is now working for me on 5.7, yay! But, it is broken on torvalds master, boo. On boot, depending on which exact commit I test, it either hangs immediately (with built-in driver, before starting initramfs), displays some errors then hangs, or spams the screen with many amdgpu errors. I bisected the black screen hang to: commit dce8d6964ebdb333383bacf5e7ab8c27df151218