R help - Jun 2023 - inconsistency in mclapply.....

Dear Ivan,
                 I am reopening this thread because there are some new
developments. I have switched to ubuntu and have used ropenblas() in ropenblas
package to download the latest openblas (it is a version of skylake as you can
see below in sessionInfo()). mclapply() is working fine now... My questions:


  1.  I was working on a two core machine just to test the code...tomorrow will
be switching to a 48 core machine. Should I again download the latest openblas
from ropenblas() or do you suggest testing the native openblas that is bundled
in Ubuntu( it was a basic version of openblas..in my two core machine I tried
the ropenblas() function without testing the native openblas...just for
curiosity) ?
  2.  How do you pass the version of openblas to be installed with ropenblas()?
The full file name? or just something like this:
libopenblas_skylakexp-r0.3.23.so  ?
> sessionInfo()
R version 4.3.0 (2023-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.2 LTS

Matrix products: default
BLAS/LAPACK: /opt/OpenBLAS/lib/libopenblas_skylakexp-r0.3.23.so;  LAPACK version
3.11.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C

time zone: Etc/UTC
tzcode source: system (glibc)

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods
[8] base

other attached packages:
[1] pbmcapply_1.5.1 forecast_8.21   attempt_0.3.1   imputeTS_3.3

loaded via a namespace (and not attached):
 [1] gtable_0.3.3      compiler_4.3.0    Rcpp_1.0.10       xml2_1.3.4
 [5] scales_1.2.1      lattice_0.20-45   ggplot2_3.4.2     R6_2.5.1
 [9] generics_0.1.3    curl_5.0.1        lmtest_0.9-40     ggtext_0.1.2
[13] tibble_3.2.1      munsell_0.5.0     nnet_7.3-17       timeDate_4022.108
[17] pillar_1.9.0      rlang_1.1.1       quantmod_0.4.22   utf8_1.2.3
[21] urca_1.3-3        quadprog_1.5-8    cli_3.6.1         magrittr_2.0.3
[25] xts_0.13.1        grid_4.3.0        gridtext_0.1.5    lifecycle_1.0.3
[29] nlme_3.1-155      fracdiff_1.5-2    vctrs_0.6.2       glue_1.6.2
[33] tseries_0.10-54   zoo_1.8-12        fansi_1.0.4       colorspace_2.1-0
[37] TTR_0.24.3        stinepack_1.4     pkgconfig_2.0.3


________________________________
From: Ivan Krylov <krylov.r00t at gmail.com>
Sent: Saturday, June 10, 2023 7:50 PM
To: akshay kulkarni <akshay_e4 at hotmail.com>
Cc: R help Mailing list <r-help at r-project.org>
Subject: Re: [R] inconsistency in mclapply.....

On Sat, 10 Jun 2023 13:46:12 +0000
akshay kulkarni <akshay_e4 at hotmail.com> wrote:
>                    What if I switch to Ubuntu?
> I had a similar ( but not exactly the same) problem way back in 2018
> and switching to Ubuntu solved it.
You shouldn't have to switch distros for problems like this one.

What you need to switch is your OpenBLAS build: instead of using
OpenBLAS built with OpenMP support (which isn't even the default
when compiling OpenBLAS), use OpenBLAS built with pthreads support. It's
also likely, but not 100% guaranteed, that setting OMP_NUM_THREADS=1
will avoid the issue by never starting any additional threads for
OpenBLAS purposes. At least some documentation says that
OPENBLAS_NUM_THREADS is ignored on OpenMP builds of OpenBLAS, you
actually have to set OMP_NUM_THREADS=1 for it to take any effect.

I know very little about RHEL, but it seems that you already have a
pthreads build of OpenBLAS installed in
/usr/lib64/libopenblasp-r0.3.15.so; you just need to adjust the path to
the BLAS implementation used by R. If not, install the openblas-threads
package.

Can you use flexiblas
<https://cran.r-project.org/bin/linux/fedora/#blaslapack-switching> to
switch the BLAS implementation to OPENBLAS-THREADS? This should be
possible by using the flexiblas R package or by setting environment
variables
<https://github.com/mpimd-csc/flexiblas#selecting-the-backend-at-runtime>.

That said, I know for sure that on Debian and Ubuntu you get a choice
between libopenblas0-openmp / libopenblas0-pthread /
libopenblas0-serial, and installing any one of these packages except
libopenblas0-openmp should avoid the problem. (And the thread limit
should be set to 1 if you're running your tasks in parallel by other
means.)

--
Best regards,
Ivan

	[[alternative HTML version deleted]]

? Sat, 17 Jun 2023 18:54:43 +0000
akshay kulkarni <akshay_e4 at hotmail.com> ?????:
>   1.  I was working on a two core machine just to test the
> code...tomorrow will be switching to a 48 core machine. Should I
> again download the latest openblas from ropenblas() or do you suggest
> testing the native openblas that is bundled in Ubuntu
What's the version of OpenBLAS in the Ubuntu release you are running?
Unless you can see significant improvements regarding your CPU
architecture appearing since the version included in Ubuntu was
released, there's little reason to install a different version
manually. The whole point of a GNU/Linux distro is not having to
compile everything yourself. Ubuntu's build of OpenBLAS contains
multiple copies of the code targeted at different processors, including
yours.
> 2.  How do you pass the version of openblas to be installed with
> ropenblas()? The full file name? or just something like this:
> libopenblas_skylakexp-r0.3.23.so  ?
It looks like this function accepts the names of git tags in the
official OpenBLAS Git repository, but it's really the job of the
package maintainer to document important arguments like this one
properly. You can call bug.report(package = 'ropenblas') or write an
e-mail to maintainer('ropenblas') or locate the author's GitHub and
ask
a question there.

-- 
Best regards,
Ivan