On Sat, 10 Jun 2023 13:46:12 +0000 akshay kulkarni <akshay_e4 at hotmail.com> wrote:> What if I switch to Ubuntu? > I had a similar ( but not exactly the same) problem way back in 2018 > and switching to Ubuntu solved it.You shouldn't have to switch distros for problems like this one. What you need to switch is your OpenBLAS build: instead of using OpenBLAS built with OpenMP support (which isn't even the default when compiling OpenBLAS), use OpenBLAS built with pthreads support. It's also likely, but not 100% guaranteed, that setting OMP_NUM_THREADS=1 will avoid the issue by never starting any additional threads for OpenBLAS purposes. At least some documentation says that OPENBLAS_NUM_THREADS is ignored on OpenMP builds of OpenBLAS, you actually have to set OMP_NUM_THREADS=1 for it to take any effect. I know very little about RHEL, but it seems that you already have a pthreads build of OpenBLAS installed in /usr/lib64/libopenblasp-r0.3.15.so; you just need to adjust the path to the BLAS implementation used by R. If not, install the openblas-threads package. Can you use flexiblas <https://cran.r-project.org/bin/linux/fedora/#blaslapack-switching> to switch the BLAS implementation to OPENBLAS-THREADS? This should be possible by using the flexiblas R package or by setting environment variables <https://github.com/mpimd-csc/flexiblas#selecting-the-backend-at-runtime>. That said, I know for sure that on Debian and Ubuntu you get a choice between libopenblas0-openmp / libopenblas0-pthread / libopenblas0-serial, and installing any one of these packages except libopenblas0-openmp should avoid the problem. (And the thread limit should be set to 1 if you're running your tasks in parallel by other means.) -- Best regards, Ivan
Dear Ivan,
A million thanks for your precious time....
Thanking you
Yours sincerely,
AkshayMKulkarni
________________________________
From: Ivan Krylov <krylov.r00t at gmail.com>
Sent: Saturday, June 10, 2023 7:50 PM
To: akshay kulkarni <akshay_e4 at hotmail.com>
Cc: R help Mailing list <r-help at r-project.org>
Subject: Re: [R] inconsistency in mclapply.....
On Sat, 10 Jun 2023 13:46:12 +0000
akshay kulkarni <akshay_e4 at hotmail.com> wrote:
> What if I switch to Ubuntu?
> I had a similar ( but not exactly the same) problem way back in 2018
> and switching to Ubuntu solved it.
You shouldn't have to switch distros for problems like this one.
What you need to switch is your OpenBLAS build: instead of using
OpenBLAS built with OpenMP support (which isn't even the default
when compiling OpenBLAS), use OpenBLAS built with pthreads support. It's
also likely, but not 100% guaranteed, that setting OMP_NUM_THREADS=1
will avoid the issue by never starting any additional threads for
OpenBLAS purposes. At least some documentation says that
OPENBLAS_NUM_THREADS is ignored on OpenMP builds of OpenBLAS, you
actually have to set OMP_NUM_THREADS=1 for it to take any effect.
I know very little about RHEL, but it seems that you already have a
pthreads build of OpenBLAS installed in
/usr/lib64/libopenblasp-r0.3.15.so; you just need to adjust the path to
the BLAS implementation used by R. If not, install the openblas-threads
package.
Can you use flexiblas
<https://cran.r-project.org/bin/linux/fedora/#blaslapack-switching> to
switch the BLAS implementation to OPENBLAS-THREADS? This should be
possible by using the flexiblas R package or by setting environment
variables
<https://github.com/mpimd-csc/flexiblas#selecting-the-backend-at-runtime>.
That said, I know for sure that on Debian and Ubuntu you get a choice
between libopenblas0-openmp / libopenblas0-pthread /
libopenblas0-serial, and installing any one of these packages except
libopenblas0-openmp should avoid the problem. (And the thread limit
should be set to 1 if you're running your tasks in parallel by other
means.)
--
Best regards,
Ivan
[[alternative HTML version deleted]]
Dear Ivan,
I am reopening this thread because there are some new
developments. I have switched to ubuntu and have used ropenblas() in ropenblas
package to download the latest openblas (it is a version of skylake as you can
see below in sessionInfo()). mclapply() is working fine now... My questions:
1. I was working on a two core machine just to test the code...tomorrow will
be switching to a 48 core machine. Should I again download the latest openblas
from ropenblas() or do you suggest testing the native openblas that is bundled
in Ubuntu( it was a basic version of openblas..in my two core machine I tried
the ropenblas() function without testing the native openblas...just for
curiosity) ?
2. How do you pass the version of openblas to be installed with ropenblas()?
The full file name? or just something like this:
libopenblas_skylakexp-r0.3.23.so ?
> sessionInfo()
R version 4.3.0 (2023-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.2 LTS
Matrix products: default
BLAS/LAPACK: /opt/OpenBLAS/lib/libopenblas_skylakexp-r0.3.23.so; LAPACK version
3.11.0
locale:
[1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
[4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
[7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
time zone: Etc/UTC
tzcode source: system (glibc)
attached base packages:
[1] parallel stats graphics grDevices utils datasets methods
[8] base
other attached packages:
[1] pbmcapply_1.5.1 forecast_8.21 attempt_0.3.1 imputeTS_3.3
loaded via a namespace (and not attached):
[1] gtable_0.3.3 compiler_4.3.0 Rcpp_1.0.10 xml2_1.3.4
[5] scales_1.2.1 lattice_0.20-45 ggplot2_3.4.2 R6_2.5.1
[9] generics_0.1.3 curl_5.0.1 lmtest_0.9-40 ggtext_0.1.2
[13] tibble_3.2.1 munsell_0.5.0 nnet_7.3-17 timeDate_4022.108
[17] pillar_1.9.0 rlang_1.1.1 quantmod_0.4.22 utf8_1.2.3
[21] urca_1.3-3 quadprog_1.5-8 cli_3.6.1 magrittr_2.0.3
[25] xts_0.13.1 grid_4.3.0 gridtext_0.1.5 lifecycle_1.0.3
[29] nlme_3.1-155 fracdiff_1.5-2 vctrs_0.6.2 glue_1.6.2
[33] tseries_0.10-54 zoo_1.8-12 fansi_1.0.4 colorspace_2.1-0
[37] TTR_0.24.3 stinepack_1.4 pkgconfig_2.0.3
________________________________
From: Ivan Krylov <krylov.r00t at gmail.com>
Sent: Saturday, June 10, 2023 7:50 PM
To: akshay kulkarni <akshay_e4 at hotmail.com>
Cc: R help Mailing list <r-help at r-project.org>
Subject: Re: [R] inconsistency in mclapply.....
On Sat, 10 Jun 2023 13:46:12 +0000
akshay kulkarni <akshay_e4 at hotmail.com> wrote:
> What if I switch to Ubuntu?
> I had a similar ( but not exactly the same) problem way back in 2018
> and switching to Ubuntu solved it.
You shouldn't have to switch distros for problems like this one.
What you need to switch is your OpenBLAS build: instead of using
OpenBLAS built with OpenMP support (which isn't even the default
when compiling OpenBLAS), use OpenBLAS built with pthreads support. It's
also likely, but not 100% guaranteed, that setting OMP_NUM_THREADS=1
will avoid the issue by never starting any additional threads for
OpenBLAS purposes. At least some documentation says that
OPENBLAS_NUM_THREADS is ignored on OpenMP builds of OpenBLAS, you
actually have to set OMP_NUM_THREADS=1 for it to take any effect.
I know very little about RHEL, but it seems that you already have a
pthreads build of OpenBLAS installed in
/usr/lib64/libopenblasp-r0.3.15.so; you just need to adjust the path to
the BLAS implementation used by R. If not, install the openblas-threads
package.
Can you use flexiblas
<https://cran.r-project.org/bin/linux/fedora/#blaslapack-switching> to
switch the BLAS implementation to OPENBLAS-THREADS? This should be
possible by using the flexiblas R package or by setting environment
variables
<https://github.com/mpimd-csc/flexiblas#selecting-the-backend-at-runtime>.
That said, I know for sure that on Debian and Ubuntu you get a choice
between libopenblas0-openmp / libopenblas0-pthread /
libopenblas0-serial, and installing any one of these packages except
libopenblas0-openmp should avoid the problem. (And the thread limit
should be set to 1 if you're running your tasks in parallel by other
means.)
--
Best regards,
Ivan
[[alternative HTML version deleted]]