R devel - Aug 2024 - Another issue using multi-processing linear algebra libraries

From the R Installation and Admin manual:

	"There is a tendency for re-distributors of R to use ?enhanced? linear
algebra libraries without explaining their downsides.?

There?s a downside not mentioned in the manual that caught and baffled me for a
while. I was using all 64 cores of an AWS instance via parallel::mclapply() and
doing matrix multiplications in the parallelized function. If the matrices were
big enough the linked BLAS or LAPACK would try to use all 64 cores for each
multiplication, which meant 64^2 processes or threads in some combination and
that was the end of all useful work. I worked around the problem by rewriting
the matrix multiply as ?colSums(x * t(y))?. It also worked to build R from
source, which I guess uses the built-in BLAS and LAPACK.

Would it make sense to add a parameter somewhere, to mclapply(), say, telling R
to not use multiprocessing libraries? Does R even know whether a linked library
is doing multi-processing? Does R build its own BLAS and LAPACK if its also
linking external ones?

Thanks,
Rob
	[[alternative HTML version deleted]]

? Tue, 6 Aug 2024 10:19:25 -0400
Rob Steele via R-devel <r-devel at r-project.org> ?????:
> Would it make sense to add a parameter somewhere, to mclapply(), say,
> telling R to not use multiprocessing libraries?
It would be great if we had a way to limit all kinds of multiprocessing
(child processes, OpenMP threads, pthreads, forks, MPI, PVM, 'parallel'
clusters, <insert many more methods here>) in a single setting, but
there is currently no such setting, and it may be impossible to
implement. Particularly problematic may be nested parallellism:
sometimes desirable (e.g. 4-machine cluster, each machine in it using
OpenMP threads), sometimes undesired (e.g. your case). A single setting
is probably far from enough.
> Does R even know whether a linked library is doing multi-processing?
Unfortunately, no, there is no standard interface for that. Best I can
recommend is to link your R installation with FlexiBLAS and then use the
'flexiblas' CRAN package to talk to it.
> Does R build its own BLAS and LAPACK if its also linking external
> ones?
I think it doesn't.

-- 
Best regards,
Ivan