R devel - Dec 2010 - R with ATLAS avoids Linux cpu affinity

Hi all,

I have a problem with cpu affinity in my R-2.11.1 installation compiled against
ATLAS running on a Linux (Ubuntu 10.04) cluster under GridEngine.  I wish to use
Grid Engine's core binding feature to bind user processes into the number of
cores they request on the cluster, thus preventing badly behaved multi-threaded
libraries from consuming more cores than requested.  An example of this is R
compiled against multithreaded ATLAS, which needs to be bound into a single core
if a user submits a 1 core job.  Grid Engine achieves this through the
sched_setaffinity system call under Linux 2.6.  For most applications (including
if I write a test C program that uses ATLAS BLAS), this works well, and prevents
threads from 'leaking' outside the cpu set they are assigned.  However,
R appears to be able to avoid the core binding.  This is *very* strange as I was
under the impression that any child processes or threads inherit the cpu
affinity of the parent.

Does anyone have experience of this and could offer a comment or solution?

Thanks,

Chris

--
Dr Chris Jewell
Department of Statistics
University of Warwick
Coventry
CV4 7AL
UK
Tel: +44 (0)24 7615 0778

Chris,

On 6 December 2010 at 15:43, Chris Jewell wrote:
| Hi all,
| 
| I have a problem with cpu affinity in my R-2.11.1 installation compiled
against ATLAS running on a Linux (Ubuntu 10.04) cluster under GridEngine.  I
wish to use Grid Engine's core binding feature to bind user processes into
the number of cores they request on the cluster, thus preventing badly behaved
multi-threaded libraries from consuming more cores than requested.  An example
of this is R compiled against multithreaded ATLAS, which needs to be bound into
a single core if a user submits a 1 core job.  Grid Engine achieves this through
the sched_setaffinity system call under Linux 2.6.  For most applications
(including if I write a test C program that uses ATLAS BLAS), this works well,
and prevents threads from 'leaking' outside the cpu set they are
assigned.  However, R appears to be able to avoid the core binding.  This is
*very* strange as I was under the impression that any child processes or threads
inherit the cpu affinity of the parent.
| 
| Does anyone have experience of this and could offer a comment or solution? 

Atlas will _always_ use all the cores 'compiled in'.  

Ubuntu's current package just uses one (as it is not a multithreaded
build). This will change with future package as per the Debian / Ubuntu
package maintainer.  If you built Atlas locally, you may be use all cores
(depending on how you built).

There is simply no way to 'scale up or down' dynamically with Atlas --
but
both MKL and GotoBLAS2 can do this.  See my package / paper on BLAS and GPU
benchmarking (cf http://dirk.eddelbuettel.com/blog/code/gcbd/ for two posts
and links) for details.

AFAICT R should not alter CPU affinity.  So if I were you I'd swap the BLAS
implementation and try again.  As you are on Ubuntu, you can try the MKL that
comes with the (now a littler older) Revolution R in Ubuntu 9.10; otherwise I
can highly recommend the GotoBLAS2 helper package listed in my paper for a
local GotoBLAS2 built.   The script may be out of sync with the fairly recent
license change of GotoBLAS2 (to the more liberal BSD license permitting
redistribution).  With some luck we will GotoBLAS2 deb packages in future
Debian and Ubuntu releases.

Hth,  Dirk

-- 
Dirk Eddelbuettel | edd at debian.org | http://dirk.eddelbuettel.com