Dear knowledgeable experts :-), I am trying to get openmpi, Rmpi and SNOW running on a Myrinet/GM cluster. I'm not an IT expert, but I surely could use a working installation of Rmpi and SNOW. I try to load the Rmpi library and get the following: > library(Rmpi) [master:07230] mca: base: component_find: unable to open osc pt2pt: file not found (ignored) -------------------------------------------------------------------------- [0,0,0]: Myrinet/GM on host master was unable to find any NICs. Another transport will be used instead, although this may result in lower performance. Then if I start a cluster, only nodes on Master can be seen and initiated Here is environment, configuration and installation information: # configuration of openmpi ./configure --prefix=/opt/openmpi --with-gm=/opt/gm/2.1.21/2.6.11-21smp --with-sge=/opt/sge --disable-mpi-f90 #installation of Rmpi /usr/local/bin/R CMD INSTALL Rmpi_0.5-5.tar.gz \ --configure-args=--with-mpi=/opt/openmpi R version 2.6.1 (2007-11-26) #master environment pearman at master:~> env LESSKEY=/etc/lesskey.bin NNTPSERVER=news INFODIR=/usr/local/info:/usr/share/info:/usr/info MANPATH=/opt/sge/man:/usr/local/man:/usr/local/share/man:/usr/share/man:/usr/X11R6/man:/opt/gnome/share/man:/opt/c3-4/man HOSTNAME=master GNOME2_PATH=/usr/local:/opt/gnome:/usr XKEYSYMDB=/usr/X11R6/lib/X11/XKeysymDB HOST=master TERM=xterm SHELL=/bin/bash PROFILEREAD=true HISTSIZE=1000 SSH_CLIENT=::ffff:10.15.1.179 51565 22 QTDIR=/usr/lib/qt3 OLDPWD=/opt/openmpi/bin SSH_TTY=/dev/pts/6 GROFF_NO_SGR=yes JRE_HOME=/usr/lib/jvm/java/jre USER=pearman LS_COLORS=no=00:fi=00:di=01;34:ln=00;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=4 0;31:ex=00;32:*.cmd=00;32:*.exe=01;32:*.com=01;32:*.bat=01;32:*.btm=01;32:*.dll=01;32:*.tar=00; 31:*.tbz=00;31:*.tgz=00;31:*.rpm=00;31:*.deb=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00 ;31:*.zoo=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.tb2=00;31:*.tz2=00;31:*.tbz2=00;31 :*.avi=01;35:*.bmp=01;35:*.fli=01;35:*.gif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mng=01;35:*.mov=01; 35:*.mpg=01;35:*.pcx=01;35:*.pbm=01;35:*.pgm=01;35:*.png=01;35:*.ppm=01;35:*.tga=01;35:*.tif=01 ;35:*.xbm=01;35:*.xpm=01;35:*.dl=01;35:*.gl=01;35:*.aiff=00;32:*.au=00;32:*.mid=00;32:*.mp3=00; 32:*.ogg=00;32:*.voc=00;32:*.wav=00;32: LD_LIBRARY_PATH=/opt/openmpi/lib:/opt/sge/lib/lx24-amd64 XNLSPATH=/usr/X11R6/lib/X11/nls HOSTTYPE=x86_64 PAGER=less XDG_CONFIG_DIRS=/usr/local/etc/xdg/:/etc/xdg/:/etc/opt/gnome/xdg/ C3_PATH=/opt/c3-4 MINICOM=-c on MAIL=/var/mail/pearman PATH=/opt/openmpi/bin:/opt/sge/bin/lx24-amd64:/opt/mpich/1.2.6..14b/x86_64/gcc//bin:/opt/gm/2.1 .21/2.6.11-21smp/bin:/opt/c3-4:/home/pearman/bin:/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/u sr/games:/opt/gnome/bin:/opt/kde3/bin:/opt/pathscale/bin CPU=x86_64 JAVA_BINDIR=/usr/lib/jvm/java/bin INPUTRC=/etc/inputrc PWD=/home/pearman JAVA_HOME=/usr/lib/jvm/java LANG=en_US.UTF-8 PYTHONSTARTUP=/etc/pythonstart SGE_ROOT=/opt/sge SDK_HOME=/usr/lib/jvm/java C3_RSH=ssh TEXINPUTS=:/home/pearman/.TeX:/usr/share/doc/.TeX:/usr/doc/.TeX JDK_HOME=/usr/lib/jvm/java SHLVL=1 HOME=/home/pearman LESS_ADVANCED_PREPROCESSOR=no OSTYPE=linux LS_OPTIONS=-N --color=tty -T 0 XCURSOR_THEME=crystalwhite WINDOWMANAGER=/usr/X11R6/bin/kde GTK_PATH=/usr/local/lib/gtk-2.0:/opt/gnome/lib/gtk-2.0:/usr/lib/gtk-2.0 GM_PATH=/opt/gm/2.1.21/2.6.11-21smp G_FILENAME_ENCODING=@locale,UTF-8,ISO-8859-15,CP1252 LESS=-M -I MACHTYPE=x86_64-suse-linux LOGNAME=pearman GTK_PATH64=/usr/local/lib64/gtk-2.0:/opt/gnome/lib64/gtk-2.0:/usr/lib64/gtk-2.0 CVS_RSH=ssh XDG_DATA_DIRS=/usr/local/share/:/usr/share/:/etc/opt/kde3/share/:/opt/kde3/share/:/opt/gnome/sh are/ MPICH_PATH=/opt/mpich/1.2.6..14b/x86_64/gcc/ ACLOCAL_FLAGS=-I /opt/gnome/share/aclocal SSH_CONNECTION=::ffff:10.15.1.179 51565 ::ffff:10.30.1.15 22 PKG_CONFIG_PATH=/opt/gnome/lib64/pkgconfig LESSOPEN=lessopen.sh %s INFOPATH=/usr/local/info:/usr/share/info:/usr/info:/opt/gnome/share/info LESSCLOSE=lessclose.sh %s %s G_BROKEN_FILENAMES=1 JAVA_ROOT=/usr/lib/jvm/java COLORTERM=1 _=/usr/bin/env pearman at master:~> which mpirun /opt/openmpi/bin/mpirun Also note: mpich is also installed and is also in the PATH, after openmpi BTW. I have seen a number of posts concerning the pt2pt error message. Still, I was unable to understand how they might apply to fixing the current problem. Help would be greatly appreciated. Peter
See http://www.open-mpi.org/community/lists/users/2007/10/4192.php Does the version of Rmpi you used have the suggested patch? On Mon, 7 Apr 2008, Peter Pearman wrote:> Dear knowledgeable experts :-), > > I am trying to get openmpi, Rmpi and SNOW running on a Myrinet/GM > cluster. I'm not an IT expert, but I surely could use a working > installation of Rmpi and SNOW. > > I try to load the Rmpi library and get the following: > > > library(Rmpi) > [master:07230] mca: base: component_find: unable to open osc pt2pt: file > not found (ignored) > -------------------------------------------------------------------------- > [0,0,0]: Myrinet/GM on host master was unable to find any NICs. > Another transport will be used instead, although this may result in > lower performance. > > Then if I start a cluster, only nodes on Master can be seen and initiated > > Here is environment, configuration and installation information: > > > # configuration of openmpi > ./configure --prefix=/opt/openmpi --with-gm=/opt/gm/2.1.21/2.6.11-21smp > --with-sge=/opt/sge --disable-mpi-f90 > > #installation of Rmpi > /usr/local/bin/R CMD INSTALL Rmpi_0.5-5.tar.gz \ > --configure-args=--with-mpi=/opt/openmpi > > R version 2.6.1 (2007-11-26) > > #master environment > pearman at master:~> env > LESSKEY=/etc/lesskey.bin > NNTPSERVER=news > INFODIR=/usr/local/info:/usr/share/info:/usr/info > MANPATH=/opt/sge/man:/usr/local/man:/usr/local/share/man:/usr/share/man:/usr/X11R6/man:/opt/gnome/share/man:/opt/c3-4/man > HOSTNAME=master > GNOME2_PATH=/usr/local:/opt/gnome:/usr > XKEYSYMDB=/usr/X11R6/lib/X11/XKeysymDB > HOST=master > TERM=xterm > SHELL=/bin/bash > PROFILEREAD=true > HISTSIZE=1000 > SSH_CLIENT=::ffff:10.15.1.179 51565 22 > QTDIR=/usr/lib/qt3 > OLDPWD=/opt/openmpi/bin > SSH_TTY=/dev/pts/6 > GROFF_NO_SGR=yes > JRE_HOME=/usr/lib/jvm/java/jre > USER=pearman > LS_COLORS=no=00:fi=00:di=01;34:ln=00;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=4 > 0;31:ex=00;32:*.cmd=00;32:*.exe=01;32:*.com=01;32:*.bat=01;32:*.btm=01;32:*.dll=01;32:*.tar=00; > 31:*.tbz=00;31:*.tgz=00;31:*.rpm=00;31:*.deb=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00 > ;31:*.zoo=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.tb2=00;31:*.tz2=00;31:*.tbz2=00;31 > :*.avi=01;35:*.bmp=01;35:*.fli=01;35:*.gif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mng=01;35:*.mov=01; > 35:*.mpg=01;35:*.pcx=01;35:*.pbm=01;35:*.pgm=01;35:*.png=01;35:*.ppm=01;35:*.tga=01;35:*.tif=01 > ;35:*.xbm=01;35:*.xpm=01;35:*.dl=01;35:*.gl=01;35:*.aiff=00;32:*.au=00;32:*.mid=00;32:*.mp3=00; > 32:*.ogg=00;32:*.voc=00;32:*.wav=00;32: > LD_LIBRARY_PATH=/opt/openmpi/lib:/opt/sge/lib/lx24-amd64 > XNLSPATH=/usr/X11R6/lib/X11/nls > HOSTTYPE=x86_64 > PAGER=less > XDG_CONFIG_DIRS=/usr/local/etc/xdg/:/etc/xdg/:/etc/opt/gnome/xdg/ > C3_PATH=/opt/c3-4 > MINICOM=-c on > MAIL=/var/mail/pearman > PATH=/opt/openmpi/bin:/opt/sge/bin/lx24-amd64:/opt/mpich/1.2.6..14b/x86_64/gcc//bin:/opt/gm/2.1 > .21/2.6.11-21smp/bin:/opt/c3-4:/home/pearman/bin:/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/u > sr/games:/opt/gnome/bin:/opt/kde3/bin:/opt/pathscale/bin > CPU=x86_64 > JAVA_BINDIR=/usr/lib/jvm/java/bin > INPUTRC=/etc/inputrc > PWD=/home/pearman > JAVA_HOME=/usr/lib/jvm/java > LANG=en_US.UTF-8 > PYTHONSTARTUP=/etc/pythonstart > SGE_ROOT=/opt/sge > SDK_HOME=/usr/lib/jvm/java > C3_RSH=ssh > TEXINPUTS=:/home/pearman/.TeX:/usr/share/doc/.TeX:/usr/doc/.TeX > JDK_HOME=/usr/lib/jvm/java > SHLVL=1 > HOME=/home/pearman > LESS_ADVANCED_PREPROCESSOR=no > OSTYPE=linux > LS_OPTIONS=-N --color=tty -T 0 > XCURSOR_THEME=crystalwhite > WINDOWMANAGER=/usr/X11R6/bin/kde > GTK_PATH=/usr/local/lib/gtk-2.0:/opt/gnome/lib/gtk-2.0:/usr/lib/gtk-2.0 > GM_PATH=/opt/gm/2.1.21/2.6.11-21smp > G_FILENAME_ENCODING=@locale,UTF-8,ISO-8859-15,CP1252 > LESS=-M -I > MACHTYPE=x86_64-suse-linux > LOGNAME=pearman > GTK_PATH64=/usr/local/lib64/gtk-2.0:/opt/gnome/lib64/gtk-2.0:/usr/lib64/gtk-2.0 > CVS_RSH=ssh > XDG_DATA_DIRS=/usr/local/share/:/usr/share/:/etc/opt/kde3/share/:/opt/kde3/share/:/opt/gnome/sh > are/ > MPICH_PATH=/opt/mpich/1.2.6..14b/x86_64/gcc/ > ACLOCAL_FLAGS=-I /opt/gnome/share/aclocal > SSH_CONNECTION=::ffff:10.15.1.179 51565 ::ffff:10.30.1.15 22 > PKG_CONFIG_PATH=/opt/gnome/lib64/pkgconfig > LESSOPEN=lessopen.sh %s > INFOPATH=/usr/local/info:/usr/share/info:/usr/info:/opt/gnome/share/info > LESSCLOSE=lessclose.sh %s %s > G_BROKEN_FILENAMES=1 > JAVA_ROOT=/usr/lib/jvm/java > COLORTERM=1 > _=/usr/bin/env > > > pearman at master:~> which mpirun > /opt/openmpi/bin/mpirun > > Also note: mpich is also installed and is also in the PATH, after openmpi > > BTW. I have seen a number of posts concerning the pt2pt error message. > Still, I was unable to understand how they might apply to fixing the > current problem. > > Help would be greatly appreciated. > > Peter > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel >-- Brian D. Ripley, ripley at stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UK Fax: +44 1865 272595
You may want to be a little careful about using R and MPI with Myrinet. We tried to get that to work around 5 years ago and found it more trouble than it was worth (we ended up sending the Myrinet equipment back and getting gigabit ethernet instead). As I recall, the problem at the time was that the Myrinet libraries used their own malloc (or assumed that their own malloc was used -- I don't recall which), whereas a standard R build would use the system malloc. The resulting combination would work for small allocations. But for large allocations allocated by R using the standard malloc (large meaning allocated by memory mapping) the Myrinet internals would assume these allocations had been allocated by their malloc and start doing things with them that lead to crashes. I'm fuzzy on the details but this should give you an idea of what to look out for. Best, luke On Mon, 7 Apr 2008, Peter Pearman wrote:> Dear knowledgeable experts :-), > > I am trying to get openmpi, Rmpi and SNOW running on a Myrinet/GM > cluster. I'm not an IT expert, but I surely could use a working > installation of Rmpi and SNOW. > > I try to load the Rmpi library and get the following: > > > library(Rmpi) > [master:07230] mca: base: component_find: unable to open osc pt2pt: file > not found (ignored) > -------------------------------------------------------------------------- > [0,0,0]: Myrinet/GM on host master was unable to find any NICs. > Another transport will be used instead, although this may result in > lower performance. > > Then if I start a cluster, only nodes on Master can be seen and initiated > > Here is environment, configuration and installation information: > > > # configuration of openmpi > ./configure --prefix=/opt/openmpi --with-gm=/opt/gm/2.1.21/2.6.11-21smp > --with-sge=/opt/sge --disable-mpi-f90 > > #installation of Rmpi > /usr/local/bin/R CMD INSTALL Rmpi_0.5-5.tar.gz \ > --configure-args=--with-mpi=/opt/openmpi > > R version 2.6.1 (2007-11-26) > > #master environment > pearman at master:~> env > LESSKEY=/etc/lesskey.bin > NNTPSERVER=news > INFODIR=/usr/local/info:/usr/share/info:/usr/info > MANPATH=/opt/sge/man:/usr/local/man:/usr/local/share/man:/usr/share/man:/usr/X11R6/man:/opt/gnome/share/man:/opt/c3-4/man > HOSTNAME=master > GNOME2_PATH=/usr/local:/opt/gnome:/usr > XKEYSYMDB=/usr/X11R6/lib/X11/XKeysymDB > HOST=master > TERM=xterm > SHELL=/bin/bash > PROFILEREAD=true > HISTSIZE=1000 > SSH_CLIENT=::ffff:10.15.1.179 51565 22 > QTDIR=/usr/lib/qt3 > OLDPWD=/opt/openmpi/bin > SSH_TTY=/dev/pts/6 > GROFF_NO_SGR=yes > JRE_HOME=/usr/lib/jvm/java/jre > USER=pearman > LS_COLORS=no=00:fi=00:di=01;34:ln=00;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=4 > 0;31:ex=00;32:*.cmd=00;32:*.exe=01;32:*.com=01;32:*.bat=01;32:*.btm=01;32:*.dll=01;32:*.tar=00; > 31:*.tbz=00;31:*.tgz=00;31:*.rpm=00;31:*.deb=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00 > ;31:*.zoo=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.tb2=00;31:*.tz2=00;31:*.tbz2=00;31 > :*.avi=01;35:*.bmp=01;35:*.fli=01;35:*.gif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mng=01;35:*.mov=01; > 35:*.mpg=01;35:*.pcx=01;35:*.pbm=01;35:*.pgm=01;35:*.png=01;35:*.ppm=01;35:*.tga=01;35:*.tif=01 > ;35:*.xbm=01;35:*.xpm=01;35:*.dl=01;35:*.gl=01;35:*.aiff=00;32:*.au=00;32:*.mid=00;32:*.mp3=00; > 32:*.ogg=00;32:*.voc=00;32:*.wav=00;32: > LD_LIBRARY_PATH=/opt/openmpi/lib:/opt/sge/lib/lx24-amd64 > XNLSPATH=/usr/X11R6/lib/X11/nls > HOSTTYPE=x86_64 > PAGER=less > XDG_CONFIG_DIRS=/usr/local/etc/xdg/:/etc/xdg/:/etc/opt/gnome/xdg/ > C3_PATH=/opt/c3-4 > MINICOM=-c on > MAIL=/var/mail/pearman > PATH=/opt/openmpi/bin:/opt/sge/bin/lx24-amd64:/opt/mpich/1.2.6..14b/x86_64/gcc//bin:/opt/gm/2.1 > .21/2.6.11-21smp/bin:/opt/c3-4:/home/pearman/bin:/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/u > sr/games:/opt/gnome/bin:/opt/kde3/bin:/opt/pathscale/bin > CPU=x86_64 > JAVA_BINDIR=/usr/lib/jvm/java/bin > INPUTRC=/etc/inputrc > PWD=/home/pearman > JAVA_HOME=/usr/lib/jvm/java > LANG=en_US.UTF-8 > PYTHONSTARTUP=/etc/pythonstart > SGE_ROOT=/opt/sge > SDK_HOME=/usr/lib/jvm/java > C3_RSH=ssh > TEXINPUTS=:/home/pearman/.TeX:/usr/share/doc/.TeX:/usr/doc/.TeX > JDK_HOME=/usr/lib/jvm/java > SHLVL=1 > HOME=/home/pearman > LESS_ADVANCED_PREPROCESSOR=no > OSTYPE=linux > LS_OPTIONS=-N --color=tty -T 0 > XCURSOR_THEME=crystalwhite > WINDOWMANAGER=/usr/X11R6/bin/kde > GTK_PATH=/usr/local/lib/gtk-2.0:/opt/gnome/lib/gtk-2.0:/usr/lib/gtk-2.0 > GM_PATH=/opt/gm/2.1.21/2.6.11-21smp > G_FILENAME_ENCODING=@locale,UTF-8,ISO-8859-15,CP1252 > LESS=-M -I > MACHTYPE=x86_64-suse-linux > LOGNAME=pearman > GTK_PATH64=/usr/local/lib64/gtk-2.0:/opt/gnome/lib64/gtk-2.0:/usr/lib64/gtk-2.0 > CVS_RSH=ssh > XDG_DATA_DIRS=/usr/local/share/:/usr/share/:/etc/opt/kde3/share/:/opt/kde3/share/:/opt/gnome/sh > are/ > MPICH_PATH=/opt/mpich/1.2.6..14b/x86_64/gcc/ > ACLOCAL_FLAGS=-I /opt/gnome/share/aclocal > SSH_CONNECTION=::ffff:10.15.1.179 51565 ::ffff:10.30.1.15 22 > PKG_CONFIG_PATH=/opt/gnome/lib64/pkgconfig > LESSOPEN=lessopen.sh %s > INFOPATH=/usr/local/info:/usr/share/info:/usr/info:/opt/gnome/share/info > LESSCLOSE=lessclose.sh %s %s > G_BROKEN_FILENAMES=1 > JAVA_ROOT=/usr/lib/jvm/java > COLORTERM=1 > _=/usr/bin/env > > > pearman at master:~> which mpirun > /opt/openmpi/bin/mpirun > > Also note: mpich is also installed and is also in the PATH, after openmpi > > BTW. I have seen a number of posts concerning the pt2pt error message. > Still, I was unable to understand how they might apply to fixing the > current problem. > > Help would be greatly appreciated. > > Peter > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel >-- Luke Tierney Chair, Statistics and Actuarial Science Ralph E. Wareham Professor of Mathematical Sciences University of Iowa Phone: 319-335-3386 Department of Statistics and Fax: 319-335-3017 Actuarial Science 241 Schaeffer Hall email: luke at stat.uiowa.edu Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu