Hi All,
we are trying to install lustre-1.4.6-x86_64
on Intel Xeon EM64T machines having topspin HCA card
and topspin switch.
OS: RHELAS4-U2
installed,
kernel-smp-2.6.9-34.EL_lustre.1.4.6.2.x86_64.rpm
kernel-source-2.6.9-34.EL_lustre.1.4.6.2.x86_64.rpm
lustre-1.4.6.2-2.6.9_34.EL_lustre.1.4.6.2smp.x86_64.rpm
lustre-debuginfo-1.4.6.2-2.6.9_34.EL_lustre.1.4.6.2smp.x86_64.rpm
lustre-modules-1.4.6.2-2.6.9_34.EL_lustre.1.4.6.2smp.x86_64.rpm
lustre-source-1.4.6.2-2.6.9_34.EL_lustre.1.4.6.2smp.x86_64.rpm
lustre RPMS,
in "/etc/modprobe.conf" file added line
options lnet networks=openib0,tcp0
in "/etc/sysconfig/network-scripts/ifcfg-ib0" file
DEVICE=ib0
BOOTPROTO=static
IPADDR=10.255.255.229
NETMASK=255.255.255.0
ONBOOT=yes
after that rebooted into luster kernel, but we are
not getting ib0 interface in "ifconfig", what may
be wrong? or missing?????
After refering to openib site and rpms, added
line in modprobe.conf to point to
modprobe-openib.conf file, that contains
alias ib0 ib_ipoib
alias ib1 ib_ipoib
install ib_ipoib /sbin/modprobe ib_mthca && \
/sbin/modprobe --first-time --ignore-install
ib_ipoib
install ib_mthca /sbin/modprobe --first-time
--ignore-install ib_mthca && \
/sbin/modprobe ib_umad && \
/sbin/modprobe ib_uverbs && \
/sbin/modprobe ib_uat && \
/sbin/modprobe ib_ucm && \
/sbin/udevstart > /dev/null 2>&1
after this I am getting the ib0 interface, but
when I try start the interface, it gives error saying
address already been used by some other host, any
idea????, I can set the same IP using ifconfig
commandline, pls suggest some solution to solve
this.
thanks in advance,
-bala-
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Wang Yibin
2006-Jun-18 11:18 UTC
[Lustre-discuss] configuring HCA interface(openib) in lustre
Bala wrote:> Hi All, > we are trying to install lustre-1.4.6-x86_64 > on Intel Xeon EM64T machines having topspin HCA card > and topspin switch. > > OS: RHELAS4-U2 > installed, > kernel-smp-2.6.9-34.EL_lustre.1.4.6.2.x86_64.rpm > kernel-source-2.6.9-34.EL_lustre.1.4.6.2.x86_64.rpm > lustre-1.4.6.2-2.6.9_34.EL_lustre.1.4.6.2smp.x86_64.rpm > lustre-debuginfo-1.4.6.2-2.6.9_34.EL_lustre.1.4.6.2smp.x86_64.rpm > lustre-modules-1.4.6.2-2.6.9_34.EL_lustre.1.4.6.2smp.x86_64.rpm > lustre-source-1.4.6.2-2.6.9_34.EL_lustre.1.4.6.2smp.x86_64.rpm > > lustre RPMS, > > in "/etc/modprobe.conf" file added line > > options lnet networks=openib0,tcp0 > > in "/etc/sysconfig/network-scripts/ifcfg-ib0" file > > DEVICE=ib0 > BOOTPROTO=static > IPADDR=10.255.255.229 > NETMASK=255.255.255.0 > ONBOOT=yes > > after that rebooted into luster kernel, but we are > not getting ib0 interface in "ifconfig", what may > be wrong? or missing????? > > After refering to openib site and rpms, added > line in modprobe.conf to point to > modprobe-openib.conf file, that contains > > alias ib0 ib_ipoib > alias ib1 ib_ipoib > install ib_ipoib /sbin/modprobe ib_mthca && \ > /sbin/modprobe --first-time --ignore-install > ib_ipoib > install ib_mthca /sbin/modprobe --first-time > --ignore-install ib_mthca && \ > /sbin/modprobe ib_umad && \ > /sbin/modprobe ib_uverbs && \ > /sbin/modprobe ib_uat && \ > /sbin/modprobe ib_ucm && \ > /sbin/udevstart > /dev/null 2>&1 > > after this I am getting the ib0 interface, but > when I try start the interface, it gives error saying > address already been used by some other host, any > idea????, I can set the same IP using ifconfig > commandline, pls suggest some solution to solve > this. >Since it indicates that IP address was used by other host, it''s probably true. I suggest you shutdown the IB interface and see if you can ping to that IP address. What does ''ifconfig -a'' show? Do you have I/B SM configured? If yes, how? Are you using a different(or the same) subnet to ethernet?> thanks in advance, > -bala- > > > > > > > __________________________________________________ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > >
Peter Kjellström
2006-Jun-18 11:18 UTC
[Lustre-discuss] configuring HCA interface(openib) in lustre
On Tuesday 13 June 2006 13:51, Bala wrote:> Hi All, > we are trying to install lustre-1.4.6-x86_64 > on Intel Xeon EM64T machines having topspin HCA card > and topspin switch. > > ... > > in "/etc/modprobe.conf" file added line > > options lnet networks=openib0,tcp0I think there is some confusion here. openib in lnet-talk is the native gen2 (openfabrics/openib) lnd, it has nothing to do with the tcp/ip interface (ib0) you''ve been fighting with. If you want to use the tcp/ip over infiniband then you only need the tcp lnd. /Peter> in "/etc/sysconfig/network-scripts/ifcfg-ib0" file > > DEVICE=ib0 > BOOTPROTO=static > IPADDR=10.255.255.229 > NETMASK=255.255.255.0 > ONBOOT=yes > > after that rebooted into luster kernel, but we are > not getting ib0 interface in "ifconfig", what may > be wrong? or missing????? > > After refering to openib site and rpms, added > line in modprobe.conf to point to > modprobe-openib.conf file, that contains > > alias ib0 ib_ipoib > alias ib1 ib_ipoib > install ib_ipoib /sbin/modprobe ib_mthca && \ > /sbin/modprobe --first-time --ignore-install > ib_ipoib > install ib_mthca /sbin/modprobe --first-time > --ignore-install ib_mthca && \ > /sbin/modprobe ib_umad && \ > /sbin/modprobe ib_uverbs && \ > /sbin/modprobe ib_uat && \ > /sbin/modprobe ib_ucm && \ > /sbin/udevstart > /dev/null 2>&1 > > after this I am getting the ib0 interface, but > when I try start the interface, it gives error saying > address already been used by some other host, any > idea????, I can set the same IP using ifconfig > commandline, pls suggest some solution to solve > this. > > thanks in advance, > -bala--- ------------------------------------------------------------ Peter Kjellstr?m | National Supercomputer Centre | Sweden | http://www.nsc.liu.se -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: not available Url : http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20060614/4a6e9e35/attachment-0001.bin
Peter Kjellström
2006-Jun-18 11:18 UTC
[Lustre-discuss] configuring HCA interface(openib) in lustre
On Wednesday 14 June 2006 12:11, Eric Barton wrote:> On Wed, 2006-06-14 at 09:53 +0200, Peter Kjellstr?m wrote: > > On Tuesday 13 June 2006 13:51, Bala wrote: > > I think there is some confusion here. openib in lnet-talk is the native > > gen2 (openfabrics/openib) lnd, it has nothing to do with the tcp/ip > > interface (ib0) you''ve been fighting with. > > Sorry to butt in, but just to be completely accurate, here are the LNET > IB network types and the stacks they interface to...Of course I had the names mixed up. I was building openmpi around the time I wrote this and they call their gen2 driver *drum roll* openib... Thanks for spotting it, Peter> LNET network name Underlying software stack > openib OpenIB gen1 / Mellanox Gold > iib Silverstorm (Infinicon) > vib Voltaire > o2ib OpenIB gen2 > cib Cisco > > > If you want to use the tcp/ip over infiniband then you only need the tcp > > lnd. > > Yes absolutely. > > Cheers, > Eric-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: not available Url : http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20060615/d77e1339/attachment-0001.bin
Eric Barton
2006-Jun-18 11:18 UTC
[Lustre-discuss] configuring HCA interface(openib) in lustre
On Wed, 2006-06-14 at 09:53 +0200, Peter Kjellstr?m wrote:> On Tuesday 13 June 2006 13:51, Bala wrote: > I think there is some confusion here. openib in lnet-talk is the native gen2 > (openfabrics/openib) lnd, it has nothing to do with the tcp/ip interface > (ib0) you''ve been fighting with.Sorry to butt in, but just to be completely accurate, here are the LNET IB network types and the stacks they interface to... LNET network name Underlying software stack openib OpenIB gen1 / Mellanox Gold iib Silverstorm (Infinicon) vib Voltaire o2ib OpenIB gen2 cib Cisco> If you want to use the tcp/ip over infiniband then you only need the tcp lnd.Yes absolutely. Cheers, Eric