Mc Carthy, Fergal
2006-May-19 07:36 UTC
[Lustre-discuss] infiniband configuration on 1.4.6.1
You need to configure an appropriate IB network in the /etc/modprobe.conf lnet
options line. The value you have used, ib0, is not one of the valid options.
Valid selections depending on software stack being used are vib, openib and iib,
i.e. you should have a line that looks like one of the following:
[for Voltaire IB stack]
options lnet networks=vib0
[for Open IB stack]
options lnet networks=openib0
[for (I think) Infinicon IB stack]
options lnet networks=iib0
Whatever option you choose the LNET modprobe will trigger pulling in the
correspond LND module, i.e. vib0 causes kviblnd.ko to be loaded, tcp0 causes
ksocklnd.ko to be loaded.
Please also note that the lnet network entry has nothing to do with the
configured IP over IB devices. The entry networks=vib0 means that you are saying
to use a Voltaire IB LND and you are giving that logical network the identifier
0. If you had an entry like networks=vib0,tcp0, that would mean that you would
be saying you wanted to use both the Voltaire IB LND and the TCP (sockets) LND,
and since you are giving both networks the same logical ID they can be
considered as part of the same logical network grouping. This logical network id
is important for more advanced Lustre/LNET functions such as routing between two
different network types.
I you read the Lustre 1.4.6 user manual there are some detailed explanations of
the lnet options settings.
Fergal.
--
Fergal.McCarthy@HP.com
(The contents of this message and any attachments to it are confidential and may
be legally privileged. If you have received this message in error you should
delete it from your system immediately and advise the sender. To any recipient
of this message within HP, unless otherwise stated, you should consider this
message and attachments as "HP CONFIDENTIAL".)
-----Original Message-----
From: lustre-discuss-bounces@clusterfs.com
[mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of ???
Sent: 07 April 2006 03:52
To: lustre-discuss@clusterfs.com
Subject: [Lustre-discuss] infiniband configuration on 1.4.6.1
Hi,
We tried to install lustre 1.4.6.1 using infiniband(Silverstorm HCA).
I don''t know how to configure luster 1.4.6.1
OS : RHEL 3 update 6
Lustre : 1.4.6.1
Nodes : 5 (1MDS+2OSS, 2Client)
HCA : silver Storm HCA
We are install rpm for lustre 1.4.6.1 on lustre web site.
kernel-smp-2.4.21-37.EL_lustre.1.4.6.1.i686.rpm
kernel-source-2.4.21-37.EL_lustre.1.4.6.1.i686.rpm
lustre-1.4.6.1-2.4.21_37.EL_lustre.1.4.6.1smp.i686.rpm
lustre-debuginfo-1.4.6.1-2.4.21_37.EL_lustre.1.4.6.1smp.i686.rpm
lustre-modules-1.4.6.1-2.4.21_37.EL_lustre.1.4.6.1smp.i686.rpm
lustre-source-1.4.6.1-2.4.21_37.EL_lustre.1.4.6.1smp.i686.rpm
we tried to setup following as lustre 1.4.6 operation Manual
add options lnet network =''ib0'' at /etc/modules.conf
modprobe lnet
lctl network up <= error occurs.
#dmesg
LustreError: 4858:0:(api-ni.c:985:lnet_startup_lndnis()) Can''t load
LND iib, module kiiblnd, rc=0
LustreError: 4865:0:(api-ni.c:985:lnet_startup_lndnis()) Can''t load
LND iib, module kiiblnd, rc=0
LustreError: 4867:0:(api-ni.c:985:lnet_startup_lndnis()) Can''t load
LND iib, module kiiblnd, rc=0
#ifconfig
ib0 Link encap:Ethernet HWaddr 06:06:6A:00:5B:EF
inet addr:10.0.0.2 Bcast:10.0.0.255 Mask:255.255.255.0
1. If we want to use infiniband, Is it impossible to install rpm-version?
2. Is there anyone who success to install lustre using infiniband,
(If Possible) Please send config file for lustre 1.4.6.1 using infiniband
========================================================Regards
Sung Jun, Kim
Supercomputer Operation Lab. Supercomputing Center
KISTI(Korea Institute of Science & Technology Information)
P.O.BOX 122 Yusong Taejon Korea
EMail : sjkim@kisti.re.kr
Phone: +82-42-869-0636
FAX : +82-42-869-0569
========================================================
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss@clusterfs.com
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
Hi,
We tried to install lustre 1.4.6.1 using infiniband(Silverstorm HCA).
I don''t know how to configure luster 1.4.6.1
OS : RHEL 3 update 6
Lustre : 1.4.6.1
Nodes : 5 (1MDS+2OSS, 2Client)
HCA : silver Storm HCA
We are install rpm for lustre 1.4.6.1 on lustre web site.
kernel-smp-2.4.21-37.EL_lustre.1.4.6.1.i686.rpm
kernel-source-2.4.21-37.EL_lustre.1.4.6.1.i686.rpm
lustre-1.4.6.1-2.4.21_37.EL_lustre.1.4.6.1smp.i686.rpm
lustre-debuginfo-1.4.6.1-2.4.21_37.EL_lustre.1.4.6.1smp.i686.rpm
lustre-modules-1.4.6.1-2.4.21_37.EL_lustre.1.4.6.1smp.i686.rpm
lustre-source-1.4.6.1-2.4.21_37.EL_lustre.1.4.6.1smp.i686.rpm
we tried to setup following as lustre 1.4.6 operation Manual
add options lnet network =''ib0'' at /etc/modules.conf
modprobe lnet
lctl network up <= error occurs.
#dmesg
LustreError: 4858:0:(api-ni.c:985:lnet_startup_lndnis()) Can''t load
LND iib, module kiiblnd, rc=0
LustreError: 4865:0:(api-ni.c:985:lnet_startup_lndnis()) Can''t load
LND iib, module kiiblnd, rc=0
LustreError: 4867:0:(api-ni.c:985:lnet_startup_lndnis()) Can''t load
LND iib, module kiiblnd, rc=0
#ifconfig
ib0 Link encap:Ethernet HWaddr 06:06:6A:00:5B:EF
inet addr:10.0.0.2 Bcast:10.0.0.255 Mask:255.255.255.0
1. If we want to use infiniband, Is it impossible to install rpm-version?
2. Is there anyone who success to install lustre using infiniband,
(If Possible) Please send config file for lustre 1.4.6.1 using infiniband
========================================================Regards
Sung Jun, Kim
Supercomputer Operation Lab. Supercomputing Center
KISTI(Korea Institute of Science & Technology Information)
P.O.BOX 122 Yusong Taejon Korea
EMail : sjkim@kisti.re.kr
Phone: +82-42-869-0636
FAX : +82-42-869-0569
=========================================================