Lustre discuss - Nov 2009 - Dual NICs issue -- How to enforce Lustre to use the second NIC

Hello list,

By searching the archive, I found a similar message dated back in January
2008 -- How do you make an MGS/OSS listen on 2 NICs? Looks like there is no
final solution and I am facing the similar situation and need your help.

I am running centos 5 on both server (MGS, MDS and OSS are on same node) and
clients: 2.6.18-128.1.6.el5_lustre.1.8.0.1smp. To simplify the issue,
suppose the network is consist of one lustre server node and two lustre
client nodes. The server node has two NICs: eth0(100Mb) and eth1(1Gb), each
client node only has one NIC:eth0. The network layout is as below.

Server node eth0: 72.203.10.1 (Public network)    <==> Switch1 <==>
Public
node eth0:  72.203.10.2 (Public network)
Server node eth1: 192.168.10.1 (Internal network) <==> Switch2 <==>
Private
node eth0: 192.168.10.2 (Internal network)

Both SELinux and Fireware are turned off. Public node does not know Private
node, but Private node do knows Public node.

The modprobe.conf likes the following:

On server: options lnet networks="tcp0(eth0),tcp1(eth1)"
On clients: options lnet networks=tcp  <--- since there is only one NIC, I
did not specify it as tcp(eth0).

The procedure to build MGS, MDT and OST on server node as bellow:

[root at server ~]# mkfs.lustre --mgs /dev/sdb1
[root at server ~]# mkdir /mgs
[root at server ~]# mount -t lustre /dev/sdb1 /mgs
[root at server ~]# mkfs.lustre --mdt --fsname=data --mgsnode="72.203.10.1
at tcp0
,192.168.10.1 at tcp1" /dev/sdb2
[root at server ~]# mkdir /data
[root at server ~]# mount -t lustre /dev/sdb2 /data
[root at server ~]# mkfs.lustre --ost --fsname=data --mgsnode="72.203.10.1
at tcp0
,192.168.10.1 at tcp1" /dev/sdc1
[root at server ~]# mkdir /mnt/data
[root at server ~]# mount -t lustre /dev/sdc1 /mnt/data

On server node:
[root at server ~]# lctl list_nids
72.203.10.1 at tcp
192.168.10.1 at tcp1

[root at server ~]# lctl ping 72.203.10.1 at tcp
12345-0 at lo
12345-72.203.10.1 at tcp
12345-192.168.10.1 at tcp1

[root at server ~]# lctl ping 192.168.10.1 at tcp1
12345-0 at lo
12345-72.203.10.1 at tcp
12345-192.168.10.1 at tcp1

Therefore, every thing on server node looks good, and I can mount /data on
public node:
[root at public ~]# mount -t lustre 72.203.10.1 at tcp:/data /data

Also I can mount /data on private node using server''s eth0 (public NIC)
[root at private ~]# mount -t lustre 72.203.10.1 at tcp:/data /data

But I can not mount /data on private node using server''s eth1 (private
NIC).

[root at private ~]# mount -t lustre 192.168.10.1 at tcp1:/data /data
mount.lustre: /data inaccessible: No such file or directory

On private node:
[root at private ~]# lctl which_nid 192.168.10.1 at tcp1
No reachable NID

[root at private ~]# lctl which_nid 192.168.10.1 at tcp
192.168.10.1 at tcp

[root at private ~]# lctl which_nid 72.203.10.1 at tcp
72.203.10.1 at tcp

[root at private ~]# lctl ping 72.203.10.1 at tcp
12345-0 at lo
12345-72.203.10.1 at tcp
12345-192.168.10.1 at tcp1

We see the NID of 192.168.10.1 at tcp from the private node, but can not ping
it:
[root at private ~]# lctl ping 192.168.10.1 at tcp1
failed to ping 192.168.10.1 at tcp1: Input/output error

[root at private ~]# lctl ping 192.168.10.1 at tcp
failed to ping 192.168.10.1 at tcp: Input/output error

The error message in /var/log/messages are
LustreError: 27435:0:(lib-move.c:1265:lnet_send()) No route to
12345-192.168.10.1 at tcp1
LustreError: 27435:0:(lib-move.c:2450:LNetGet()) error sending GET to
12345-192.168.10.1 at tcp1: -113

But I can ping 192.168.10.1 using regular ping.

According to the response of Marc:
"As far as I know, LNET will use the shortest path on the network, so if
you
have two equivalent tcp networks, tcp0 and tcp1, LNET will just  use the
first one.  If it fails, it should use the second one. If both NICs are in
the same tcp network, LNET should use both.....".

Therefore, I can understand why I can mount /data on private node using
server''s eth0. But looks like LNET is not smart enough to pick up the
"closer" NIC to mount the /data.

My questions are:
1. Is something wrong of the eth1configuration?
2. How to enforce LNET to use the second NIC(eth1) to mount /data on private
node? I prefer to use eth1 not eth0 because eth1 connects to switch2
directly and fast (eth1-1Gb) than eth0 (100Mb).

Thanks for reading the detailed procedure and appreciate for your help in
advance,

Goodman
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091111/c1c68d79/attachment.html

On Wed, Nov 11, 2009 at 04:07:39PM -0600, Daneil Goodman
wrote:
>    Hello list,
>    By searching the archive, I found a similar message dated back in
>    January 2008 -- How do you make an MGS/OSS listen on 2 NICs? Looks like
>    there is no final solution and I am facing the similar situation and
>    need your help.
>    I am running centos 5 on both server (MGS, MDS and OSS are on same
>    node) and clients: 2.6.18-128.1.6.el5_lustre.1.8.0.1smp. To simplify
>    the issue, suppose the network is consist of one lustre server node and
>    two lustre client nodes. The server node has two NICs: eth0(100Mb) and
>    eth1(1Gb), each client node only has one NIC:eth0. The network layout
>    is as below.
>    Server node eth0: 72.203.10.1 (Public network)    <==> Switch1
<==>
>    Public node eth0:  72.203.10.2 (Public network)
>    Server node eth1: 192.168.10.1 (Internal network) <==> Switch2
<==>
>    Private node eth0: 192.168.10.2 (Internal network)
>    Both SELinux and Fireware are turned off. Public node does not know
>    Private node, but Private node do knows Public node.
>    The modprobe.conf likes the following:
>    On server: options lnet networks="tcp0(eth0),tcp1(eth1)"
>    On clients: options lnet networks=tcp  <--- since there is only one
I think you''d need to make clients in the 72.203.10.* network use tcp0
and clients in the 192.168.10.* tcp1. To create a uniform module
option that works across the whole cluster, ''ip2nets'' is your
friend:

options lnet ''ip2nets="tcp0(eth0) 72.203.10.*; tcp1(eth1)
192.168.10.[1-10]; tcp1(eth0) 192.168.10.[100-200]"

(assuming that servers are 192.168.10.[1-10] and clients are
192.168.10.[100-200].)

Isaac

On Wed, Nov 11, 2009 at 9:20 PM, Isaac Huang <He.Huang at sun.com> wrote:
> On Wed, Nov 11, 2009 at 04:07:39PM -0600, Daneil Goodman wrote:
> >    Hello list,
> >    By searching the archive, I found a similar message dated back in
> >    January 2008 -- How do you make an MGS/OSS listen on 2 NICs? Looks
> like
> >    there is no final solution and I am facing the similar situation
and
> >    need your help.
> >    I am running centos 5 on both server (MGS, MDS and OSS are on same
> >    node) and clients: 2.6.18-128.1.6.el5_lustre.1.8.0.1smp. To
simplify
> >    the issue, suppose the network is consist of one lustre server node
> and
> >    two lustre client nodes. The server node has two NICs: eth0(100Mb)
and
> >    eth1(1Gb), each client node only has one NIC:eth0. The network
layout
> >    is as below.
> >    Server node eth0: 72.203.10.1 (Public network)    <==>
Switch1 <==>
> >    Public node eth0:  72.203.10.2 (Public network)
> >    Server node eth1: 192.168.10.1 (Internal network) <==>
Switch2 <==>
> >    Private node eth0: 192.168.10.2 (Internal network)
> >    Both SELinux and Fireware are turned off. Public node does not know
> >    Private node, but Private node do knows Public node.
> >    The modprobe.conf likes the following:
> >    On server: options lnet networks="tcp0(eth0),tcp1(eth1)"
> >    On clients: options lnet networks=tcp  <--- since there is only
one
>
> I think you''d need to make clients in the 72.203.10.* network use
tcp0
> and clients in the 192.168.10.* tcp1. To create a uniform module
>
It did the trick!  Thanks!

option that works across the whole cluster, ''ip2nets'' is your
friend:
>
> options lnet ''ip2nets="tcp0(eth0) 72.203.10.*; tcp1(eth1)
> 192.168.10.[1-10]; tcp1(eth0) 192.168.10.[100-200]"
>
> (assuming that servers are 192.168.10.[1-10] and clients are
> 192.168.10.[100-200].)
>
>
There are three small issues about ip2nets:

1. Looks like LNET does not like the single quotes
''ip2nets="tcp0(eth0)
72.203.10.*; tcp1(eth1) 192.168.10.*"''. It says

lnet: Unknown parameter `''ip2nets''

After removed single quotes, I can load lnet module.

2. According to my observation, on public network, to mount /data using
above ip2nets option is slower than networks option.

3. On private network node, I cannot start LNET using ip2nets option
[root at private ~]# lsmod |grep lnet
lnet                  273084  1 ksocklnd
libcfs                136180  2 ksocklnd,lnet
[root at private ~]# lctl network configure
LNET configure error 100: Network is down

/var/log/messages shows:
LustreError: 31927:0:(socklnd.c:2545:ksocknal_startup()) Interface eth1 is
down
LustreError: 105-4: Error -100 starting up LNI tcp

But if you use networks option (options lnet networks=tcp1), it works well.
Do you think what is the problem?

Thanks,
Goodman
> Isaacc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091112/07529e77/attachment.html

>
> 3. On private network node, I cannot start LNET using ip2nets option
> [root at private ~]# lsmod |grep lnet
> lnet                  273084  1 ksocklnd
> libcfs                136180  2 ksocklnd,lnet
> [root at private ~]# lctl network configure
> LNET configure error 100: Network is down
>
> /var/log/messages shows:
> LustreError: 31927:0:(socklnd.c:2545:ksocknal_startup()) Interface eth1 is
> down
> LustreError: 105-4: Error -100 starting up LNI tcp
>
> But if you use networks option (options lnet networks=tcp1), it works well.
> Do you think what is the problem?
>
I fixed this problem by changing the order of the interfaces in ip2nets
line.

>
> Thanks,
> Goodman
>
>> Isaacc
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091112/95f349e3/attachment.html