Lustre discuss - Aug 2008 - multiple NLI's/interfaces to listen on?

Hi All,

Got this little problem wich is driving me nuts.

I have a small network with a combined Lustre MDT/MGS/OST server and 3 
clients

I am trying to let the server listen on 3 different NIC''s in a seperate
network.

e.g.:

eth1: 10.1.34.50
eth2: 10.1.35.50
eth3: 10.1.36.50

I want to connect each client directly to one of the interfaces but when 
trying that only eth1 works.

from dmesg:
Lustre: Added LNI 10.1.34.51 at tcp [8/256]
Lustre: Accept secure, port 988

it appears the server only is listening on eth1 and not the other NIC''s
Is there a possibility to let the other NIC''s also listen so i can use
them?

Kind regards
Robert H

-- 

Met vriendelijke groet,

Robert Hassing
Coordinator ICT Infrastructuur 

Factotum B.V.

Pr. Bernhardplantsoen 206, 7551HT Hengelo
Telefoon: 074-2554083
E-mail: rhassing at factotum.nl
www.factotum.nl

Hi Robert,

You''d need to adjust the lnet options line in /etc/modprobe.conf to
force
Lustre to bind to all your NICs, I believe it binds to the first one
available unless instructed otherwise.

Try this:

-- cut --
options lnet networks="tcp0(eth1),tcp1(eth2),tcp3(eth3)"
-- cut --

which will serve up the Lustre volume on each of your interfaces via three
different LNET networks. You may be able to organize them all under a single
LNET network, but without the right routing in place, it gets very
complicated.

hth,
Klaus

On 8/25/08 5:23 AM, "Robert Hassing" <rhassing at
factotum.nl>did etch on stone
tablets:
> Hi All,
> 
> Got this little problem wich is driving me nuts.
> 
> I have a small network with a combined Lustre MDT/MGS/OST server and 3
> clients
> 
> I am trying to let the server listen on 3 different NIC''s in a
seperate
> network.
> 
> e.g.:
> 
> eth1: 10.1.34.50
> eth2: 10.1.35.50
> eth3: 10.1.36.50
> 
> I want to connect each client directly to one of the interfaces but when
> trying that only eth1 works.
> 
> from dmesg:
> Lustre: Added LNI 10.1.34.51 at tcp [8/256]
> Lustre: Accept secure, port 988
> 
> it appears the server only is listening on eth1 and not the other
NIC''s
> Is there a possibility to let the other NIC''s also listen so i can
use them?
> 
> Kind regards
> Robert H

Klaus,

This is exactly the solution where i was looking for!
Everything runs at a glance wright now!

Thanks
Robert

-----Original Message-----
From: Klaus Steden [mailto:klaus.steden at thomson.net] 
Sent: maandag 25 augustus 2008 21:48
To: rhassing at factotum.nl; lustre-discuss at lists.lustre.org
Subject: Re: [Lustre-discuss] multiple NLI''s/interfaces to listen on?


Hi Robert,

You''d need to adjust the lnet options line in /etc/modprobe.conf to
force
Lustre to bind to all your NICs, I believe it binds to the first one
available unless instructed otherwise.

Try this:

-- cut --
options lnet networks="tcp0(eth1),tcp1(eth2),tcp3(eth3)"
-- cut --

which will serve up the Lustre volume on each of your interfaces via three
different LNET networks. You may be able to organize them all under a single
LNET network, but without the right routing in place, it gets very
complicated.

hth,
Klaus

On 8/25/08 5:23 AM, "Robert Hassing" <rhassing at
factotum.nl>did etch on stone
tablets:
> Hi All,
> 
> Got this little problem wich is driving me nuts.
> 
> I have a small network with a combined Lustre MDT/MGS/OST server and 3
> clients
> 
> I am trying to let the server listen on 3 different NIC''s in a
seperate
> network.
> 
> e.g.:
> 
> eth1: 10.1.34.50
> eth2: 10.1.35.50
> eth3: 10.1.36.50
> 
> I want to connect each client directly to one of the interfaces but when
> trying that only eth1 works.
> 
> from dmesg:
> Lustre: Added LNI 10.1.34.51 at tcp [8/256]
> Lustre: Accept secure, port 988
> 
> it appears the server only is listening on eth1 and not the other
NIC''s
> Is there a possibility to let the other NIC''s also listen so i can
use
them?
> 
> Kind regards
> Robert H

I have exactly the same problem. 
[murray at nasmds ~]$ cat /etc/modprobe.conf
alias scsi_hostadapter aacraid
alias scsi_hostadapter1 sata_nv
alias ib0 ib_ipoib
alias ib1 ib_ipoib
alias net-pf-27 ib_sdp

options lnet networks=tcp0(eth0),tcp1(eth1),o2ib0(ib0)
alias eth0 forcedeth
alias eth1 forcedeth

[murray at nasmds ~]$ uname -a
Linux nasmds 2.6.18-53.1.14.el5_lustre.1.6.5.1smp #1 SMP Wed Jun 18 
19:45:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

I can mount via eth0, and ib0, but mounts via eth1 fail.
If a client machine has both eth0 and eth1 networks enabled, then the 
mount via eth1 "seems" work,
but if I ifdown eth0, then the eth1 mount won''t work.

thanks,
murray smigel

On Tue, Aug 26, 2008 at 10:15:28AM -0400, murray smigel
wrote:
> I have exactly the same problem. 
> [murray at nasmds ~]$ cat /etc/modprobe.conf
> alias scsi_hostadapter aacraid
> alias scsi_hostadapter1 sata_nv
> alias ib0 ib_ipoib
> alias ib1 ib_ipoib
> alias net-pf-27 ib_sdp
> 
> options lnet networks=tcp0(eth0),tcp1(eth1),o2ib0(ib0)
> alias eth0 forcedeth
> alias eth1 forcedeth
> 
> [murray at nasmds ~]$ uname -a
> Linux nasmds 2.6.18-53.1.14.el5_lustre.1.6.5.1smp #1 SMP Wed Jun 18 
> 19:45:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
> 
> I can mount via eth0, and ib0, but mounts via eth1 fail.
> If a client machine has both eth0 and eth1 networks enabled, then the 
> mount via eth1 "seems" work,
> but if I ifdown eth0, then the eth1 mount won''t work.
I''m having similiar trouble with eth and IB.. I can

''mount -t lustre server at o2ib:/lustre /lustre'' , and it
works
but packets go over the ethernet interface, and performance is around
100Mb/sec. (gig-E speeds)

FYI, I''m also running 2.6.18, from Debian. What OFED stack are you
running BTW?

Troy Benjegerdes wrote:
> On Tue, Aug 26, 2008 at 10:15:28AM -0400, murray smigel wrote:
>   
>> I have exactly the same problem. 
>> [murray at nasmds ~]$ cat /etc/modprobe.conf
>> alias scsi_hostadapter aacraid
>> alias scsi_hostadapter1 sata_nv
>> alias ib0 ib_ipoib
>> alias ib1 ib_ipoib
>> alias net-pf-27 ib_sdp
>>
>> options lnet networks=tcp0(eth0),tcp1(eth1),o2ib0(ib0)
>> alias eth0 forcedeth
>> alias eth1 forcedeth
>>
>> [murray at nasmds ~]$ uname -a
>> Linux nasmds 2.6.18-53.1.14.el5_lustre.1.6.5.1smp #1 SMP Wed Jun 18 
>> 19:45:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
>>
>> I can mount via eth0, and ib0, but mounts via eth1 fail.
>> If a client machine has both eth0 and eth1 networks enabled, then the 
>> mount via eth1 "seems" work,
>> but if I ifdown eth0, then the eth1 mount won''t work.
>>     
>
> I''m having similiar trouble with eth and IB.. I can
>
> ''mount -t lustre server at o2ib:/lustre /lustre'' , and it
works
> but packets go over the ethernet interface, and performance is around
> 100Mb/sec. (gig-E speeds)
>
> FYI, I''m also running 2.6.18, from Debian. What OFED stack are you
> running BTW?
>   
**
OFED-1.3
I have seen the failure to use o2ib in the past as well. Try changing 
the order
of the lnet networks list so as to put the o2ib interface first. If that 
does not work,
try with only the o2ib interface defined.
murray smigel


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080826/e87ca469/attachment.html