Lustre discuss - Feb 2008 - Lustre behaviour when multiple network paths are available?

Hi there,

When Lustre is configured in an environment where there are multiple paths
to the same destination of the same length (i.e. two paths, each one hop
away), which path(s) will be used for sending and receiving data?

I have my cluster configured with two OSTs with two GigE NICs in each. I am
seeing identical performance metrics when I use LACP to aggregate, and when
I use two separate network addresses to connect them (ditto on the client
side).

So what I''m wondering is if I''ve hit the peak performance of
my disk array,
or if Lustre is just using only one path. The numbers I''m seeing in
both
scenarios indicate 95% utilization of GigE, times two.

How can I get Lustre to use both paths simultaneously?

thanks,
Klaus

On Feb 07, 2008  15:05 -0800, Klaus Steden wrote:
> When Lustre is configured in an environment where there are multiple paths
> to the same destination of the same length (i.e. two paths, each one hop
> away), which path(s) will be used for sending and receiving data?
That depends on how you configure it in /etc/modprobe.conf.
> I have my cluster configured with two OSTs with two GigE NICs in each. I am
> seeing identical performance metrics when I use LACP to aggregate, and when
> I use two separate network addresses to connect them (ditto on the client
> side).
> 
> So what I''m wondering is if I''ve hit the peak performance
of my disk array,
> or if Lustre is just using only one path. The numbers I''m seeing
in both
> scenarios indicate 95% utilization of GigE, times two.
I''m not sure I understand - if you are getting aggregate performance
that
is 190% of a single GigE from the client then you _have_ to be using both
paths (assuming there are two GigE NICs in the client, and not four).
> How can I get Lustre to use both paths simultaneously?
ifconfig should show you clearly via TX/RX byte counts which NICs are
being used in each configuration.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Hello Andreas,

I figured out what the issue was ...

My lnet configuration looked like this:

-- cut --
options lnet networks="tcp0(eth0),tcp1(bond0)"
-- cut --

So all of the traffic in the cluster was being routed over tcp0 using the
single link; I reconfigured my MGS parameters and changed my lnet config to
use ''tcp0(bond0)'' and now I''m seeing the results I
expected.

I get about 350 MB/s from two OSSes -- each with a 2 pair LACP link (4 GigE
total, so about 75% utilization).

cheers,
Klaus

On 2/7/08 11:16 PM, "Andreas Dilger" <adilger at sun.com>did
etch on stone
tablets:
> On Feb 07, 2008  15:05 -0800, Klaus Steden wrote:
>> When Lustre is configured in an environment where there are multiple
paths
>> to the same destination of the same length (i.e. two paths, each one
hop
>> away), which path(s) will be used for sending and receiving data?
> 
> That depends on how you configure it in /etc/modprobe.conf.
> 
>> I have my cluster configured with two OSTs with two GigE NICs in each.
I am
>> seeing identical performance metrics when I use LACP to aggregate, and
when
>> I use two separate network addresses to connect them (ditto on the
client
>> side).
>> 
>> So what I''m wondering is if I''ve hit the peak
performance of my disk array,
>> or if Lustre is just using only one path. The numbers I''m
seeing in both
>> scenarios indicate 95% utilization of GigE, times two.
> 
> I''m not sure I understand - if you are getting aggregate
performance that
> is 190% of a single GigE from the client then you _have_ to be using both
> paths (assuming there are two GigE NICs in the client, and not four).
> 
>> How can I get Lustre to use both paths simultaneously?
> 
> ifconfig should show you clearly via TX/RX byte counts which NICs are
> being used in each configuration.
> 
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>