Lustre discuss - Oct 2011 - MDS network traffic question

We have a small lustre install consisting of an MDS and 5 OSS servers.
Historically the MDS and OSS servers had both a 1Gbit ethernet interface
(tcp0) to workstations and a QDR IB interface (ib0) to our cluster.

   We''re planning on adding a MTU 9000 10Gbit ethernet (tcp1) interface
to the MDS and OSS nodes and workstations for faster access.  Our
software has a pretty high IO to CPU component.

   I just discovered that our MDS can''t in fact take another PCIe 8x
card but it does have a spare GigE port.  The 10gbit Ethernet switch
can support 1gbit and 10gbit interfaces.

   We''d then have 3 networks
tcp0 at 1gbit to slow clients
tcp1 at 10gbit to faster clients
ib0 to cluster

   My question is:

   Is there a risk of congestion or overrunning that 2nd GigE MDS 
interface if our workstations and OSS servers communicate over tcp1 at
10gbit but the MDS tcp1 is connected at 1Gbit.  The bulk of our traffic
will continue to be between the cluster and lustre over IB but the
workstations can trivially over run ethernet hence the desire for
10gbit between them and the OSSes.

   My gut feeling is it should be fine, particularly with the larger MTU,
there''s not that much traffic to the MDS but I''d easily
believe it if
somebody said it''s risky thing to do.

   The alternative is to buy a new MDS and swap disks into it.

James Robnett
National Radio Astronomy Observatory
Array Operations Center

I would replace the 1GigE with the 10GigE: have all Ethernet traffic go over the
10GigE links, rather than add another "tcp1" network for Lustre.  This
will keep your configuration much simpler, and make the migration as painless as
possible (just move the IP address to the 10GigE port on the servers).

The MDS traffic _volume_ is much lower than it is for the OSS nodes.  The big
win from 10GigE would be the lower latency: if you approach 100MB/s of MDS
traffic, you have much bigger problems than a 10GigE NIC can solve.

Kevin


On Oct 11, 2011, at 3:54 PM, James Robnett wrote:
> 
>   We have a small lustre install consisting of an MDS and 5 OSS servers.
> Historically the MDS and OSS servers had both a 1Gbit ethernet interface
> (tcp0) to workstations and a QDR IB interface (ib0) to our cluster.
> 
>   We''re planning on adding a MTU 9000 10Gbit ethernet (tcp1)
interface
> to the MDS and OSS nodes and workstations for faster access.  Our
> software has a pretty high IO to CPU component.
> 
>   I just discovered that our MDS can''t in fact take another PCIe
8x
> card but it does have a spare GigE port.  The 10gbit Ethernet switch
> can support 1gbit and 10gbit interfaces.
> 
>   We''d then have 3 networks
> tcp0 at 1gbit to slow clients
> tcp1 at 10gbit to faster clients
> ib0 to cluster
> 
>   My question is:
> 
>   Is there a risk of congestion or overrunning that 2nd GigE MDS 
> interface if our workstations and OSS servers communicate over tcp1 at
> 10gbit but the MDS tcp1 is connected at 1Gbit.  The bulk of our traffic
> will continue to be between the cluster and lustre over IB but the
> workstations can trivially over run ethernet hence the desire for
> 10gbit between them and the OSSes.
> 
>   My gut feeling is it should be fine, particularly with the larger MTU,
> there''s not that much traffic to the MDS but I''d easily
believe it if
> somebody said it''s risky thing to do.
> 
>   The alternative is to buy a new MDS and swap disks into it.
> 
> James Robnett
> National Radio Astronomy Observatory
> Array Operations Center
> 

Confidentiality Notice: This e-mail message, its contents and any attachments to
it are confidential to the intended recipient, and may contain information that
is privileged and/or exempt from disclosure under applicable law. If you are not
the intended recipient, please immediately notify the sender and destroy the
original e-mail message and any attachments (and any copies that may have been
made) from your system or otherwise. Any unauthorized use, copying, disclosure
or distribution of this information is strictly prohibited.