Lustre discuss - Aug 2006 - IB Multirail (was LZW block-level compression)

Sean,
> On Tue, 2006-08-01 at 11:15, Peter J. Braam wrote:
> [snip]
>> We are typically targetting servers with 2 Gbytes / sec now.
>> 700 MB on a 10 GIGE is normal, saturating 3-4 Gige is wit say
>> 2 64bit processors, and 2-4 I/B NICs will be used multirail
>> soon.
> [snip]
>
> Sorry to diverge from the topic, but I was under the impression
> that "channel-bonding-like" approaches weren''t planned. 
That
> is, things like multirail IB wouldn''t be supported by Luster.
> Or do you mean some sort of driver-level support similar
> transparent to all applications, including Lustre?  Or Did I
> misunderstand completely?
>
> Thanks,
> Sean
There are several different ways in which multiple interfaces can
be exploited, so it might help to enumerate them.

1. Multiple interfaces for inter-cluster connectivity

   Servers placed on nodes with > 1 type of network can serve to
   clients on all networks.  Alternatively, servers in the
   "interior" of a cluster can be accessed via LNET routers that
   forward traffic between the networks.

   This is fully supported currently.  

2. Multiple interfaces on "fat" servers

   Multiple interfaces on "fat" servers can be assigned to
   separate LNET networks, one per interface.  Clients are each
   assigned statically to one of these networks to spread the
   load over the servers'' interfaces.

   This is supported currently only on TCP/IP networks, but this
   will spread to all network types as we convert the other LNDs
   (lustre network drivers) from single to multi-instance.

   We are also considering generic support for multiple
   interfaces on one node on the same LNET network and
   load-balancing between them so that clients do not have to be
   assigned statically to different LNET networks.

3. Multirail.

   We are planning to support generic LNET network aggregation so
   that nodes that are members of the same aggregated LNET
   network can load-balance communications over all their
   underlying LNET networks.

As you may know, some of these features were implemented
previously in specific LNDs.  However our overall thrust is to
move the "smarts" out of the drivers and up into generic LNET so
that all network types benefit.

Alternatively, you can do some of this below the level of the
LND.  For example, the linux bonding driver does this for TCP
networks.  And the Elan LND fully utilises multirail Elan
clusters because the Quadrics driver below the qswlnd supports
multirail transparently.

-- 

                Cheers,
                        Eric

---------------------------------------------------
|Eric Barton        Barton Software               |
|9 York Gardens     Tel:    +44 (117) 330 1575    |
|Clifton            Mobile: +44 (7909) 680 356    |
|Bristol BS8 4LL    Fax:    call first            |
|United Kingdom     E-Mail: eeb@bartonsoftware.com|
---------------------------------------------------