Lustre discuss - Jul 2006 - Can Lustre route traffic to different NICs based on target OST or filesystem?

Hello,

I have three different lustre filesystems mounted on a Lustre client
that has three NICs. Is there a way to get lustre to use a different
NIC when accessing different filesystems? I am using 1.5.91

Here is how I have each of the three filesystems mounted on the client:
<snip>
172.19.64.14@tcp3:/fs0 on /fs0 type lustre (rw)
172.16.64.1@tcp:/fs1 on /fs1 type lustre (rw)
172.17.64.15@tcp1:/fs2 on /fs2 type lustre (rw)
</snip>

Here is what I have in /etc/modprobe.conf:
<snip>
options lnet ip2nets="tcp1(eth1) 172.17.*.*; tcp0(eth0) 172.16.*.*;
tcp3(eth3) 172.19.*.*"
</snip>

If I change the order of the ip2nets in /etc/modprobe.conf, then
Lustre will exclusively use the first network that is listed. I can''t
seem to route Lustre traffic through a specific NIC when accessing a
specific Lustre filesystem.

And here is what ifconfig reports:
<snip>
eth0      Link encap:Ethernet  HWaddr 00:E0:81:43:F3:72
          inet addr:172.16.64.18  Bcast:172.16.255.255  Mask:255.255.0.0
          inet6 addr: fe80::2e0:81ff:fe43:f372/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:104237 errors:0 dropped:0 overruns:0 frame:0
          TX packets:29471 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:111414252 (106.2 MiB)  TX bytes:3375591 (3.2 MiB)
          Interrupt:153

eth1      Link encap:Ethernet  HWaddr 00:E0:81:43:F3:73
          inet addr:172.17.64.18  Bcast:172.17.255.255  Mask:255.255.0.0
          inet6 addr: fe80::2e0:81ff:fe43:f373/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3654523 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3916944 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3411944276 (3.1 GiB)  TX bytes:4386498616 (4.0 GiB)
          Interrupt:161

eth3      Link encap:Ethernet  HWaddr 00:04:23:CE:75:AB
          inet addr:172.19.64.18  Bcast:172.19.255.255  Mask:255.255.0.0
          inet6 addr: fe80::204:23ff:fece:75ab/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:9006 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10111 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1461830 (1.3 MiB)  TX bytes:1510962 (1.4 MiB)
          Base address:0x8c00 Memory:fc8e0000-fc900000
</snip>

The problem is that all of the Lustre traffic for all three
filesystems is being routed through eth1, even though each of the
filesystems is mounted on a different ip2nets network.

My kernel routing table seems to have no effect on Lustre routing, either:
<snip>
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.19.0.0      *               255.255.0.0     U     0      0        0 eth3
172.16.0.0      *               255.255.0.0     U     0      0        0 eth0
172.17.0.0      *               255.255.0.0     U     0      0        0 eth1
</snip>

How can I have Lustre route traffic to different NICs based on which
Lustre filesystem is being accessed?

Thanks in advance!

Lustre folks,

        I''ve been doing some testing of exporting Lustre over CIFS
    with Samba.  It seems to be working fine but I wonder if
    the test coverage is sufficient.  I do not know of any 
    standardized CIFS validation suite. Does anyone know of
    a validation suite ? What sort of testing does CFS do to
    validate that things are working correctly ? Are there any
    known limitations that I should be aware of ? Example:
    CIFS has a rich set of locking primitives, do they work
    on top of Lustre ? Is there a suite that one can use to 
    validate that they are working ?

Enjoy,
Don Capps