Lustre discuss - Jan 2010 - Preferred network for OST data transfer

I''ve just been tuning a small Lustre installation, with two Ethernet 
physical networks and have noticed that the bulk of the network traffic 
is not going through the intended network.  The two networks are 
labelled tcp0 and tcp1, and tcp1 is in principle the best network for 
OST traffic, because of hardware and other traffic on tcp0.

The system is running 1.8.1.1 on the server and 1.8.0.1 on the clients, 
and Centos 5.3.  It was running 1.6 on the server previously, but the 
upgrade did not seem to have any effect.  Originally - and briefly - 
there was only tcp0.

So modprobe.conf now has tcp1 specified first, and lctl list_nids and 
pings all look good.  The mgsnode parameter in the (for now) single OST 
specifies the tcp1 network.  Then tunefs.lustre --writeconf has been 
run.  Client mounts works fine, using the tcp1 NID.  However, as shown 
by the transfer speed and also by iptraf, the actual file data is being 
transferred over tcp0.

This makes some sort of sense, as I suppose that client to MDS contact 
takes place over the network specified in the mount command, while 
actual data transfer can take place by another route.  On the other hand 
the manual does state that the first network given in the module options 
is preferred.

So I guess there are two questions - the general one of what is 
happening and the specific one of is there another parameter than can be 
set to tell the OSC which network to use?

Thanks in advance,
David Boyes

> I''ve just been tuning a small Lustre installation, with two
> Ethernet physical networks and have noticed that the bulk of
> the network traffic is not going through the intended network.
> The two networks are labelled tcp0 and tcp1, and tcp1 is in
> principle the best network for OST traffic, because of hardware
> and other traffic on tcp0. [ ... ] So I guess there are two
> questions - the general one of what is happening and the
> specific one of is there another parameter than can be set to
> tell the OSC which network to use?
The answer to both comes from considering this trick question:
how are LNET IP packets exempted from kernel IP routing?

As far as I know IP packets are not exempted from normal kernel routing, 
but I may be wrong and would be happy to learn something here.  I should 
have mentioned that the two physical networks are on two separate IPv4 
subnetworks, without routers, and that both of these networks seem to be 
functioning correctly.

But in any case I shut down the Lustre system at a quiet moment, removed 
all mention of the tcp0 network and rebooted several nodes including the 
MDS server.  The mount requests of course remained with tcp1.  Hey 
presto, the traffic was taking the route we wanted, viz. tcp1.

Then, mainly out of curiosity, I restored tcp0 to the MDS and to a 
client node with no effect - that is lctl list_nids looks OK (with both 
NID''s) and traffic keeps going via tcp1.  This is not quite the mirror 
image of the original setup, as mount requests have always been by tcp1, 
but the modprobe.conf parameters on the MDS and on the selected client 
are exactly as at the start.

So we''ve got the traffic going the way we want.  However it does seem 
that something in the system is retaining a preference for a specificc 
NID for OST traffic, and that this is not reflected in the modprobe.conf 
or the documented filesystem parameters.

Best regards,
David Boyes