Lustre discuss - Mar 2010 - How to force client-oss communication over IB when the MDS has only ethernet?

Hi,
our MDS is temporarily missing IB connection and has only eth available. 
However, OSS and clients have both IB and eth.

At the moment, it seems that all the traffic between clients-OSS goes 
also through the slow eth connection.  Is it possible to force them to 
use faster IB interfaces when communication with each other, and only 
use eth to communicate with the MDS?

Both clients and OSS have interfaces configured as
options lnet networks=o2ib0(ib0),tcp0(eth0)  (in modprobe.conf) also 
lctl ping works fine over IB.   The documents seem to suggests that the 
first interface is preferred, but apparently it is not when MDS is only 
available through the other.

  -t

>At the moment, it seems that all the traffic between clients-OSS goes 
>also through the slow eth connection.  Is it possible to force them to 
>use faster IB interfaces when communication with each other, and only 
>use eth to communicate with the MDS?
Stupid question time: is it possible that you added the IB interface to
the OSSes _after_ you created the filesystems on the OSSes?

(I know that the MDS remembers the interfaces that are on it at MDS
creation time, and you have to do an extra step to tell it about any
new interfaces; I think the same is true of the OSSes, but I am not
100% sure).

--Ken

>> At the moment, it seems that all the traffic between clients-OSS goes 
>> also through the slow eth connection.  Is it possible to force them to 
>> use faster IB interfaces when communication with each other, and only 
> Stupid question time: is it possible that you added the IB interface to
> the OSSes _after_ you created the filesystems on the OSSes?
> (I know that the MDS remembers the interfaces that are on it at MDS
> creation time, and you have to do an extra step to tell it about any
> new interfaces; I think the same is true of the OSSes, but I am not
> 100% sure).

Yes! I created OSTs and did some testing with plain eth configuration 
first, so this would explain things.  How can I tell the MDS that things 
have changed?

  -Tero

>Yes! I created OSTs and did some testing with plain eth configuration 
>first, so this would explain things.  How can I tell the MDS that things 
>have changed?
You''ll have to run tunefs.lustre --writeconf (the Lustre manual
explains this in a bit more detail).  I had some vague memory that you
only used to have to do it on the MDT, but last time I did it I had to
do it on the MDT and all OSTs.

--Ken

Tero Hakala wrote:
> Hi,
> our MDS is temporarily missing IB connection and has only eth available. 
> However, OSS and clients have both IB and eth.
> 
> At the moment, it seems that all the traffic between clients-OSS goes 
> also through the slow eth connection.  Is it possible to force them to 
> use faster IB interfaces when communication with each other, and only 
> use eth to communicate with the MDS?
> 
> Both clients and OSS have interfaces configured as
> options lnet networks=o2ib0(ib0),tcp0(eth0)  (in modprobe.conf) also 
> lctl ping works fine over IB.   The documents seem to suggests that the 
> first interface is preferred, but apparently it is not when MDS is only 
> available through the other.
> 
I believe in this case since you are using the tcp NID for the client 
mount, lnet assumes that you are using tcp0, since tcp0 can reach the 
OSS. I think you might be able to use options lnet ip2nets on the 
clients to force the client-OSS connection to use ib0.

cliffw
>   -t
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss