Lustre discuss - Apr 2009 - Infiniband hot spot avoidance with LMC>0

Greetings,

Does Lustre''s o2ib LND take advantage of Infiniband''s LID Mask
Count
(LMC) capability?  Might it be included in the future?  I''m looking for
something similar to the "MV2_USE_HSAM=1" option for Hot-Spot
Avoidance
with MVAPICH2.

For anyone not familiar with LMC, see the "--lmc" option in the
"opensm"
man page.

Thanks,
Nathan

On Mon, Apr 27, 2009 at 12:21:41PM -0600, Nathan Dauchy
wrote:
> Greetings,
> 
> Does Lustre''s o2ib LND take advantage of Infiniband''s LID
Mask Count
> (LMC) capability?  Might it be included in the future?  I''m
looking for
> something similar to the "MV2_USE_HSAM=1" option for Hot-Spot
Avoidance
> with MVAPICH2.
Nothing like this exists so far in the o2iblnd. Currently between any
two ports there''s only one QP which uses the LIDs as returned by the
SM.

The MVAPICH2 seems to be striping outgoing data over multiple paths
and adjusting path weights dynamically based on perceived speed of the
paths. We''d be interested to take a look if there''s a
high-level
description of the mechanism.

Thanks,
Isaac

Isaac Huang wrote:
> On Mon, Apr 27, 2009 at 12:21:41PM -0600, Nathan Dauchy wrote:
>> Greetings,
>>
>> Does Lustre''s o2ib LND take advantage of Infiniband''s
LID Mask Count
>> (LMC) capability?  Might it be included in the future?  I''m
looking for
>> something similar to the "MV2_USE_HSAM=1" option for Hot-Spot
Avoidance
>> with MVAPICH2.
> 
> Nothing like this exists so far in the o2iblnd. Currently between any
> two ports there''s only one QP which uses the LIDs as returned by
the
> SM.
OK, thanks for confirming, Isaac.
> The MVAPICH2 seems to be striping outgoing data over multiple paths
> and adjusting path weights dynamically based on perceived speed of the
> paths. We''d be interested to take a look if there''s a
high-level
> description of the mechanism.
> 
The best high-level document I can find is a presentation: "Hot-Spot
Avoidance with Multi-Pathing over InfiniBand: An MPI Perspective", by
           Abhinav Vishnu, Matthew Koop, et. al.
http://nowlab.cse.ohio-state.edu/publications/conf-presentations/2007/vishnu-ccgrid07.pdf

I imagine the Ohio State guys would be happy to work with you.

And, if Sun''s code licensing plans allow for it, you can always look at
the MVAPICH code. :)

-Nathan