Is there an issue around mounting multiple lustrefs''s using different lnds? I have a situation in the not too distant future where I''m going to want to be able to mount at least two, one over socklnd, and one over a custom lnd. It occured to me that I don''t know how to configure that, and don''t know if there''s anything in the code which will prevent it. Anybody ever done anything similar? Hints? Thanks in advance...
If a server only has one NID there is no ambiguity. Otherwise, lustre chooses a
NID by asking LNET what the network "distance" to
all the server''s NIDs are, and picks the "best" one as
follows...
1. Compare hop counts. The NID that is reached by traversing fewer routers
wins.
2. Compare rank. Find the local network that you must send on to reach the
given NID and compare the order in which the these
networks appear in the "networks" or "ip2nets" module
parameter. The local network listed first wins.
Cheers,
Eric
---------------------------------------------------
|Eric Barton Barton Software |
|9 York Gardens Tel: +44 (117) 330 1575 |
|Clifton Mobile: +44 (7909) 680 356 |
|Bristol BS8 4LL Fax: call first |
|United Kingdom E-Mail: eeb@bartonsoftware.com|
---------------------------------------------------
> -----Original Message-----
> From: lustre-discuss-bounces@clusterfs.com
> [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of
> John R. Dunning
> Sent: 24 January 2007 1:30 PM
> To: lustre-discuss@clusterfs.com
> Subject: [Lustre-discuss] Multiple LNDs?
>
> Is there an issue around mounting multiple lustrefs''s using
> different lnds?
>
> I have a situation in the not too distant future where I''m
> going to want to be
> able to mount at least two, one over socklnd, and one over a
> custom lnd. It
> occured to me that I don''t know how to configure that, and
> don''t know if
> there''s anything in the code which will prevent it. Anybody ever
done
> anything similar? Hints?
>
> Thanks in advance...
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss@clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
>
Eric, Are you working on supporting multiple instances of the non-TCP LNDs (e.g. multiple o2ib)? If so, let me know so I can work on mxlnd. Thanks, Scott On Jan 24, 2007, at 8:49 AM, Eric Barton wrote:> If a server only has one NID there is no ambiguity. Otherwise, > lustre chooses a NID by asking LNET what the network "distance" to > all the server''s NIDs are, and picks the "best" one as follows... > > 1. Compare hop counts. The NID that is reached by traversing fewer > routers wins. > > 2. Compare rank. Find the local network that you must send on to > reach the given NID and compare the order in which the these > networks appear in the "networks" or "ip2nets" module parameter. > The local network listed first wins. > > Cheers, > Eric > > --------------------------------------------------- > |Eric Barton Barton Software | > |9 York Gardens Tel: +44 (117) 330 1575 | > |Clifton Mobile: +44 (7909) 680 356 | > |Bristol BS8 4LL Fax: call first | > |United Kingdom E-Mail: eeb@bartonsoftware.com| > --------------------------------------------------- > > >> -----Original Message----- >> From: lustre-discuss-bounces@clusterfs.com >> [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of >> John R. Dunning >> Sent: 24 January 2007 1:30 PM >> To: lustre-discuss@clusterfs.com >> Subject: [Lustre-discuss] Multiple LNDs? >> >> Is there an issue around mounting multiple lustrefs''s using >> different lnds? >> >> I have a situation in the not too distant future where I''m >> going to want to be >> able to mount at least two, one over socklnd, and one over a >> custom lnd. It >> occured to me that I don''t know how to configure that, and >> don''t know if >> there''s anything in the code which will prevent it. Anybody ever >> done >> anything similar? Hints? >> >> Thanks in advance... >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss@clusterfs.com >> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >> > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >
From: "Eric Barton" <eeb@bartonsoftware.com>
Date: Wed, 24 Jan 2007 13:49:11 -0000
If a server only has one NID there is no ambiguity.
Ok. So as long as I can arrange for everybody with a presence on the local
lnd to have a different NID than all the external ones, everything should just
work. That should be easy to arrange.
Otherwise, lustre chooses a NID by asking LNET what the network
"distance" to
all the server''s NIDs are, and picks the "best" one as
follows...
1. Compare hop counts. The NID that is reached by traversing fewer routers
wins.
Presumably that means lnet routers, not IP ones or anything else, correct?
2. Compare rank. Find the local network that you must send on to reach the
given NID and compare the order in which the these
networks appear in the "networks" or "ip2nets" module
parameter. The local network listed first wins.
Ah. I guess I didn''t realize that it had that much awareness of
topology, but
I suppose it needs to, doesn''t it? I guess I should go read up on
those
module parms.
At startup time, is it as simple as making sure all the right modules are
loaded, and that the NID name in the mount command gets correctly parsed? So
for instance, if I''ve got some external hosts accessable over socklnd
as
# mount -t lustre my-server:/lustrefs /mnt/external-lustre
and some internal nodes as well, can I just say something like
# mount -t lustre SC!123:/lustrefs /mnt/internal-lustre
as long as my custom lnd will turn "SC!123" into an internal NID?
> Eric, > > Are you working on supporting multiple instances of the non-TCP LNDs > (e.g. multiple o2ib)?Yes, o2ib supports that already.
John,> > Otherwise, lustre chooses a NID by asking LNET what the network "distance" > > to all the server''s NIDs are, and picks the "best" one as follows... > > > > 1. Compare hop counts. The NID that is reached by traversing fewer routers > > wins. > > Presumably that means lnet routers, not IP ones or anything else, correct?Yes.> > 2. Compare rank. Find the local network that you must send on to reach the > > given NID and compare the order in which the these networks appear in the > > "networks" or "ip2nets" module parameter. The local network listed first > > wins. > > Ah. I guess I didn''t realize that it had that much awareness of topology, > but I suppose it needs to, doesn''t it? I guess I should go read up on those > module parms.Please holler if it''s not clear in the documentation.> At startup time, is it as simple as making sure all the right modules are > loaded, and that the NID name in the mount command gets correctly parsed?You can debug connectivity before trying any lustre stuff... 1. Get LNET running on relevent nodes: modprobe lnet && lctl net up 2. lctl ping <NID>> So for instance, if I''ve got some external hosts accessable over socklnd as > > # mount -t lustre my-server:/lustrefs /mnt/external-lustre > > and some internal nodes as well, can I just say something like > > # mount -t lustre SC!123:/lustrefs /mnt/internal-lustre > > as long as my custom lnd will turn "SC!123" into an internal NID?I think "my-server" really means "my-server@tcp0" because the network type of a NID defaults to "tcp" and the network number defaults to "0". If the new network type your LND implements is "sc", then you should be able to say... # mount -t lustre 123@sc:/lustrefs /mnt/internal-lustre Note that all the string/NID conversion stuff is in <lustre>/lnet/libcfs/nidstrings.c Cheers, Eric
From: "Eric Barton" <eeb@bartonsoftware.com>
Date: Wed, 24 Jan 2007 14:44:49 -0000
Please holler if it''s not clear in the documentation.
Wilco :-}
> At startup time, is it as simple as making sure all the right modules
are
> loaded, and that the NID name in the mount command gets correctly
parsed?
You can debug connectivity before trying any lustre stuff...
1. Get LNET running on relevent nodes: modprobe lnet && lctl net up
2. lctl ping <NID>
Yes. I''m not quite at the point of being able to do that, but getting
closer.
I think "my-server" really means "my-server@tcp0"
because the network type
of a NID defaults to "tcp" and the network number defaults to
"0".
Ah, right, I forgot about that.
If the new network type your LND implements is "sc", then you
should be able to
say...
# mount -t lustre 123@sc:/lustrefs /mnt/internal-lustre
Note that all the string/NID conversion stuff is in
<lustre>/lnet/libcfs/nidstrings.c
Yup, I remember that now. We actually went over all that a while back, but
I''d forgotten that tidbit in all the mayhem.
Thanks!