In 1.6, when servers are started for the first time, they report all of their locally available nids to the MGS. The MGS then creates startup logs for the clients (and MDT) with all the known nids. As Eric said, the mgsnode= parameter used when formatting an OST (or the <mgsnid>:/<fsname> parameter when starting a client) is just used for the initial contact to the MGS; it does not determine which nids are used for normal Lustre communication between nodes. LNET makes decisions about which of the available networks to use based on the local networks available (configured in modprobe.conf) and the full list of nids for each server (delivered in the configuration logs.) To find out the network a particular pair will talk to each other on, try this: # remote_nids=`ssh remote_node lctl list_nids` # lctl which_nid $remote_nids (you''ll have to "lctl network up" first probably) Eric Barton wrote:> Does the file system know the xxx@sc NIDs of its servers? If not, you might > just be using the SC network to get to the MGS, but then it tells the client > only about TCP NIDs. The second possibility is that the MGS _does_ know the > SC NIDs of the servers, but the client is chosing the TCP network in > preference. If so, make sure the SC network appears first in the list of > "networks=..." so that it is chosen in preference. > > Cheers, > Eric > > --------------------------------------------------- > |Eric Barton Barton Software | > |9 York Gardens Tel: +44 (117) 330 1575 | > |Clifton Mobile: +44 (7909) 680 356 | > |Bristol BS8 4LL Fax: call first | > |United Kingdom E-Mail: eeb@bartonsoftware.com| > --------------------------------------------------- > > > >> -----Original Message----- >> From: John R. Dunning [mailto:jrd@sicortex.com] >> Sent: 28 March 2007 6:17 PM >> To: eeb@bartonsoftware.com >> Subject: Yet still more lnd >> >> Ok, so more stuff is starting to work. I''m trying to mount >> an OST. It >> connects to the MGS ok, but after a little while they seem to >> be using tcp as >> well as sc. I formatted the unit with mgsnode=2@sc, but they >> seem to have >> discovered that they also have a tcp path, and are using that >> in preference to >> sc. >> >> Can I set the preference values? Should I just stop bringing >> up the tcp >> network entirely? >> > > >
From: Nathaniel Rutman <nathan@clusterfs.com>
Date: Wed, 28 Mar 2007 11:58:15 -0700
In 1.6, when servers are started for the first time, they report all of
their locally available nids to the MGS. The MGS then creates startup
logs for the clients (and MDT) with all the known nids.
As Eric said, the mgsnode= parameter used when formatting an OST (or the
<mgsnid>:/<fsname> parameter when starting a client) is just
used for
the initial contact to the MGS; it does not determine which nids are
used for normal Lustre communication between nodes.
Ah. Ok.
LNET makes decisions about which of the available networks to use based
on the local networks available (configured in modprobe.conf) and the
full list of nids for each server (delivered in the configuration logs.)
Well, ok, it sounds like for testing, I should just take tcp out of the
networks= clause in modprobe.conf, right? I want to force it to exclusively
use my transport.
To find out the network a particular pair will talk to each other on,
try this:
# remote_nids=`ssh remote_node lctl list_nids`
# lctl which_nid $remote_nids
(you''ll have to "lctl network up" first probably)
Eric Barton wrote:
> Does the file system know the xxx@sc NIDs of its servers? If not, you
might
> just be using the SC network to get to the MGS, but then it tells the
client
> only about TCP NIDs. The second possibility is that the MGS _does_
know the
> SC NIDs of the servers, but the client is chosing the TCP network in
> preference. If so, make sure the SC network appears first in the list
of
> "networks=..." so that it is chosen in preference.
>
> Cheers,
> Eric
>
> ---------------------------------------------------
> |Eric Barton Barton Software |
> |9 York Gardens Tel: +44 (117) 330 1575 |
> |Clifton Mobile: +44 (7909) 680 356 |
> |Bristol BS8 4LL Fax: call first |
> |United Kingdom E-Mail: eeb@bartonsoftware.com|
> ---------------------------------------------------
>
>
>
>> -----Original Message-----
>> From: John R. Dunning [mailto:jrd@sicortex.com]
>> Sent: 28 March 2007 6:17 PM
>> To: eeb@bartonsoftware.com
>> Subject: Yet still more lnd
>>
>> Ok, so more stuff is starting to work. I''m trying to
mount
>> an OST. It
>> connects to the MGS ok, but after a little while they seem to
>> be using tcp as
>> well as sc. I formatted the unit with mgsnode=2@sc, but they
>> seem to have
>> discovered that they also have a tcp path, and are using that
>> in preference to
>> sc.
>>
>> Can I set the preference values? Should I just stop bringing
>> up the tcp
>> network entirely?
>>
>
>
>