Lustre discuss - Mar 2011 - Can one node mount more than one lustre cluster?

Hi,

What are the constraints on a client mounting more than one lustre
file system?

I realise that a lustre cluster can have more than one file system
configured, but can a client mount different file systems from different
lustre clusters on the same network?;ie.

Assume a Single IB fabric and two Lustre clusters with separate
MGS/MDS/OSS. One lustre is Lincoln and the other is Washington

 >list_nids
192.168.1.10 at o2ib

 >mount -t lustre lincoln:/lustre  /lincoln
 >mount -t lustre washington:/lustre /washinton

Is this doable or should they be on separate IB fabrics



-- 
Brian O''Connor
-----------------------------------------------------------------------
SGI Consulting
Email: briano at sgi.com, Mobile +61 417 746 452
Phone: +61 3 9963 1900, Fax:  +61 3 9963 1902
357 Camberwell Road, Camberwell, Victoria, 3124
AUSTRALIA
http://www.sgi.com/support/services
-----------------------------------------------------------------------

Yes, you can mount both filesystems which are created with separate MDS/OSS on a
single fabric.
e.g. ) assume two filesystems (both filesystem name is
''lustre'') are configured below.
FS1 - MDS/MGS:192.168.1.1 and OSS: 192.168.1.2
FS2 - MDS/MGS:192.168.1.11 and OSS: 192.168.1.12

You should be able to do on the client.
mount -t lustre 192.168.1.1 at o2ib:/lustre /lustre1
mount -t lustre 192.168.1.11 at o2ib:/lustre /lustre2

btw, even you configured two filesystems on same MDS/OSS by different filesystem
name, you can mount both on the client.

Thanks
Ihara

On 3/15/11 3:15 PM, Brian O''Connor wrote:
>
> Hi,
>
> What are the constraints on a client mounting more than one lustre
> file system?
>
> I realise that a lustre cluster can have more than one file system
> configured, but can a client mount different file systems from different
> lustre clusters on the same network?;ie.
>
> Assume a Single IB fabric and two Lustre clusters with separate
> MGS/MDS/OSS. One lustre is Lincoln and the other is Washington
>
>   >list_nids
> 192.168.1.10 at o2ib
>
>   >mount -t lustre lincoln:/lustre  /lincoln
>   >mount -t lustre washington:/lustre /washinton
>
> Is this doable or should they be on separate IB fabrics
>
>
>

I will add to the previous comment of yes you can, that you should
strongly consider naming them differently when they are built.

  Otherwise it''s difficult to differentiate them on clients in
/proc/fs/lustre/llite/{fsname}-{string} where fsname is
''lustre'' by
default.

  If they were both created with the default fsname then you''ll
get two nearly indistinguishable directories.  Been there, done that,
would make a tshirt but nobody would get it.

James Robnett
NRAO/NM

On 15/03/2011 17:15, Brian O''Connor wrote:
> Hi,
>
> What are the constraints on a client mounting more than one lustre
> file system?
>
> I realise that a lustre cluster can have more than one file system
> configured, but can a client mount different file systems from different
> lustre clusters on the same network?;ie.
>
> Assume a Single IB fabric and two Lustre clusters with separate
> MGS/MDS/OSS. One lustre is Lincoln and the other is Washington
>
>   >list_nids
> 192.168.1.10 at o2ib
>
>   >mount -t lustre lincoln:/lustre  /lincoln
>   >mount -t lustre washington:/lustre /washinton
>
> Is this doable or should they be on separate IB fabrics
>
 From my understanding, there should only be one MGS for the entire 
environment (although as many MDTs and OSTs as are required). This is 
because clients will only communicate with exactly one MGS (and will 
communicate with the MGS of the last FS mounted) and will only receive 
updates from the MGS with which it is registered. So, in the above 
example if there is a change to the "lincoln" file system (e.g. a 
failover event, some configuration changes), clients will not receive 
notification.

There''s not an issue with having multiple MGS''s on a site,
only with
mounting lustre file systems from multiple "domains" on the same
client,
IIRC.

Malcolm.

On 2011-03-15, at 4:19 PM, Malcolm Cowe wrote:
> On 15/03/2011 17:15, Brian O''Connor wrote:
>> What are the constraints on a client mounting more than one lustre
>> file system?
>> 
>> I realise that a lustre cluster can have more than one file system
>> configured, but can a client mount different file systems from
different
>> lustre clusters on the same network?;ie.
>> 
>> Assume a Single IB fabric and two Lustre clusters with separate
>> MGS/MDS/OSS. One lustre is Lincoln and the other is Washington
>> 
>>> list_nids
>> 192.168.1.10 at o2ib
>> 
>>> mount -t lustre lincoln:/lustre  /lincoln
>>> mount -t lustre washington:/lustre /washinton
>> 
>> Is this doable or should they be on separate IB fabrics
> 
> From my understanding, there should only be one MGS for the entire 
> environment (although as many MDTs and OSTs as are required). This is 
> because clients will only communicate with exactly one MGS (and will 
> communicate with the MGS of the last FS mounted) and will only receive 
> updates from the MGS with which it is registered. So, in the above 
> example if there is a change to the "lincoln" file system (e.g. a
> failover event, some configuration changes), clients will not receive 
> notification.
> 
> There''s not an issue with having multiple MGS''s on a
site, only with
> mounting lustre file systems from multiple "domains" on the same
client,
> IIRC.
That discussion was had on the list a few months ago, and the correct answer is
that it should just work.  The "only use last mounted MGS" problem was
fixed at some point, though I don''t have the exact version handy.


Cheers, Andreas
--
Andreas Dilger 
Principal Engineer
Whamcloud, Inc.

FYI, I''m using Lustre 1.8.5 to mount two filesystems from separate
domains (one in Lafayette, IN, and one in Bloomington, IN, run by two
different institutions) on 900+ nodes, and things just work.

--
Mike Shuey



On Tue, Mar 15, 2011 at 6:24 PM, Andreas Dilger <adilger at whamcloud.com>
wrote:
> On 2011-03-15, at 4:19 PM, Malcolm Cowe wrote:
>> On 15/03/2011 17:15, Brian O''Connor wrote:
>>> What are the constraints on a client mounting more than one lustre
>>> file system?
>>>
>>> I realise that a lustre cluster can have more than one file system
>>> configured, but can a client mount different file systems from
different
>>> lustre clusters on the same network?;ie.
>>>
>>> Assume a Single IB fabric and two Lustre clusters with separate
>>> MGS/MDS/OSS. One lustre is Lincoln and the other is Washington
>>>
>>>> list_nids
>>> 192.168.1.10 at o2ib
>>>
>>>> mount -t lustre lincoln:/lustre ?/lincoln
>>>> mount -t lustre washington:/lustre /washinton
>>>
>>> Is this doable or should they be on separate IB fabrics
>>
>> From my understanding, there should only be one MGS for the entire
>> environment (although as many MDTs and OSTs as are required). This is
>> because clients will only communicate with exactly one MGS (and will
>> communicate with the MGS of the last FS mounted) and will only receive
>> updates from the MGS with which it is registered. So, in the above
>> example if there is a change to the "lincoln" file system
(e.g. a
>> failover event, some configuration changes), clients will not receive
>> notification.
>>
>> There''s not an issue with having multiple MGS''s on a
site, only with
>> mounting lustre file systems from multiple "domains" on the
same client,
>> IIRC.
>
> That discussion was had on the list a few months ago, and the correct
answer is that it should just work. ?The "only use last mounted MGS"
problem was fixed at some point, though I don''t have the exact version
handy.
>
>
> Cheers, Andreas
> --
> Andreas Dilger
> Principal Engineer
> Whamcloud, Inc.
>
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>

We have a few customers I can think of who do this, have numerous lustre
filesystems each with their own MGS and have clients which mount more than one
of them.

Ashley.