Brian O''Connor
2011-Mar-15 06:15 UTC
[Lustre-discuss] Can one node mount more than one lustre cluster?
Hi, What are the constraints on a client mounting more than one lustre file system? I realise that a lustre cluster can have more than one file system configured, but can a client mount different file systems from different lustre clusters on the same network?;ie. Assume a Single IB fabric and two Lustre clusters with separate MGS/MDS/OSS. One lustre is Lincoln and the other is Washington >list_nids 192.168.1.10 at o2ib >mount -t lustre lincoln:/lustre /lincoln >mount -t lustre washington:/lustre /washinton Is this doable or should they be on separate IB fabrics -- Brian O''Connor ----------------------------------------------------------------------- SGI Consulting Email: briano at sgi.com, Mobile +61 417 746 452 Phone: +61 3 9963 1900, Fax: +61 3 9963 1902 357 Camberwell Road, Camberwell, Victoria, 3124 AUSTRALIA http://www.sgi.com/support/services -----------------------------------------------------------------------
Shuichi Ihara
2011-Mar-15 06:37 UTC
[Lustre-discuss] Can one node mount more than one lustre cluster?
Yes, you can mount both filesystems which are created with separate MDS/OSS on a single fabric. e.g. ) assume two filesystems (both filesystem name is ''lustre'') are configured below. FS1 - MDS/MGS:192.168.1.1 and OSS: 192.168.1.2 FS2 - MDS/MGS:192.168.1.11 and OSS: 192.168.1.12 You should be able to do on the client. mount -t lustre 192.168.1.1 at o2ib:/lustre /lustre1 mount -t lustre 192.168.1.11 at o2ib:/lustre /lustre2 btw, even you configured two filesystems on same MDS/OSS by different filesystem name, you can mount both on the client. Thanks Ihara On 3/15/11 3:15 PM, Brian O''Connor wrote:> > Hi, > > What are the constraints on a client mounting more than one lustre > file system? > > I realise that a lustre cluster can have more than one file system > configured, but can a client mount different file systems from different > lustre clusters on the same network?;ie. > > Assume a Single IB fabric and two Lustre clusters with separate > MGS/MDS/OSS. One lustre is Lincoln and the other is Washington > > >list_nids > 192.168.1.10 at o2ib > > >mount -t lustre lincoln:/lustre /lincoln > >mount -t lustre washington:/lustre /washinton > > Is this doable or should they be on separate IB fabrics > > >
James Robnett
2011-Mar-15 13:10 UTC
[Lustre-discuss] Can one node mount more than one lustre cluster?
I will add to the previous comment of yes you can, that you should strongly consider naming them differently when they are built. Otherwise it''s difficult to differentiate them on clients in /proc/fs/lustre/llite/{fsname}-{string} where fsname is ''lustre'' by default. If they were both created with the default fsname then you''ll get two nearly indistinguishable directories. Been there, done that, would make a tshirt but nobody would get it. James Robnett NRAO/NM
Malcolm Cowe
2011-Mar-15 22:19 UTC
[Lustre-discuss] Can one node mount more than one lustre cluster?
On 15/03/2011 17:15, Brian O''Connor wrote:> Hi, > > What are the constraints on a client mounting more than one lustre > file system? > > I realise that a lustre cluster can have more than one file system > configured, but can a client mount different file systems from different > lustre clusters on the same network?;ie. > > Assume a Single IB fabric and two Lustre clusters with separate > MGS/MDS/OSS. One lustre is Lincoln and the other is Washington > > >list_nids > 192.168.1.10 at o2ib > > >mount -t lustre lincoln:/lustre /lincoln > >mount -t lustre washington:/lustre /washinton > > Is this doable or should they be on separate IB fabrics >From my understanding, there should only be one MGS for the entire environment (although as many MDTs and OSTs as are required). This is because clients will only communicate with exactly one MGS (and will communicate with the MGS of the last FS mounted) and will only receive updates from the MGS with which it is registered. So, in the above example if there is a change to the "lincoln" file system (e.g. a failover event, some configuration changes), clients will not receive notification. There''s not an issue with having multiple MGS''s on a site, only with mounting lustre file systems from multiple "domains" on the same client, IIRC. Malcolm.
Andreas Dilger
2011-Mar-15 22:24 UTC
[Lustre-discuss] Can one node mount more than one lustre cluster?
On 2011-03-15, at 4:19 PM, Malcolm Cowe wrote:> On 15/03/2011 17:15, Brian O''Connor wrote: >> What are the constraints on a client mounting more than one lustre >> file system? >> >> I realise that a lustre cluster can have more than one file system >> configured, but can a client mount different file systems from different >> lustre clusters on the same network?;ie. >> >> Assume a Single IB fabric and two Lustre clusters with separate >> MGS/MDS/OSS. One lustre is Lincoln and the other is Washington >> >>> list_nids >> 192.168.1.10 at o2ib >> >>> mount -t lustre lincoln:/lustre /lincoln >>> mount -t lustre washington:/lustre /washinton >> >> Is this doable or should they be on separate IB fabrics > > From my understanding, there should only be one MGS for the entire > environment (although as many MDTs and OSTs as are required). This is > because clients will only communicate with exactly one MGS (and will > communicate with the MGS of the last FS mounted) and will only receive > updates from the MGS with which it is registered. So, in the above > example if there is a change to the "lincoln" file system (e.g. a > failover event, some configuration changes), clients will not receive > notification. > > There''s not an issue with having multiple MGS''s on a site, only with > mounting lustre file systems from multiple "domains" on the same client, > IIRC.That discussion was had on the list a few months ago, and the correct answer is that it should just work. The "only use last mounted MGS" problem was fixed at some point, though I don''t have the exact version handy. Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc.
Michael Shuey
2011-Mar-15 22:29 UTC
[Lustre-discuss] Can one node mount more than one lustre cluster?
FYI, I''m using Lustre 1.8.5 to mount two filesystems from separate domains (one in Lafayette, IN, and one in Bloomington, IN, run by two different institutions) on 900+ nodes, and things just work. -- Mike Shuey On Tue, Mar 15, 2011 at 6:24 PM, Andreas Dilger <adilger at whamcloud.com> wrote:> On 2011-03-15, at 4:19 PM, Malcolm Cowe wrote: >> On 15/03/2011 17:15, Brian O''Connor wrote: >>> What are the constraints on a client mounting more than one lustre >>> file system? >>> >>> I realise that a lustre cluster can have more than one file system >>> configured, but can a client mount different file systems from different >>> lustre clusters on the same network?;ie. >>> >>> Assume a Single IB fabric and two Lustre clusters with separate >>> MGS/MDS/OSS. One lustre is Lincoln and the other is Washington >>> >>>> list_nids >>> 192.168.1.10 at o2ib >>> >>>> mount -t lustre lincoln:/lustre ?/lincoln >>>> mount -t lustre washington:/lustre /washinton >>> >>> Is this doable or should they be on separate IB fabrics >> >> From my understanding, there should only be one MGS for the entire >> environment (although as many MDTs and OSTs as are required). This is >> because clients will only communicate with exactly one MGS (and will >> communicate with the MGS of the last FS mounted) and will only receive >> updates from the MGS with which it is registered. So, in the above >> example if there is a change to the "lincoln" file system (e.g. a >> failover event, some configuration changes), clients will not receive >> notification. >> >> There''s not an issue with having multiple MGS''s on a site, only with >> mounting lustre file systems from multiple "domains" on the same client, >> IIRC. > > That discussion was had on the list a few months ago, and the correct answer is that it should just work. ?The "only use last mounted MGS" problem was fixed at some point, though I don''t have the exact version handy. > > > Cheers, Andreas > -- > Andreas Dilger > Principal Engineer > Whamcloud, Inc. > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >