Can a lustre file system have more than one MGS? Isn''t it only one per site? I saw some examples where target type mgs was mentioned during mkfs.lustre for MDS and OSS nodes. Is it correct and when is it used? Any inputs? ~ Thanks, CS.
On Fri, 2009-06-26 at 10:52 -0500, Carlos Santana wrote:> Can a lustre file system have more than one MGS?No. It can have multiple failover paths to a single MGS, but max. 1 MGS per filesystem.> Isn''t it only one per > site?It can be, and that''s how it was designed. IOW, a single MGS can serve multiple filesystems.> I saw some examples where target type mgs was mentioned during > mkfs.lustre for MDS and OSS nodes.Yes. You have to tell the MDSes and OSSes (and clients, via their mount target) where the MGS(es, for failover) are. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090626/f97aecfa/attachment.bin
On Fri, Jun 26, 2009 at 10:57 AM, Brian J. Murrell<Brian.Murrell at sun.com> wrote:> On Fri, 2009-06-26 at 10:52 -0500, Carlos Santana wrote: >> Can a lustre file system have more than one MGS? > > No. ?It can have multiple failover paths to a single MGS, but max. 1 MGS > per filesystem. > >> Isn''t it only one per >> site? > > It can be, and that''s how it was designed. ?IOW, a single MGS can serve > multiple filesystems. > >> I saw some examples where target type mgs was mentioned during >> mkfs.lustre for MDS and OSS nodes. > > Yes. ?You have to tell the MDSes and OSSes (and clients, via their mount > target) where the MGS(es, for failover) are.I know we need to specify --mgsnode, however I saw target type ''--mgs'' being specified on MGS and OSS. May be it was for failover.> > b. > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >
On Fri, 2009-06-26 at 13:18 -0500, Carlos Santana wrote:> > I know we need to specify --mgsnode, however I saw target type ''--mgs'' > being specified on MGS and OSS.This would be an error. Where did you see it? If you saw it in some text from Sun, you can file a bug at our bugzilla.> May be it was for failover.No. Specifying --mgs does nothing for failover on an OSS. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090626/a8455a0d/attachment.bin
Lustre 1.8 manual PDF -> 4.4.Operational Scenarios -> ''IP Network, Single MDS, Single OST, No Failover'' (page# 106) Seems to be mistake in 1.6 PDF as well, but not in HTML. Thanks for clarifying. ~ CS. On Fri, Jun 26, 2009 at 1:23 PM, Brian J. Murrell<Brian.Murrell at sun.com> wrote:> On Fri, 2009-06-26 at 13:18 -0500, Carlos Santana wrote: >> >> I know we need to specify --mgsnode, however I saw target type ''--mgs'' >> being specified on MGS and OSS. > > This would be an error. ?Where did you see it? ?If you saw it in some > text from Sun, you can file a bug at our bugzilla. > >> May be it was for failover. > > No. ?Specifying --mgs does nothing for failover on an OSS. > > b. > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >
On Jun 26, 2009 10:52 -0500, Carlos Santana wrote:> Can a lustre file system have more than one MGS? Isn''t it only one per > site? I saw some examples where target type mgs was mentioned during > mkfs.lustre for MDS and OSS nodes. Is it correct and when is it used?It makes sense generally to have a single MGS per site for multiple filesystems, because if clients are mounting more than one filesystem they can only communicate with a single MGS at a time. In some cases there will of course be a need for multiple MGSes in a single site (e.g. secure and open networks), which is fine as long as clients don''t try to mount from multiple MGSes at once. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
On Fri, 2009-06-26 at 22:05 +0200, Andreas Dilger wrote:> On Jun 26, 2009 10:52 -0500, Carlos Santana wrote: > > Can a lustre file system have more than one MGS? Isn''t it only one per > > site? I saw some examples where target type mgs was mentioned during > > mkfs.lustre for MDS and OSS nodes. Is it correct and when is it used? > > It makes sense generally to have a single MGS per site for multiple > filesystems, because if clients are mounting more than one filesystem > they can only communicate with a single MGS at a time. > > In some cases there will of course be a need for multiple MGSes in > a single site (e.g. secure and open networks), which is fine as long > as clients don''t try to mount from multiple MGSes at once.What kind of problems could this cause? We''ve had this configuration on both segments of Jaguar for some time now with no ill effects that we could attribute to mounting from two MGSes at the same time -- well, maybe that is not entirely true, as one is an MGS, and the other is a combined MGS/MDS. Some clients are talking with up to five separate combined MGS/MDS. -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office
On Jun 26, 2009 16:34 -0400, David Dillow wrote:> On Fri, 2009-06-26 at 22:05 +0200, Andreas Dilger wrote: > > In some cases there will of course be a need for multiple MGSes in > > a single site (e.g. secure and open networks), which is fine as long > > as clients don''t try to mount from multiple MGSes at once. > > What kind of problems could this cause? > > We''ve had this configuration on both segments of Jaguar for some time > now with no ill effects that we could attribute to mounting from two > MGSes at the same time -- well, maybe that is not entirely true, as one > is an MGS, and the other is a combined MGS/MDS. Some clients are talking > with up to five separate combined MGS/MDS.Well, there is only ever a single MGC configured on a client at one time, so if you have multiple MGSes I would suspect that this will cause the clients to be evicted from all but the last MGS, and as a result any config changes made to the first-mounted filesystems will not be seen by the clients. This might not be noticable until you make a config change and half of the clients don''t notice e.g. the new OST or similar. Some time in the future the MGS will also be involved in client recovery (imperative recovery) because the MGS will be the one transmitting the health messages between the clients and servers. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
On Fri, 2009-06-26 at 23:17 +0200, Andreas Dilger wrote:> On Jun 26, 2009 16:34 -0400, David Dillow wrote: > > On Fri, 2009-06-26 at 22:05 +0200, Andreas Dilger wrote: > > > In some cases there will of course be a need for multiple MGSes in > > > a single site (e.g. secure and open networks), which is fine as long > > > as clients don''t try to mount from multiple MGSes at once. > > > > What kind of problems could this cause? > > > > We''ve had this configuration on both segments of Jaguar for some time > > now with no ill effects that we could attribute to mounting from two > > MGSes at the same time -- well, maybe that is not entirely true, as one > > is an MGS, and the other is a combined MGS/MDS. Some clients are talking > > with up to five separate combined MGS/MDS. > > Well, there is only ever a single MGC configured on a client at one time, > so if you have multiple MGSes I would suspect that this will cause the > clients to be evicted from all but the last MGS, and as a result any config > changes made to the first-mounted filesystems will not be seen by the > clients. This might not be noticable until you make a config change and > half of the clients don''t notice e.g. the new OST or similar.This may explain an issue we''ve seen about config changes not propagating, but that occurred on the last filesystem mounted, so I''m not sure. Are there any plans to eliminate this restriction? Data transfer nodes for gridftp want to be able to mount separate Lustre filesystems from different compute resources, which quite often have a MGS for each filesystem, especially if they are Cray systems that have seen upgrades from Lustre 1.4. Is it possible to split an MGT from a combined MGT/MDT, and then combine those into one MGT? That would be a migration path for those systems with multiple filesystems -- much better than a complete reformat, anyways. It would still force sites with mulitple systems that used to have internal MGS service to have all but one with an external MGS now. -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office
On Jun 26, 2009 18:20 -0400, David Dillow wrote:> Are there any plans to eliminate this restriction? Data transfer nodes > for gridftp want to be able to mount separate Lustre filesystems from > different compute resources, which quite often have a MGS for each > filesystem, especially if they are Cray systems that have seen upgrades > from Lustre 1.4.So far it hasn''t come up as a requirement.> Is it possible to split an MGT from a combined MGT/MDT, and then combine > those into one MGT? That would be a migration path for those systems > with multiple filesystems -- much better than a complete reformat, > anyways.Yes, this is in the 1.4->1.6 documentation. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
We have three lustre file systems of which one is "stand alone" and uses it''s own MGS. We mount all three file systems on every client. The only difficulty I came across when mounting all three file systems on a client was that there could be only one MGS per NID. So to make client communicating with both MGSs I created an alias network interface and then in modprobe.conf I configured two NIDs options lnet networks=tcp1(eth1),tcp2(eth1:0) mount -t lustre 10.142.10.201 at tcp1:10.142.10.202 at tcp1:/scratch /scratch mount -t lustre 10.142.10.201 at tcp1:10.142.10.202 at tcp1:/data /data mount -t lustre 10.42.10.203 at tcp2:10.42.10.204 at tcp2:/work /work Regards, Wojciech 2009/6/26 Andreas Dilger <adilger at sun.com>> On Jun 26, 2009 10:52 -0500, Carlos Santana wrote: > > Can a lustre file system have more than one MGS? Isn''t it only one per > > site? I saw some examples where target type mgs was mentioned during > > mkfs.lustre for MDS and OSS nodes. Is it correct and when is it used? > > It makes sense generally to have a single MGS per site for multiple > filesystems, because if clients are mounting more than one filesystem > they can only communicate with a single MGS at a time. > > In some cases there will of course be a need for multiple MGSes in > a single site (e.g. secure and open networks), which is fine as long > as clients don''t try to mount from multiple MGSes at once. > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >-- -- Wojciech Turek Assistant System Manager High Performance Computing Service University of Cambridge Email: wjt27 at cam.ac.uk Tel: (+)44 1223 763517 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090627/210f92de/attachment.html