As of Lustre 1.6, servers can be started in any order (after the initial registration at first startup). Internally, this required significant bending of our connection rules, and with a move toward ZFS becomes even more burdensome. So my question to the Lustre community is this: would anyone strenuously object to a startup ordering requirement that the MGS must be started before any other servers? This would probably be in the Lustre 3.0 timeframe. It is also likely that we will have to divorce the MGS and MDT onto separate devices -- no more "combo" MDT/MGSes.> > NR> I think the only reason to have a local config file is to be able to > > NR> start a server in the absence of the MGS. How much effort do we want > > NR> to expend to be able to keep that ability? I don''t think it''s a huge > > NR> burden to say "MGS must be started first".
On Wed, 2009-03-25 at 12:48 -0700, Nathaniel Rutman wrote:> As of Lustre 1.6, servers can be started in any order (after the initial registration at first startup). Internally, this required significant bending of our connection rules, and with a move toward ZFS becomes even more burdensome. > So my question to the Lustre community is this: would anyone strenuously object to a startup ordering requirement that the MGS must be started before any other servers? > This would probably be in the Lustre 3.0 timeframe. It is also likely that we will have to divorce the MGS and MDT onto separate devices -- no more "combo" MDT/MGSes.I''ll ask before anyone else does... would this require the MGS be available at any time a server (or client?) needs to start rather than the optional behaviour (for anything but first time server starts) that we currently enjoy? IOW, does this make the MGS a much more critical component of the filesystem than it is currently? b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-devel/attachments/20090325/67ee0ef3/attachment.bin
Nathaniel Rutman wrote:> As of Lustre 1.6, servers can be started in any order (after the initial > registration at first startup). Internally, this required significant > bending of our connection rules, and with a move toward ZFS becomes even > more burdensome. So my question to the Lustre community is this: would > anyone strenuously object to a startup ordering requirement that the MGS > must be started before any other servers? This would probably be in the > Lustre 3.0 timeframe. It is also likely that we will have to divorce > the MGS and MDT onto separate devices -- no more "combo" MDT/MGSes. >> > NR> I think the only reason to have a local config file is to be >> able to >> > NR> start a server in the absence of the MGS. How much effort do >> we want >> > NR> to expend to be able to keep that ability? I don''t think it''s >> a huge >> > NR> burden to say "MGS must be started first". >This is virtually ensured today due to how the timeouts and ordering works. 1) It is a real PITA to script up different server start orders to deal with reformat and write_conf. It is easier to just script one correct way of doing this - KISS if you will. 2) At large scale, the timeout cascading on the OSTs (many OSTs per OSS) from a missing MGS requires it be started first. 3) With bug 14134 and --nomgs and --nosvc options for starting, it makes starting a combo MGS and MDS "correctly" much easier. Nic
On Wed, 2009-03-25 at 12:48 -0700, Nathaniel Rutman wrote:> As of Lustre 1.6, servers can be started in any order (after the > initial registration at first startup). Internally, this required > significant bending of our connection rules, and with a move toward > ZFS becomes even more burdensome. > So my question to the Lustre community is this: would anyone > strenuously object to a startup ordering requirement that the MGS must > be started before any other servers? > This would probably be in the Lustre 3.0 timeframe. It is also likely > that we will have to divorce the MGS and MDT onto separate devices -- > no more "combo" MDT/MGSes. >"server startup" being, things wont start working until the MGS is up, or, server startup commands will fail if the MGS is not up? The former, is much better then the latter. It allows the system to potentially bring itself back up automatically if things get rebooted (power bump, spike, etc). Kevin> > > NR> I think the only reason to have a local config file is to be > able to > > > NR> start a server in the absence of the MGS. How much effort do > we want > > > NR> to expend to be able to keep that ability? I don''t think > it''s a huge > > > NR> burden to say "MGS must be started first". > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >
Robert LeBlanc wrote:> For what it''s worth, we have multiple Lustre filesystems, so we already have > our MGS and MDT separate. We also have heartbeat start up the MGS first. > >From what I recall, the mounts are pretty quick with heartbeat and I''m not > sure the MGS and MDTs are ready to go when heartbeat starts on the next file > system. I know the OSTs take some time for timeout. >What would happen with the new process is that the OST and MDT mounts would block (probably with an eventual timeout) until the MGS was ready. So as long as you''re not serializing mounts in the wrong order, this should be fine.> > On 3/25/09 1:48 PM, "Nathaniel Rutman" <Nathan.Rutman at Sun.COM> wrote: > > >> As of Lustre 1.6, servers can be started in any order (after the initial >> registration at first startup). Internally, this required significant bending >> of our connection rules, and with a move toward ZFS becomes even more >> burdensome. >> So my question to the Lustre community is this: would anyone strenuously >> object to a startup ordering requirement that the MGS must be started before >> any other servers? >> This would probably be in the Lustre 3.0 timeframe. It is also likely that we >> will have to divorce the MGS and MDT onto separate devices -- no more "combo" >> MDT/MGSes. >> >> >>>> NR> I think the only reason to have a local config file is to be able to >>>> NR> start a server in the absence of the MGS. How much effort do we want >>>> NR> to expend to be able to keep that ability? I don''t think it''s a huge >>>> NR> burden to say "MGS must be started first". >>>> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> >> > >
James Beal wrote:> > On 25 Mar 2009, at 19:48, Nathaniel Rutman wrote: > >> As of Lustre 1.6, servers can be started in any order (after the >> initial registration at first startup). Internally, this required >> significant bending of our connection rules, and with a move toward >> ZFS becomes even more burdensome. >> So my question to the Lustre community is this: would anyone >> strenuously object to a startup ordering requirement that the MGS >> must be started before any other servers? >> This would probably be in the Lustre 3.0 timeframe. It is also >> likely that we will have to divorce the MGS and MDT onto separate >> devices -- no more "combo" MDT/MGSes. >> >>>> NR> I think the only reason to have a local config file is to be >>>> able to >>>> NR> start a server in the absence of the MGS. How much effort do >>>> we want >>>> NR> to expend to be able to keep that ability? I don''t think it''s >>>> a huge >>>> NR> burden to say "MGS must be started first". > > Our current lustre heartbeat configurations rely on being able to > start servers in any order. > > While ensuring that the MGS is started before the MDT would be > relatively simple. Ensuring it starts before the OSTs would be > difficult as we a system where we have pairs of heartbeat systems and > they do not know the state of one and another. We have found small > heartbeat systems to be more reliable than large ones and this is the > reason we have multiple simple heartbeat systems. >Thanks - this is exactly the kind of feedback I was hoping for. You wouldn''t actually have to ensure the MGS starts first -- the OST and MDT mounts would block until the MGS was ready. So if they are truly independent, everything will simply wait until the MGS has started, if I understand your setup correctly.
Brian J. Murrell wrote:> On Wed, 2009-03-25 at 12:48 -0700, Nathaniel Rutman wrote: > >> As of Lustre 1.6, servers can be started in any order (after the initial registration at first startup). Internally, this required significant bending of our connection rules, and with a move toward ZFS becomes even more burdensome. >> So my question to the Lustre community is this: would anyone strenuously object to a startup ordering requirement that the MGS must be started before any other servers? >> This would probably be in the Lustre 3.0 timeframe. It is also likely that we will have to divorce the MGS and MDT onto separate devices -- no more "combo" MDT/MGSes. >> > > I''ll ask before anyone else does... would this require the MGS be > available at any time a server (or client?) needs to start rather than > the optional behaviour (for anything but first time server starts) that > we currently enjoy? IOW, does this make the MGS a much more critical > component of the filesystem than it is currently? >yes. It''s already required for client starts. What we would do is have the MDT / OST server mounts block until the MGS is up, probably with a timeout. And yes, this will be some amount less flexible than the current startup order, but we gain advantages from it - simplified import states, more centralized configuration, single-path disk access. The question is how burdensome really will this limitation be. So far I have not heard very much gnashing of teeth and rending of garments.
Nathaniel Rutman
2009-Apr-01 15:16 UTC
[Lustre-devel] [Lustre-discuss] Start the MGS first?
Kevin Fox wrote:> On Wed, 2009-03-25 at 12:48 -0700, Nathaniel Rutman wrote: > >> As of Lustre 1.6, servers can be started in any order (after the >> initial registration at first startup). Internally, this required >> significant bending of our connection rules, and with a move toward >> ZFS becomes even more burdensome. >> So my question to the Lustre community is this: would anyone >> strenuously object to a startup ordering requirement that the MGS must >> be started before any other servers? >> This would probably be in the Lustre 3.0 timeframe. It is also likely >> that we will have to divorce the MGS and MDT onto separate devices -- >> no more "combo" MDT/MGSes. >> >> > > "server startup" being, things wont start working until the MGS is up, > or, server startup commands will fail if the MGS is not up? > > The former, is much better then the latter. It allows the system to > potentially bring itself back up automatically if things get rebooted > (power bump, spike, etc). > >The former. Server mount commands would just block until the MGS was available (probably with an optional timeout). So the only thing that would fail is serialized startup of OST before MGS -- independent, unordered startup would sort itself out.> Kevin > > >>>> NR> I think the only reason to have a local config file is to be >>>> >> able to >> >>>> NR> start a server in the absence of the MGS. How much effort do >>>> >> we want >> >>>> NR> to expend to be able to keep that ability? I don''t think >>>> >> it''s a huge >> >>>> NR> burden to say "MGS must be started first". >>>> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> >> >> > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel >