Greetings,
I am deactivating an older Lustre filesystem in favor of a newer one
(already up and stable).
The message in Lustre-discuss Digest Vol 45 Issue 22 stated (with two
of my comments in-line):
Message: 2
Date: Sun, 11 Oct 2009 19:05:07 +0200
From: Bernd Schubert <bs_lists at aakef.fastmail.fm>
Subject: Re: [Lustre-discuss] Moving MGS to separate device
To: lustre-discuss at lists.lustre.org
Message-ID: <200910111905.08076.bs_lists at aakef.fastmail.fm>
Content-Type: Text/Plain; charset="iso-8859-15"
Hello Wojciech,
I already did this several times, here are the steps I so far used:
1) Remove MGS from MDT-device
tunefs.lustre --nogms /dec/mdt_device
^^^^
Megan: I am assuming "--nomgs" here.
2) Create new MGS
mkfs.lustre --mgs /dev/mgs_device
3) Make sure OSTs and MDTs re-register with the MGS:
tunefs.lustre --writeconf /dev/device
Megan: Do I need to do this even if the MGS is being moved
from a shared device with an MDT to its own device/hard drive on the
same physical server (same MAC addr, IP, hostname etc.....)?
I''m not sure if writeconf is really necessary, but so far I always did
it to
make sure everything goes smoothly (clients shouldn''t have the
filesystem
mounted at this step).
5) Mount MGS, MDT, OSTs
4) Re-apply settings done with lctl.
Megan: Why are the above ordered the way they are? Shouldn''t
I mount first and then apply the settings? (I didn''t think I could
"lctl" an unmounted OST/MDT etc.)
As you also wrote (private mail), it might be possible to just copy over the
CONFIGS directory, but I never tried to do that.
Hope it helps,
Bernd
Megan: Thanks!
Additionally in Lustre-discuss Digest Vol 45 Issue 36 A. Dilger opined:
"...128Mb is the max size the MGS would ever need."
Wow! That is small. I know the MGS does not store the sort of data
retained on an MDT for a multi-Tb filesystem, but really 128Mb max
would be used?
Thanks for you advice!
Megan Larko
On 2009-10-22, at 13:41, Ms. Megan Larko wrote:> Additionally in Lustre-discuss Digest Vol 45 Issue 36 A. Dilger > opined: > "...128Mb is the max size the MGS would ever need." > > Wow! That is small. I know the MGS does not store the sort of data > retained on an MDT for a multi-Tb filesystem, but really 128Mb max > would be used?I honestly don''t have much data on this, so I can''t say with absolute certainty whether that is the max. There are only a handful of smallish config files per server, and one for all of the clients. The MGS doesn''t need very much space. Maybe someone with access to ORNL Spider (largest Lustre filesystem) can comment on how much space is used in the /CONFIGS/ directory? Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
On Fri, 2009-10-23 at 03:00 -0600, Andreas Dilger wrote:> On 2009-10-22, at 13:41, Ms. Megan Larko wrote: > > Additionally in Lustre-discuss Digest Vol 45 Issue 36 A. Dilger > > opined: > > "...128Mb is the max size the MGS would ever need." > > > > Wow! That is small. I know the MGS does not store the sort of data > > retained on an MDT for a multi-Tb filesystem, but really 128Mb max > > would be used? > > > I honestly don''t have much data on this, so I can''t say with absolute > certainty whether that is the max. There are only a handful of smallish > config files per server, and one for all of the clients. The MGS > doesn''t > need very much space. > > Maybe someone with access to ORNL Spider (largest Lustre filesystem) can > comment on how much space is used in the /CONFIGS/ directory?With one filesystem configured -- 672 OSTs -- CONFIGS contains 676 files and uses 124 MB or so. We expect to have another few more filesystems using this MGS, so extrapolating from current usage I guess we''ll top out at around 496 MB for the large filesystems, and perhaps a few MB more for the smaller ones. -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office
On Thursday 22 October 2009, Ms. Megan Larko wrote:> Greetings, > > I am deactivating an older Lustre filesystem in favor of a newer one > (already up and stable). > The message in Lustre-discuss Digest Vol 45 Issue 22 stated (with two > of my comments in-line): > > Message: 2 > Date: Sun, 11 Oct 2009 19:05:07 +0200 > From: Bernd Schubert <bs_lists at aakef.fastmail.fm> > Subject: Re: [Lustre-discuss] Moving MGS to separate device > To: lustre-discuss at lists.lustre.org > Message-ID: <200910111905.08076.bs_lists at aakef.fastmail.fm> > Content-Type: Text/Plain; charset="iso-8859-15" > > Hello Wojciech, > > I already did this several times, here are the steps I so far used: > > 1) Remove MGS from MDT-device > > tunefs.lustre --nogms /dec/mdt_device > ^^^^ > Megan: I am assuming "--nomgs" here.Yes, sorry a typo.> > 2) Create new MGS > > mkfs.lustre --mgs /dev/mgs_device > > 3) Make sure OSTs and MDTs re-register with the MGS: > > tunefs.lustre --writeconf /dev/device > > Megan: Do I need to do this even if the MGS is being moved > from a shared device with an MDT to its own device/hard drive on the > same physical server (same MAC addr, IP, hostname etc.....)?I did it to make sure MDT and OSTs re-register with the MGS, so to make sure the MGS really knows about them. In the end the MGS is only there to know about OSTs and MDTs and to provide those information to clients. I think on mounting of MDTs and OSTs they always contact the MGS, but if the MGS is not available they will give up, so it **might** happen that they wouldn''t contact the MGS and so wouldn''t be registered. --writeconf will ensure that. As I also wrote, it might work to simply copy the CONFIGS directory, but I didn''t test that yet.> > I''m not sure if writeconf is really necessary, but so far I always did it > to make sure everything goes smoothly (clients shouldn''t have the > filesystem mounted at this step). > > 5) Mount MGS, MDT, OSTs > > 4) Re-apply settings done with lctl. > > Megan: Why are the above ordered the way they are? Shouldn''t > I mount first and then apply the settings? (I didn''t think I could > "lctl" an unmounted OST/MDT etc.)And another typo, actually even two. lctl settings can be applied only with the filesystem being mounted. So 4) Mount MGS, MDT, OSTs 5) Re-apply settings done with lctl. Cheers, Bernd -- Bernd Schubert DataDirect Networks