Hello list, the manual (May 2008) describes on page 4-14 how to remove an OST from LUSTRE. Its says "deactivate the OST (make it read-only). Hm, i''am a bit puzzled here. When i deactivate the OST on the MDS using ''lctl --device 18 conf_param foo-OST000d.osc.active=0'' the device is deactivated but i cannot read any files from it any longer. Quite logical. On the other hand using lctl "readonly" option is specified as "dangerous". So what is the correct approach of replacing an OST and/or what is the correct syntax for lctl (or whatever command) putting a device in read-only mode ? When the OST is in read-only mode and i "recopy" the file over the original one what happens to the directory structure ? Is it necessary to recreate it ? Or is it enough to do something like this (mentioned under 3. on the above lustre manual page): cp /lustrefs/dir1/foo/dat.file /tmp/. cp /tmp/dat.file /lustrefs/dir1/foo/. Sorry if this has been asked before but i want to be sure not to break our productional system. Thanks and Regards Heiko
schroete at iup.physik.uni-bremen.de
2009-Jan-16 14:18 UTC
[Lustre-discuss] Removing an OST
Sorry, i forgot. Lustre Version 1.6.6 Kernel 2.6.22.19 vanilla with lustre patches. Thanks and Regards Heiko
On Jan 16, 2009 13:45 +0100, Heiko Schroeter wrote:> the manual (May 2008) describes on page 4-14 how to remove an OST from LUSTRE. > Its says "deactivate the OST (make it read-only). > > Hm, i''am a bit puzzled here. > When i deactivate the OST on the MDS using > ''lctl --device 18 conf_param foo-OST000d.osc.active=0'' the device is > deactivated but i cannot read any files from it any longer. Quite logical.Ah, it looks like you are confusing two different commands here. By using "lctl conf_param ..." you are storing this setting in the filesystem-wide configuration file, permanently deactivating the device on all nodes. If you just want to deactivate the OST only on the MDS you can use: lctl set_param osc.foo-OST000d.active=0 This means the MDS will not allocate any new objects on this OST. It will still be active on all of the clients, so they can read or write to this file, or delete it.> On the other hand using lctl "readonly" option is specified as "dangerous".Yes, this isn''t something you should be using - it is for regression testing only and doesn''t do what you want.> So what is the correct approach of replacing an OST and/or what is the > correct syntax for lctl (or whatever command) putting a device in > read-only mode ? > > When the OST is in read-only mode and i "recopy" the file over the original > one what happens to the directory structure?If the OST is deactivated on only the MDS, but not the clients, then the deleted file will be cleaned up correctly on the OST.> Is it necessary to recreate > it? Or is it enough to do something like this (mentioned under 3. on the > above lustre manual page):> cp /lustrefs/dir1/foo/dat.file /tmp/. > cp /tmp/dat.file /lustrefs/dir1/foo/.Using "cp" likely isn''t what you want to do, because "cp" doesn''t change the inode (object) allocation (even on local filesystems). More likely you want to do: cp .../dat.file .../dat.file.tmp mv .../dat.file.tmp .../dat.file This will create a new file (not on the OST deactivated at the MDS), and delete the old one when the new one is moved over top of it. Once you have moved all of your files off of this OST then it is safe to remove and the "conf_param" command can be used to deactivate it permanently. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Thanks very much for the clarification and it works. Actually i wasn''t aware of the differences between set_param and config_param. Ashes on me. Is it possible to get these infos into the documentation as it may hit one or the other in the lustre community ? The manual only gives two ''general'' approaches one of which is to ''deactivate the OST'', the other one just says to remount the partition with "ext3" which didn''t work for the lustre partitions here. Maybe something like (if it is correct): Removing/Replacing an OST with recreating the files into lustre: Deactivate the OST for the MDS only, using ''set_param'' instead of ''config_param'', so clients are able to read the data. i.e:> lctl set_param foo-OST000d-osc.active=0Copy the data on a client to a temp storage outside (does it has to be outside?) the lustre system and ''move'' them back into lustre to create new inodes entries on the MDS.> > cp .../dat.file .../dat.file.tmp > mv .../dat.file.tmp .../dat.file >After moving all files the OST can be permantly set inactive with.i.e.:> lctl config_param foo-OST000d-osc.active=0Thanks and Regards Heiko
Andreas, thank you very much for this log sought-after explanation! As far as I have read manuals and the mailing list, no one ever put these two commands side by side. One would only find the advice to "make the OST read-only" - and the how-to side I also only found the "lctl --conf_param..." version. I think anyhow the meaning and syntax of the various set_param commands is never really explained in the manual? Regards, Thomas Andreas Dilger wrote:> On Jan 16, 2009 13:45 +0100, Heiko Schroeter wrote: >> the manual (May 2008) describes on page 4-14 how to remove an OST from LUSTRE. >> Its says "deactivate the OST (make it read-only). >> >> Hm, i''am a bit puzzled here. >> When i deactivate the OST on the MDS using >> ''lctl --device 18 conf_param foo-OST000d.osc.active=0'' the device is >> deactivated but i cannot read any files from it any longer. Quite logical. > > Ah, it looks like you are confusing two different commands here. > By using "lctl conf_param ..." you are storing this setting in the > filesystem-wide configuration file, permanently deactivating the device > on all nodes. > > If you just want to deactivate the OST only on the MDS you can use: > > lctl set_param osc.foo-OST000d.active=0 > > This means the MDS will not allocate any new objects on this OST. It will > still be active on all of the clients, so they can read or write to this > file, or delete it. > >> On the other hand using lctl "readonly" option is specified as "dangerous". > > Yes, this isn''t something you should be using - it is for regression > testing only and doesn''t do what you want. > >> So what is the correct approach of replacing an OST and/or what is the >> correct syntax for lctl (or whatever command) putting a device in >> read-only mode ? >> >> When the OST is in read-only mode and i "recopy" the file over the original >> one what happens to the directory structure? > > If the OST is deactivated on only the MDS, but not the clients, then the > deleted file will be cleaned up correctly on the OST. > >> Is it necessary to recreate >> it? Or is it enough to do something like this (mentioned under 3. on the >> above lustre manual page): > > >> cp /lustrefs/dir1/foo/dat.file /tmp/. >> cp /tmp/dat.file /lustrefs/dir1/foo/. > > Using "cp" likely isn''t what you want to do, because "cp" doesn''t change > the inode (object) allocation (even on local filesystems). More likely > you want to do: > > cp .../dat.file .../dat.file.tmp > mv .../dat.file.tmp .../dat.file > > This will create a new file (not on the OST deactivated at the MDS), > and delete the old one when the new one is moved over top of it. > > Once you have moved all of your files off of this OST then it is safe > to remove and the "conf_param" command can be used to deactivate it > permanently. > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-- -------------------------------------------------------------------- Thomas Roth Gesellschaft f?r Schwerionenforschung Planckstr. 1 - 64291 Darmstadt, Germany Department: Informationstechnologie Location: SB3 1.262 Phone: +49-6159-71 1453 Fax: +49-6159-71 2986 -------------- next part -------------- A non-text attachment was scrubbed... Name: t_roth.vcf Type: text/x-vcard Size: 298 bytes Desc: not available Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090119/271c59d4/attachment.vcf
On Mon, 2009-01-19 at 16:45 +0100, Thomas Roth wrote:> Andreas, thank you very much for this log sought-after explanation! > > As far as I have read manuals and the mailing list, no one ever put these two commands side by side. > One would only find the advice to "make the OST read-only" - and the how-to side I also only found > the "lctl --conf_param..." version. > > I think anyhow the meaning and syntax of the various set_param commands is never really explained in > the manual?That would be good to file a bug on then. We always welcome documentation bugs. Sometimes it''s difficult for people with a good understanding of a piece of software to see how under-explained a topic in the documentation might be. Thanx! b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090119/4d46a13c/attachment.bin
On Jan 19, 2009 10:18 +0100, Heiko Schroeter wrote:> Is it possible to get these infos into the documentation as it may hit one or > the other in the lustre community ?Yes, we always welcome improvements to the documentation. As you can imagine, I rarely need to reference the documentation myself, so I don''t know whether it is up-to-date or not, or if it explains a given topic clearly enough or not.> The manual only gives two ''general'' approaches one of which is to ''deactivate > the OST'', the other one just says to remount the partition with "ext3" which > didn''t work for the lustre partitions here. > > Maybe something like (if it is correct): > > Removing/Replacing an OST with recreating the files into lustre: > Deactivate the OST for the MDS only, using ''set_param'' instead > of ''config_param'', so clients are able to read the data. i.e: > > > lctl set_param foo-OST000d-osc.active=0This should be "lctl set_param osc.foo-OST000d-osc.active=0> Copy the data on a client to a temp storage outside (does it has to be > outside?) the lustre system and ''move'' them back into lustre to create new > inodes entries on the MDS.Actually, it should NOT be on storage outside Lustre in this case.> > cp .../dat.file .../dat.file.tmp > > mv .../dat.file.tmp .../dat.file> After moving all files the OST can be permantly set inactive with.i.e.: > > lctl config_param foo-OST000d-osc.active=0Correct. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Hm, i do got it that one has to ''move'' the file back. But ...> > Copy the data on a client to a temp storage outside (does it has to be > > outside?) the lustre system and ''move'' them back into lustre to create > > new inodes entries on the MDS. > > Actually, it should NOT be on storage outside Lustre in this case. >why staying inside lustre (in this case) ? (Bandwidth ?, Performance ?...) What would be a different case/scenario that one has to go ''out'' of lustre ? Thanks and Regards Heiko
You have to copy the file to a new name _on Lustre_ to allocate a new inode (new storage) on Lustre, and then "mv" it to replace the original file. Mv across filesystems is basically a cp, which will overwrite the destination file, not replace the inode. Kevin On Jan 20, 2009, at 3:55 AM, Heiko Schroeter <schroete at iup.physik.uni-bremen.de > wrote:> Hm, i do got it that one has to ''move'' the file back. But ... > >>> Copy the data on a client to a temp storage outside (does it has >>> to be >>> outside?) the lustre system and ''move'' them back into lustre to >>> create >>> new inodes entries on the MDS. >> >> Actually, it should NOT be on storage outside Lustre in this case. >> > > why staying inside lustre (in this case) ? (Bandwidth ?, > Performance ?...) > > What would be a different case/scenario that one has to go ''out'' of > lustre ? > > Thanks and Regards > Heiko > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss